(nmedia/Shutterstock)
Organizations have been engaged on turning into extra data-driven for a few years at this level, with combined outcomes. We perceive that the worth of information is plain. Nonetheless, it has now turn out to be extra essential than ever for organizations to operate as data-driven entities. The urgency of this situation is underscored by the emergence of AI and ML.
The 2024 State of Apache Airflow helps us perceive the expansion, utilization, and consumer neighborhood dynamics of navigating the complexities of information orchestration, information pipelines, and information supply.
The info for the report was compiled by surveying 281 professionals who have been immediately or not directly liable for managing their group’s Airflow cases. This included information engineers, software program engineers, DevOps engineers, and options architects.
The report highlights three essential traits – fast development in Airflow adoption, rising reliability of Airflow for mission-critical information supply, and Airflow turning into the usual for AI pipelines.
Apache Airflow is an open-source platform for creating, scheduling, and monitoring information and computing workflows. With its graphical UI, integration with main cloud platforms, and ease of use, Airflow is turning into a well-liked device for orchestrating workflows or pipelines.
One of many key findings of the report is that AL and ML services by Airflow spiked by 24 p.c 12 months over 12 months. The highest purposes for Airflow included the inner analytics dashboard, GenAI, and ML. There has additionally been a 68 p.c enhance in Airflow downloads 12 months on 12 months with over 165.7 million downloads. Airflow contributors outpace Apache tasks Spark and Kafka with 2.8K contributors
Whereas the AI market is projected to proceed rising, there are some issues, together with fragmented device stacks, siloed instruments, and a looming menace of stricter compliance and operational necessities.
In line with Apache, a rising variety of companies are addressing these issues utilizing Astronomer – the unified information platform constructed on Apache Airflow service. Astronomer permits organizations to make sure information privateness, and unify information throughout the cloud, group, and deployments. It additionally helps speed up improvement by way of seamless integrations.
(Andrey Suslov/Shutterstock)
The report reveals that Apache Airflow has advanced from a distinct segment device to a mission-critical asset. Two-thirds (67 p.c) of corporations have greater than 6 folks utilizing Airflow. This means a rising pattern of information groups utilizing Airflow to ship enterprise outcomes.
It isn’t simply the variety of folks utilizing Airflow that’s spectacular, the frequency of interplay can also be excessive. The report reveals that 55 p.c of customers work together with Airflow every day, with one other 26 p.c utilizing it no less than as soon as every week. It seems that corporations are rising of their confidence in Airflow and their imaginative and prescient of what the platform can accomplish.
Round half (46 p.c) of respondents shared that they really feel Airflows is a “crucial” device for his or her enterprise operations. Outages and the necessity for scalability have been the highest two issues of the respondents. The respondents additionally shared that probably the most regarding features of information downtime embody inner programs being negatively affected, throttling of information group productiveness, and points with revenue-generation apps or companies.
The outcomes of the survey spotlight how Apache Airflow meets the rising challenges of information orchestration, scaling, and facilitation of greatest practices. Maybe the best impression of Airflow has been in AI pipeline supply. The momentum positions Airflows as one of many main information orchestration platforms for AI improvement and deployment.
Associated Objects
How Airflow 2.8 Makes Constructing and Working Knowledge Pipelines Simpler
Astronomer’s Apache Airflow-Powered Astro Outperforms Opponents with 30M Month-to-month Downloads
AWS Publicizes Basic Availability of Amazon Managed Workflows for Apache Airflow
