Services for Organizations

Using our research, best practices and expertise, we help you understand how to optimize your business processes using applications, information and technology. We provide advisory, education, and assessment services to rapidly identify and prioritize areas for improvement and perform vendor selection

Consulting & Strategy Sessions

Ventana On Demand

    Services for Investment Firms

    We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

    Consulting & Strategy Sessions

    Ventana On Demand

      Services for Technology Vendors

      We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

      Analyst Relations

      Demand Generation

      Product Marketing

      Market Coverage

      Request a Briefing


        Ventana Research Analyst Perspectives

        << Back to Blog Index

        Astronomer’s Cloud-Based Data Orchestration Brings Efficiency

        I have recently written about the organizational and cultural aspects of being data-driven, and the potential advantages data-driven organizations stand to gain by responding faster to worker and customer demands for more innovative, data-rich applications and personalized experiences. I have also explained that data-driven processes require more agile, continuous data processing, with an increased focus on extract, load and transform processes — as well as change data capture and automation and orchestration — as part of a DataOps approach to data management. Safeguarding the health of data pipelines is fundamental to ensuring data is integrated and processed in the sequence required to generate business intelligence. The significance of these data pipelines to delivering data-driven business strategies has led to the emergence of vendors, such as Astronomer, focused on enabling organizations to orchestrate data engineering pipelines and workflows.

        Astronomer was founded in 2018, building on a background of providing data engineering services to create a business around the Apache Airflow open-source workflow monitoring and management project. Apache Airflow began as an internal development project within Airbnb in 2014, but became an Apache Software Foundation project in 2016. It enables data engineers to use Python to programmatically author, schedule and monitor workflows. Astronomer’s workforce has been heavily involved in the Apache Airflow development project since the company’s foundation, contributing to significant enhancements such as the scheduler performance and high-availability capabilities delivered in 2020 with Apache Airflow 2.0, for example.

        It was also in 2020 that Astronomer made the decision to focus its attention on building a cloud offering through which it could deliver Apache Airflow as a managed platform, rather than providing technical support services for Apache Airflow deployments. The resulting Astro service was launched in June 2022. Its development was financed by venture capital funding, including a $213 million Series C round announced in March 2022 that was led by Insight Partners, along with Meritech Capital, Salesforce Ventures, J.P. Morgan, K5 Global, Sutter Hill Ventures, Venrock and Sierra Ventures. The funding round also facilitated Astronomer’s acquisition of data pipeline observability and data lineage specialist Datakin, which was founded by the creators of the OpenLineage and Marquez open-source projects. The addition of data lineage capabilities based on the OpenLineage project adds data observability capabilities to Astronomer’s Airflow-as-a-service capabilities, with the company positioning the combined offering as a data orchestration cloud service.

        The need for more agile data pipelines is driven by the need for real-time data processing. More frequent data analysis requires data to be integrated, cleansed, enriched, transformed and processed for analysis in a continuous and agile process. As such, data-driven organizations are increasingly treating the steps involved in extracting, integrating, aggregating, preparing, transforming and loading data as a continual process, with data pipelines used to enable the flow of information through the organization, increasingly scheduled, automated and orchestrated by data engineers without the need for constant manual intervention. I assert that by 2024, 6 in ten organizations will adopt data engineering processes that span data integration, transformation and preparation, producing repeatable data pipelines that create more agile information architectures.

        VR_2022_Data_Managemen_Assertion_4_Square (2)-pngApache Airflow was designed to support robust and healthy data pipelines by providing a platform that can be used by data engineers to programmatically author, schedule and monitor data workflows. These could be data integration workflows to extract and combine data from multiple sources and load it into a target data platform. Additionally, Airflow can be used to schedule and orchestrate data science and machine learning pipelines involving multiple data integration and processing steps as well as operational analytics pipelines to generate immediate insight from operational applications. As such, Airflow has a potential role to play in both MLOps and Analytics Ops.

        While data engineers can deploy and run Apache Airflow on-premises or in the cloud, Astronomer’s Astro service delivers these capabilities as a managed service available on Amazon Web Services, Google Cloud or Microsoft Azure. In developing Astro, Astronomer reengineered Airflow for the cloud, with optimized configuration and auto-scaling capabilities, while the managed service approach is also designed to reduce infrastructure consumption for long-term tasks as well as reducing the need for data engineers to shoulder security, upgrading and other management responsibilities, enabling them to focus on data pipelines. Astro also offers users the ability to visually monitor activity and data pipeline dependencies and, thanks to the acquisition of Datakin, the ability to collect lineage metadata as well as identify and monitor data quality metrics to improve trust in data. As I noted earlier this year, monitoring the quality and reliability of data is a key component of data observability's role in ensuring healthy data pipelines.

        VR_2022_Data_Operations_Assertion_4_Square (1)-pngAlthough the capabilities currently offered by Astronomer do not equate to a fully-fledged data observability offering today, information collected via OpenLineage can be used to support data observability as well as root cause analysis and impact planning. I assert that, through 2025, data observability will continue to be a priority for the evolution of data operations products as vendors deliver more automated approaches to data engineering, improving trust in enterprise data.

        While Apache Airflow and OpenLineage provide the core building blocks of the Astro data orchestration cloud service, there are opportunities for Astronomer to expand on this functionality with automated anomaly detection, alerting and root cause analysis, for example. That said, I recommend that all organizations currently managing Apache Airflow deployments or considering the use of Apache Airflow for evaluating data platforms to orchestrate data engineering pipelines and workflows include Astronomer in the evaluations.

        Regards,

        Matt Aslett

        Authors:

        Matt Aslett
        Director of Research, Analytics and Data

        Matt Aslett leads the software research and advisory for Analytics and Data at Ventana Research, now part of ISG, covering software that improves the utilization and value of information. His focus areas of expertise and market coverage include analytics, data intelligence, data operations, data platforms, and streaming and events.

        JOIN OUR COMMUNITY

        Our Analyst Perspective Policy

        • Ventana Research’s Analyst Perspectives are fact-based analysis and guidance on business, industry and technology vendor trends. Each Analyst Perspective presents the view of the analyst who is an established subject matter expert on new developments, business and technology trends, findings from our research, or best practice insights.

          Each is prepared and reviewed in accordance with Ventana Research’s strict standards for accuracy and objectivity and reviewed to ensure it delivers reliable and actionable insights. It is reviewed and edited by research management and is approved by the Chief Research Officer; no individual or organization outside of Ventana Research reviews any Analyst Perspective before it is published. If you have any issue with an Analyst Perspective, please email them to ChiefResearchOfficer@ventanaresearch.com

        View Policy

        Our Analysts

        Subscribe to Email Updates

        Posts by Month

        see all

        Posts by Topic

        see all


        Analyst Perspectives Archive

        See All