dagster.io

# Dagster

> Dagster is a data orchestrator built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability. It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports. With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Dagster's design allows it to model and manage the flow of data and the execution of compute tasks across various systems, which can include tasks such as data ingestion, transformation, and analysis. Unlike traditional task-centric orchestrators, Dagster's core abstractions of 'ops', 'assets', and 'resources' facilitate code-native pipeline definitions with an asset-first approach that focuses on the data products you want to create.

**Key Features:**
- **Asset-centric orchestration**: Model data assets (tables, ML models, reports) rather than just tasks
- **Software engineering best practices**: Built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production
- **Integrated observability**: Built-in lineage tracking, data quality monitoring, and operational metadata
- **Python-native**: Python-native data orchestrator for complex, modern data pipelines
- **Flexible deployment**: From local development to production clusters
- **Rich integrations**: Works with dbt, Snowflake, Spark, Databricks, and other modern data tools

**Core Concepts:**
- **Assets**: Data assets are a fundamental concept in Dagster. They represent the tangible outputs of your data pipelines, and are ultimately the end product your stakeholders care about
- **Ops**: Individual units of computation that can be composed into jobs
- **Jobs**: Collections of ops that define how to compute a set of assets
- **Resources**: Configurable objects that provide external services (databases, APIs, etc.)
- **Schedules and Sensors**: Dagster allows you to define schedules to run your data pipelines at a specific frequency, and sensors to trigger pipeline runs based on external events
- **Partitions**: For batch computations that need to be run over a dataset sliced by time or another dimension, Dagster provides partitions and partition sets to organize and execute these computations

## Product

- [Product Overview](https://dagster.io/platform-overview): Break data silos and ship faster with Dagster! Your unified control plane for building, observing, and scaling reliable data and AI pipelines
- [Data Orchestration](https://dagster.io/platform-overview/data-orchestration): Data orchestration doesn’t have to be complicated. With Dagster, you can automate workflows, scale effortlessly, and keep everything running smoothly
- [Data Catalog](https://dagster.io/platform-overview/data-catalog): Dagster’s data catalog helps you easily find, organize, and trust your data - all in one place. Discover smarter data management today
- [Data Quality](https://dagster.io/platform-overview/data-quality): Bad data leads to bad decisions. Dagster helps you catch issues early, validate data in real-time, and ensure your pipelines run reliably
- [Cost Insights](https://dagster.io/platform-overview/cost-insights): Get clear cost insights into your workflows - track spending, optimize resources, and cut unnecessary costs with ease
- [Compass](https://compass.dagster.io): Ask questions in plain language and get instant insights, visualizations, and definitions — all inside Slack. Governed by your data team, Compass helps every team make data-driven decisions in seconds
- [Integrations](https://dagster.io/integrations): Easily connect Dagster with your favorite tools - Airflow, Snowflake, DBT, AWS, and more. Seamless integrations to keep your workflows running smoothly
- [Enterprise](https://dagster.io/enterprise): Discover how Dagster Enterprise helps teams build reliable, scalable data workflows with modern orchestration, security, and real human support

## Solutions

### Industries

- [Finance](https://dagster.io/use-case/finance): Handling finance data is a headache? Dagster automates workflows, keeps your data reliable, and makes compliance easy. Check it out - you’ll love it
- [Software & Technology](https://dagster.io/use-case/software-technology): Treat data like code. Dagster gives software teams modern tools to plan, test, and deploy data pipelines with confidence and engineering best practices
- [Retail & E-commerce](https://dagster.io/use-case/retail-e-commerce): Retail data all over the place? Dagster automates inventory, sales, and ERP data workflows so everything stays in sync. No stress, just smooth operations
- [Life Sciences](https://dagster.io/use-case/life-sciences): Managing life sciences data is complex, but Dagster makes it easier. Automate research and lab data workflows to keep everything organized and compliant

### Workflows

- [ETL/ELT Pipelines](https://dagster.io/solutions/etl-elt-pipleines): Stop fighting broken ETL/ELT pipelines. Dagster automates, monitors, and orchestrates your data workflows for seamless, reliable data movement
- [AI & Machine Learning](https://dagster.io/solutions/ai): Stop wrestling with AI & ML pipelines. Dagster automates data prep, model training, and deployment - so you can focus on building, not fixing
- [Data Modernization](https://dagster.io/solutions/data-modernization): Dagster is the data modernization platform that simplifies migrations, automates workflows, and ensures data reliability at scale
- [Data Products](https://dagster.io/solutions/data-products): Dagster delivers real-time monitoring, alerts, and automated error handling to keep your data products reliable and running smoothly


## Pricing

- [Pricing](https://dagster.io/pricing): Explore Dagster’s transparent pricing. Choose a plan that fits your team’s needs - from local development to production-scale orchestration

## Company

- [About Us](https://dagster.io/company/about-us): Meet the team behind Dagster. We're on a mission to help data teams build, observe, and scale pipelines with software engineering best practices
- [Careers](https://dagster.io/company/careers): Join the Dagster team and help build the future of data engineering. We're hiring curious, driven people who care about developer experience
- [Partners](https://dagster.io/company/partners): Partner with Dagster to bring modern data orchestration to your clients. Explore technology and service partnerships that scale with your business
- [Brand Kit](https://dagster.io/brand): Download official Dagster logos, brand guidelines, and assets. Everything you need to represent Dagster accurately and consistently

## Resources

### Resources

- [Blog](https://dagster.io/blog): Read the latest from the Dagster team: insights, tutorials, and updates on data engineering, orchestration, and building better pipelines
- [Events](https://dagster.io/events): Stay up to date on Dagster events. From live product sessions to community meetups and webinars. Join the conversation and connect with our team
- [Docs](https://docs.dagster.io): Dagster's Documentation
- [Customer Stories](https://dagster.io/customers): See how data teams use Dagster to build reliable pipelines, improve visibility, and scale operations, real stories from real users
- [Community](https://dagster.io/community): Join the Dagster community - connect with other developers, contribute to open source, and be part of shaping the future of data engineering
- [University](https://dagster.io/university): Master Dagster with guided courses and hands-on lessons. Learn how to build, test, and scale data pipelines with confidence
- [GitHub](https://github.com/dagster-io/dagster): An orchestration platform for the development, production, and observation of data assets
- [Slack](https://join.slack.com/t/dagster/shared_invite/zt-3f3scycuq-8idTyWg0Y1CIFXC4VYrwYw): Slack is a new way to communicate with your team. It’s faster, better organized, and more secure than email

### How we Compare

- [Dagster vs Airflow](https://dagster.io/vs/dagster-vs-airflow): See how Dagster compares to Airflow in features, developer experience, and performance. Find out which orchestration tool fits your team’s workflow best
- [Dagster vs Prefect](https://dagster.io/vs/dagster-vs-prefect): Compare Dagster and Prefect side by side. Explore key differences in pipeline design, observability, and developer experience to find your best-fit tool
- [Dagster vs dbt Cloud](https://dagster.io/vs/dagster-vs-dbt-cloud): Compare Dagster and dbt Cloud to see how they support modern data teams. Explore key differences in orchestration, visibility, and developer experience
- [Dagster vs Azure Data Factory](https://dagster.io/vs/dagster-vs-azure-data-factory): Compare Dagster and Azure Data Factory across pipeline design, observability, and deployment. See which orchestration tool fits your team’s data workflows
- [Dagster vs AWS Step Functions](https://dagster.io/vs/dagster-vs-aws-step-functions): Compare Dagster and AWS Step Functions to see how they handle orchestration, observability, and developer workflows. Find the right tool for your data team

### Learning Center

- [Data Engineering](https://dagster.io/learn/data-engineering): Data engineering focuses on the practical application of data collection and processing techniques
- [Data Pipeline](https://dagster.io/learn/data-pipeline): A data pipeline automates the transfer of data between systems and its subsequent processing
- [Data Platform](https://dagster.io/learn/data-platform): A data platform is a system to manage, process, store, and analyze data from various sources
- [Data Quality](https://dagster.io/learn/data-quality): Data quality refers to the condition and usefulness of a set of values of qualitative or quantitative variables

### Featured resources

- [Forrester TEI Report](https://dagster.io/forrester-tei-report): Discover how Dagster drives ROI and engineering efficiency. Read the Forrester TEI report for data-backed insights on impact and cost savings
- [Scaling Data Teams eBook](https://dagster.io/how-to-scale-data-teams-ebook): Download Dagster's free eBook to learn how to build systems that scale with clarity, reliability, and confidence