Skip to main content

​
Metaflow Dagster

Deploy and run Metaflow flows as Dagster jobs with full observability

Get StartedView on GitHub

​
Why Metaflow Dagster?

metaflow-dagster generates a self-contained Dagster definitions file from any Metaflow flow, letting you schedule, monitor, and launch your pipelines through Dagster while keeping all your existing Metaflow code unchanged.

Keep Your Code

No modifications to your existing Metaflow flows required

Full Observability

View runs, logs, and artifacts in the Dagster UI

All Graph Shapes

Linear, branching, foreach, and conditional flows all supported

Event-Driven

Built-in sensors and scheduling capabilities

​
Quick Example

# Generate Dagster definitions from your Metaflow flow
python my_flow.py dagster create dagster_defs.py

# Launch the Dagster UI
dagster dev -f dagster_defs.py
Your Metaflow flow is now running in Dagster with full visibility into execution, timing, and artifacts.

​
Key Features

Parameter Forwarding

Metaflow Parameters automatically become Dagster Config

Retry & Timeout

Built-in retry policies and timeout handling

Resume Failed Runs

Skip completed steps and resume from failures

Resource Hints

Forward CPU, memory, and GPU hints to compute backends

Scheduling

Automatic schedule definitions from @schedule decorator

Sensors

Event-driven triggers with @trigger decorators

​
Get Started

Installation

Install via pip or from source

Quickstart

Your first Dagster job in 5 minutes

Examples

Real-world flow patterns