​

Pipeline Orchestration

​

Why Pipeline Orchestration?

• Modularize workflows: Break complex ML workflows into manageable, reusable components
• Schedule and automate: Run training and inference jobs on schedules or triggers
• Track dependencies: Automatically manage task dependencies and execution order
• Monitor execution: Visualize pipeline runs, debug failures, and track metrics
• Scale workloads: Run computationally intensive tasks on Kubernetes or distributed systems

​

Orchestration Frameworks

​

Pipeline Architecture

​

Training Pipeline

1. Load Training Data: Download or prepare datasets (e.g., SST-2, SQL context data)
2. Train Model: Fine-tune models with specified configurations
3. Save Artifacts: Store model weights, tokenizers, and configs
4. Upload to Registry: Push trained models to W&B or other registries

​

Inference Pipeline

1. Load Inference Data: Prepare input data for predictions
2. Load Trained Model: Fetch model artifacts from registry
3. Run Inference: Generate predictions using loaded model
4. Save Results: Store predictions and evaluation metrics

​

Prerequisites

kind create cluster --name ml-in-production

export WANDB_PROJECT=your-project-name
export WANDB_API_KEY=your-api-key

k9s -A

​

Framework Comparison

​

Learning Objectives

• Deploy and configure Airflow, Kubeflow, and Dagster
• Build training pipelines that load data, train models, and upload artifacts
• Create inference pipelines that fetch models and generate predictions
• Compare orchestration frameworks for your ML use case
• Integrate with W&B for experiment tracking
• Run containerized workloads on Kubernetes

​

Module Resources

• Airflow Setup & DAGs
• Kubeflow Pipelines
• Dagster Assets & Checks
• Practice Exercises

​

Additional Reading

• Why data scientists shouldn’t need to know Kubernetes
• Orchestration for Machine Learning
• Awesome Open Source Workflow Engines
• How we Reduced our ML Training Costs by 78%