Pipeline Orchestration

Overview

Pipeline Orchestration lets you chain multiple workflow templates together into multi-step execution pipelines. Each pipeline defines a sequence of templates that run in order, with configurable failure handling (stop or continue) and real-time execution monitoring.

Pipelines are ideal for complex workflows like CI/CD, data processing, or multi-stage agent tasks that require sequential execution with visibility into each step’s status.

Architecture

Pipelines consist of three layers:

Workflow Templates

Reusable templates that define agent model, prompt, and parameters. Templates are created once and referenced by multiple pipelines.

Pipeline Definitions

Named sequences of workflow templates with failure handling rules. Stored in the pipelines table.

interface Pipeline {
  id: number
  name: string
  description: string | null
  steps: PipelineStep[]        // Template sequence
  use_count: number             // Execution counter
  last_used_at: number | null
  runs: {                       // Aggregated run stats
    total: number
    completed: number
    failed: number
    running: number
  }
}

Pipeline Runs

Execution instances that track progress through steps. Each run maintains state for all steps.

interface PipelineRun {
  id: number
  pipeline_id: number
  status: 'pending' | 'running' | 'completed' | 'failed' | 'cancelled'
  current_step: number          // 0-based index
  steps_snapshot: RunStepState[] // Execution state per step
  started_at: number | null
  completed_at: number | null
  triggered_by: string          // User or system
  created_at: number
}

Pipeline Steps

Each step references a workflow template and defines failure behavior:

interface PipelineStep {
  template_id: number        // References workflows table
  template_name?: string     // Denormalized for display
  on_failure: 'stop' | 'continue'
}

Stop vs Continue: on_failure: "stop" terminates the pipeline immediately. on_failure: "continue" marks the step as failed but proceeds to the next step. Use “continue” for non-critical steps like notifications.

Run State Tracking

Each run maintains detailed state for every step:

Step State
State Transitions
Example Run

interface RunStepState {
  step_index: number
  template_id: number
  template_name: string
  status: 'pending' | 'running' | 'completed' | 'failed' | 'skipped'
  spawn_id: string | null      // References spawned agent session
  started_at: number | null
  completed_at: number | null
  error: string | null
}

{
  "id": 42,
  "pipeline_id": 7,
  "status": "running",
  "current_step": 2,
  "steps_snapshot": [
    {
      "step_index": 0,
      "template_name": "Run Tests",
      "status": "completed",
      "spawn_id": "spawn-abc123",
      "started_at": 1709568000,
      "completed_at": 1709568120,
      "error": null
    },
    {
      "step_index": 1,
      "template_name": "Build Assets",
      "status": "completed",
      "spawn_id": "spawn-def456",
      "started_at": 1709568121,
      "completed_at": 1709568200,
      "error": null
    },
    {
      "step_index": 2,
      "template_name": "Deploy to Staging",
      "status": "running",
      "spawn_id": "spawn-ghi789",
      "started_at": 1709568201,
      "completed_at": null,
      "error": null
    }
  ],
  "started_at": 1709568000,
  "triggered_by": "admin"
}

UI Components

The Pipeline Tab provides a full management interface: Shows currently running pipelines at the top:

function ActiveRunCard({ run, onAdvance, onCancel }) {
  return (
    <div className="p-2.5 rounded-lg border border-amber-500/30 bg-amber-500/5">
      <div className="flex items-center justify-between mb-1.5">
        <div className="flex items-center gap-2">
          <span className="w-2 h-2 rounded-full bg-amber-500 animate-pulse" />
          <span className="text-xs font-medium">
            {run.pipeline_name} — Run #{run.id}
          </span>
        </div>
        <span className="text-2xs text-muted-foreground">
          Step {run.current_step + 1}/{run.steps_snapshot.length}
        </span>
      </div>
      <RunStepsViz steps={run.steps_snapshot} />
      <div className="flex gap-1 mt-2">
        <button onClick={() => onAdvance(run.id, true)}>Step Done</button>
        <button onClick={() => onAdvance(run.id, false)}>Step Failed</button>
        <button onClick={() => onCancel(run.id)}>Cancel</button>
      </div>
    </div>
  )
}

Pipeline Visualization

Displays step sequence with arrows:

function PipelineViz({ steps }) {
  return (
    <div className="flex items-center gap-1 overflow-x-auto py-1">
      {steps.map((s, i) => (
        <div key={i} className="flex items-center gap-1 shrink-0">
          <div className="flex flex-col items-center gap-0.5">
            <div className="px-2 py-1.5 rounded-md border bg-secondary">
              {s.template_name}
            </div>
            {s.on_failure === 'continue' && (
              <span className="text-2xs text-amber-400">continue on fail</span>
            )}
          </div>
          {i < steps.length - 1 && (
            <svg viewBox="0 0 20 12" className="w-5 h-3">
              <path d="M0 6h16M13 2l4 4-4 4" />
            </svg>
          )}
        </div>
      ))}
    </div>
  )
}

Run Steps Visualization

Shows execution progress with colored status dots:

function RunStepsViz({ steps }) {
  return (
    <div className="flex items-center gap-1">
      {steps.map((s, i) => (
        <div key={i} className="flex items-center gap-1">
          <span className={`w-2 h-2 rounded-full ${
            s.status === 'completed' ? 'bg-green-500' :
            s.status === 'running' ? 'bg-amber-500 animate-pulse' :
            s.status === 'failed' ? 'bg-red-500' :
            s.status === 'skipped' ? 'bg-gray-500' : 'bg-gray-600'
          }`} />
          <span className="text-2xs">{s.template_name}</span>
          {i < steps.length - 1 && <svg>...</svg>}
        </div>
      ))}
    </div>
  )
}

API Reference

Create Pipeline

curl -X POST "http://localhost:3000/api/pipelines" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Deploy to Production",
    "description": "Full deployment pipeline with testing",
    "steps": [
      { "template_id": 1, "on_failure": "stop" },
      { "template_id": 2, "on_failure": "stop" },
      { "template_id": 3, "on_failure": "continue" }
    ]
  }'

List Pipelines

curl "http://localhost:3000/api/pipelines" \
  -H "x-api-key: YOUR_API_KEY"

Start Pipeline Run

curl -X POST "http://localhost:3000/api/pipelines/run" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "start",
    "pipeline_id": 5
  }'

Advance Run Step

curl -X POST "http://localhost:3000/api/pipelines/run" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "advance",
    "run_id": 42,
    "success": true
  }'

Cancel Pipeline Run

curl -X POST "http://localhost:3000/api/pipelines/run" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "cancel",
    "run_id": 42
  }'

Workflow Templates

Pipelines reference workflow templates. Create templates first:

curl -X POST "http://localhost:3000/api/workflows" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Run Tests",
    "model": "claude-3-5-sonnet-20241022",
    "prompt": "Run the test suite and report results",
    "parameters": {
      "temperature": 0.2,
      "max_tokens": 4000
    }
  }'

Best Practices

Pipeline Design
Failure Handling
Monitoring

Keep pipelines focused: 3-7 steps is optimal
Use descriptive names: “Deploy Production” not “Pipeline 1”
Group related workflows: Create separate pipelines for different domains
Test in isolation: Validate each template before adding to pipeline

Critical steps: Use on_failure: "stop" for tests, builds, deploys
Non-critical steps: Use on_failure: "continue" for notifications, cleanup
Order matters: Place critical validation steps early
Add recovery steps: Include rollback templates after risky operations

Watch the Active Run Banner for in-progress pipelines
Check run history in the expanded pipeline card
Use GET /api/pipelines/run?limit=N to fetch recent runs
Monitor spawn_id to track which agent executed each step

Example Pipelines

{
  "name": "CI/CD Pipeline",
  "description": "Full continuous integration and deployment",
  "steps": [
    { "template_id": 1, "template_name": "Run Linter", "on_failure": "stop" },
    { "template_id": 2, "template_name": "Run Unit Tests", "on_failure": "stop" },
    { "template_id": 3, "template_name": "Build Docker Image", "on_failure": "stop" },
    { "template_id": 4, "template_name": "Deploy to Staging", "on_failure": "stop" },
    { "template_id": 5, "template_name": "Run E2E Tests", "on_failure": "stop" },
    { "template_id": 6, "template_name": "Notify Team", "on_failure": "continue" },
    { "template_id": 7, "template_name": "Deploy to Production", "on_failure": "stop" }
  ]
}

Troubleshooting

Pipeline won't start

Verify all referenced template IDs exist
Check that pipeline has at least 2 steps
Ensure user has operator role or higher

Steps not advancing

Use the UI “Step Done” / “Step Failed” buttons to manually advance
Check that spawned agent sessions are completing
Verify spawn_id is being set correctly

Run stuck in 'running' state

Use POST /api/pipelines/run with action: "cancel" to force stop
Check for orphaned agent sessions
Review Mission Control logs for errors

Agent Management - Managing agent execution
Task Board - Task management and workflows
Background Automation - Automated scheduling

Get Started

Core Features

Integrations

Deployment

Advanced

Overview

Architecture

Pipeline Steps

Run State Tracking

UI Components

Active Run Banner

Pipeline Visualization

Run Steps Visualization

API Reference

Create Pipeline

List Pipelines

Start Pipeline Run

Advance Run Step

Cancel Pipeline Run

Workflow Templates

Best Practices

Example Pipelines

Troubleshooting

Get Started

Core Features

Integrations

Deployment

Advanced

​Overview

​Architecture

​Pipeline Steps

​Run State Tracking

​UI Components

​Active Run Banner

​Pipeline Visualization

​Run Steps Visualization

​API Reference

​Create Pipeline

​List Pipelines

​Start Pipeline Run

​Advance Run Step

​Cancel Pipeline Run

​Workflow Templates

​Best Practices

​Example Pipelines

​Troubleshooting

​Related

Overview

Architecture

Pipeline Steps

Run State Tracking

UI Components

Active Run Banner

Pipeline Visualization

Run Steps Visualization

API Reference

Create Pipeline

List Pipelines

Start Pipeline Run

Advance Run Step

Cancel Pipeline Run

Workflow Templates

Best Practices

Example Pipelines

Troubleshooting

Related