dagster-daemon CLI

The dagster-daemon command-line interface manages the Dagster daemon process, which is responsible for executing schedules, sensors, backfills, and other background tasks.

Installation

The daemon is included with the main Dagster package:

pip install dagster

For the full webserver functionality, install:

pip install dagster-webserver

Overview

The Dagster daemon is a long-running process that must be running for:

Schedules to trigger runs automatically
Sensors to evaluate and launch runs
Backfills to execute partitioned runs
Run queue management for controlling concurrency
Auto-materializing assets (asset sensors)
Run retries and failure handling

The daemon is automatically started when using dagster dev. For production deployments, you’ll need to run it separately.

Global Options

--version

flag

Show the installed Dagster version

--help

flag

Show help message and exit

dagster-daemon run

Start the Dagster daemon process.

Usage

dagster-daemon run [OPTIONS]

Options

--code-server-log-level

choice

default:"warning"

Set the log level for code servers spun up by the daemonChoices: critical, error, warning, info, debug

--log-level

choice

default:"info"

Set the log level for the daemon processChoices: critical, error, warning, info, debugEnvironment Variable: DAGSTER_DAEMON_LOG_LEVEL

--log-format

choice

default:"colored"

Format of the log output from the daemonChoices: colored, json, rich

Workspace Options

-w, --workspace

path

Path to workspace file. Can be specified multiple times

-f, --python-file

path

Path to a Python file containing Dagster definitions

-m, --module-name

string

Python module containing Dagster definitions

-d, --working-directory

path

Working directory to use when loading the repository

-a, --attribute

string

Attribute within the module or file that defines the Definitions object

Example

# Start daemon with default settings
dagster-daemon run

Output

2024-01-15 10:30:45 -0800 - dagster.daemon - INFO - Starting Dagster daemon
2024-01-15 10:30:45 -0800 - dagster.daemon.SchedulerDaemon - INFO - Daemon started
2024-01-15 10:30:45 -0800 - dagster.daemon.SensorDaemon - INFO - Daemon started  
2024-01-15 10:30:45 -0800 - dagster.daemon.BackfillDaemon - INFO - Daemon started
2024-01-15 10:30:45 -0800 - dagster.daemon.AssetDaemon - INFO - Daemon started

Daemon Types

The daemon automatically runs multiple sub-daemons:

SchedulerDaemon - Evaluates schedules and launches runs
SensorDaemon - Evaluates sensors and launches runs
BackfillDaemon - Executes backfill jobs
AssetDaemon - Auto-materializes assets based on policies
RunMonitoringDaemon - Monitors run status
RunTimeoutDaemon - Handles run timeouts
RunQueueDaemon - Manages run queue and concurrency

dagster-daemon liveness-check

Check for recent heartbeats from the daemon to verify it’s running.

Usage

dagster-daemon liveness-check

Example

# Check if daemon is alive
dagster-daemon liveness-check

Output

Daemon live

Exit Codes

0 - Daemon is running
1 - Daemon is not running or unhealthy

Use Cases

Health Check in Docker:

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD dagster-daemon liveness-check || exit 1

Kubernetes Liveness Probe:

livenessProbe:
  exec:
    command:
    - dagster-daemon
    - liveness-check
  initialDelaySeconds: 30
  periodSeconds: 30

Monitoring Script:

#!/bin/bash
if ! dagster-daemon liveness-check; then
  echo "Daemon is down, restarting..."
  systemctl restart dagster-daemon
fi

dagster-daemon wipe

Wipe all heartbeat data from storage.

This command removes heartbeat metadata. It does not affect runs, schedules, or other data.

Usage

dagster-daemon wipe

Example

# Clean up heartbeat storage
dagster-daemon wipe

Output

Daemon heartbeats wiped

When to Use

After changing daemon configuration
When troubleshooting daemon issues
After instance migration
To clear stale heartbeat data

dagster-daemon debug

Daemon debugging utilities.

dagster-daemon debug heartbeat

Read and write a test heartbeat to verify storage connectivity.

dagster-daemon debug heartbeat

dagster-daemon debug heartbeat-dump

Log all current heartbeat statuses.

dagster-daemon debug heartbeat-dump

Output

DaemonStatus(
  daemon_type=SCHEDULER,
  healthy=True,
  last_heartbeat_time=2024-01-15T18:30:45.123456,
  last_heartbeat_errors=[]
)
DaemonStatus(
  daemon_type=SENSOR,
  healthy=True,
  last_heartbeat_time=2024-01-15T18:30:45.234567,
  last_heartbeat_errors=[]
)

When to Use Debug Commands

Diagnosing daemon connectivity issues
Verifying storage backend is working
Troubleshooting heartbeat problems
Checking daemon health status

Production Deployment

Running as a Service

Systemd Service (Linux):

# /etc/systemd/system/dagster-daemon.service
[Unit]
Description=Dagster Daemon
After=network.target

[Service]
Type=simple
User=dagster
Group=dagster
WorkingDirectory=/opt/dagster
Environment="DAGSTER_HOME=/opt/dagster/dagster_home"
ExecStart=/opt/dagster/venv/bin/dagster-daemon run
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Start the service:

sudo systemctl daemon-reload
sudo systemctl enable dagster-daemon
sudo systemctl start dagster-daemon
sudo systemctl status dagster-daemon

Docker Deployment

Dockerfile:

FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy code
COPY . .

# Set environment
ENV DAGSTER_HOME=/app/dagster_home

# Run daemon
CMD ["dagster-daemon", "run"]

docker-compose.yml:

version: '3.8'
services:
  dagster-daemon:
    build: .
    container_name: dagster-daemon
    restart: always
    environment:
      - DAGSTER_HOME=/opt/dagster/dagster_home
      - DAGSTER_POSTGRES_HOST=postgres
      - DAGSTER_POSTGRES_DB=dagster
      - DAGSTER_POSTGRES_USER=dagster
      - DAGSTER_POSTGRES_PASSWORD=dagster
    volumes:
      - ./dagster_home:/opt/dagster/dagster_home
    depends_on:
      - postgres
    command: dagster-daemon run

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dagster-daemon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dagster-daemon
  template:
    metadata:
      labels:
        app: dagster-daemon
    spec:
      containers:
      - name: dagster-daemon
        image: your-registry/dagster-daemon:latest
        command: ["dagster-daemon", "run"]
        env:
        - name: DAGSTER_HOME
          value: "/opt/dagster/dagster_home"
        - name: DAGSTER_POSTGRES_HOST
          value: "postgres-service"
        volumeMounts:
        - name: dagster-home
          mountPath: /opt/dagster/dagster_home
        livenessProbe:
          exec:
            command:
            - dagster-daemon
            - liveness-check
          initialDelaySeconds: 30
          periodSeconds: 30
      volumes:
      - name: dagster-home
        persistentVolumeClaim:
          claimName: dagster-home-pvc

Configuration

dagster.yaml

Configure daemon behavior in your dagster.yaml file:

# dagster.yaml
run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 10
    tag_concurrency_limits:
      - key: "database"
        value: "postgres"
        limit: 2

run_monitoring:
  enabled: true
  start_timeout_seconds: 180
  cancel_timeout_seconds: 180
  max_runtime_seconds: 3600

schedule_executor:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler
  
sensor_executor:
  module: dagster.core.scheduler.sensor
  class: DagsterDaemonSensorExecutor

auto_materialize:
  enabled: true
  run_tags:
    source: "auto_materialize"
  max_materializations_per_minute: 10

Environment Variables

DAGSTER_HOME

path

Directory containing the dagster.yaml instance configuration

DAGSTER_DAEMON_LOG_LEVEL

string

Default log level for the daemon (overrides —log-level)

DAGSTER_DAEMON_HEARTBEAT_TOLERANCE

integer

Seconds before considering a daemon unhealthy (default: 300)

Monitoring

Logs

Daemon logs are crucial for debugging. Common log locations: Development:

# Logs print to stdout when running interactively
dagster-daemon run

Systemd:

# View systemd logs
journalctl -u dagster-daemon -f

Docker:

# View container logs
docker logs -f dagster-daemon

Metrics

Important metrics to monitor:

Heartbeat Health - Use liveness-check regularly
Schedule Success Rate - Track successful schedule evaluations
Sensor Lag - Time between sensor evaluations
Queue Depth - Number of queued runs
Daemon Uptime - Continuous operation time

Alerts

Set up alerts for:

Daemon process down
Heartbeat failures
Schedule evaluation errors
Sensor evaluation errors
Run queue backlog

Troubleshooting

Daemon Not Starting

Check instance configuration:

dagster instance info

Verify workspace loads:

python -c "from my_project import definitions"

Check permissions:

ls -la $DAGSTER_HOME

Schedules Not Running

Verify daemon is running:
```
dagster-daemon liveness-check
```
Check schedule is started:
```
dagster schedule list --running
```
Review daemon logs for errors:
```
journalctl -u dagster-daemon -n 100
```
Verify schedule cron expression: Test your cron at crontab.guru

Sensors Not Triggering

Check sensor evaluation logs:
```
dagster-daemon run --log-level debug
```
Verify sensor is enabled: Check the UI or run dagster sensor list
Check sensor evaluation time: Sensors evaluate every 30 seconds by default

Memory Issues

If the daemon is consuming too much memory:

Reduce concurrent evaluations:

# dagster.yaml
sensor_executor:
  config:
    num_workers: 2  # Default is 4

Limit run queue size:

run_coordinator:
  config:
    max_concurrent_runs: 5

Monitor and restart periodically:

# Add to crontab
0 2 * * * systemctl restart dagster-daemon

High CPU Usage

Reduce sensor evaluation frequency:

@sensor(
    job=my_job,
    minimum_interval_seconds=300  # Evaluate every 5 minutes
)
def my_sensor():
    ...

Optimize sensor code: Avoid expensive computations in sensor evaluation
Check for sensor errors: Failing sensors may retry aggressively

Storage Connection Issues

Test storage connectivity:

dagster-daemon debug heartbeat

Verify database is accessible:

psql -h localhost -U dagster -d dagster -c "SELECT 1;"

Check connection pool settings:

# dagster.yaml
storage:
  postgres:
    postgres_url: postgresql://user:pass@host:5432/dagster
    pool_size: 10
    max_overflow: 20

dagster CLI - Main Dagster CLI reference
Instance Configuration - Configure your instance
Schedules - Working with schedules
Sensors - Working with sensors
Run Coordinator - Configure run execution

Overview

Definitions

Execution

Types & Config

CLI

​dagster-daemon CLI

​Installation

​Overview

​Global Options

​dagster-daemon run

​Usage

​Options

​Workspace Options

​Example

​Output

​Daemon Types

​dagster-daemon liveness-check

​Usage

​Example

​Output

​Exit Codes

​Use Cases

​dagster-daemon wipe

​Usage

​Example

​Output

​When to Use

​dagster-daemon debug

​dagster-daemon debug heartbeat

​dagster-daemon debug heartbeat-dump

​Output

​When to Use Debug Commands

​Production Deployment

​Running as a Service

​Docker Deployment

​Kubernetes Deployment

​Configuration

​dagster.yaml

​Environment Variables

​Monitoring

​Logs

​Metrics

​Alerts

​Troubleshooting

​Daemon Not Starting

​Schedules Not Running

​Sensors Not Triggering

​Memory Issues

​High CPU Usage

​Storage Connection Issues

​Related

Build docs developers (and LLMs) love

dagster-daemon CLI

Installation

Overview

Global Options

dagster-daemon run

Usage

Options

Workspace Options

Example

Output

Daemon Types

dagster-daemon liveness-check

Usage

Example

Output

Exit Codes

Use Cases

dagster-daemon wipe

Usage

Example

Output

When to Use

dagster-daemon debug

dagster-daemon debug heartbeat

dagster-daemon debug heartbeat-dump

Output

When to Use Debug Commands

Production Deployment

Running as a Service

Docker Deployment

Kubernetes Deployment

Configuration

dagster.yaml

Environment Variables

Monitoring

Logs

Metrics

Alerts

Troubleshooting

Daemon Not Starting

Schedules Not Running

Sensors Not Triggering

Memory Issues

High CPU Usage

Storage Connection Issues

Related