metaflow debug task

Create debugging notebooks and scripts for Metaflow tasks. The metaflow debug task command downloads a task’s code package, sets up the environment, and generates debugging scripts and Jupyter notebooks that allow you to inspect and debug task execution.

Synopsis

metaflow debug [OPTIONS] COMMAND [ARGS]...

Global Options

--quiet

flag

default:"false"

Suppress unnecessary messages

Commands

Create a new debugging notebook based on a task pathspec.

metaflow debug task [OPTIONS] PATHSPEC

Arguments

pathspec

string

required

Pathspec to a task in the format FlowName/RunId/StepName/TaskId. Can also be:

Flow only: FlowName (uses latest successful run’s end step)
Flow and Run: FlowName/RunId (uses end step)
Flow, Run, and Step: FlowName/RunId/StepName (if step has a single task)

Options

--metaflow-root-dir

path

required

Root directory where the code package and debug scripts/notebooks should be generated

--override-env

string

Name of the named environment to use for debugging the task. Overrides the Conda environment from this task

--override-env-from-pathspec

string

Pathspec to use as environment for debugging the task. Overrides the Conda environment from this task

--inspect

flag

default:"false"

If true, allows the user to inspect the state of the task after it has finished running

Description

The debug task command performs the following operations:

Resolves the task: Converts the pathspec to a specific task, using defaults for missing components
Sets up the environment: Creates or reuses the Conda environment used by the task
Downloads code package: Extracts the task’s code package to the specified directory
Generates escape trampolines: Creates import wrappers to enable proper debugging
Creates debug scripts: Generates a Python script that replicates the task’s execution context
Creates debug notebook: Generates a Jupyter notebook pre-configured with the task’s environment

Pathspec Resolution

The command intelligently resolves partial pathspecs:

MyFlow → MyFlow/<latest_run_id>/end/<task_id>
MyFlow/123 → MyFlow/123/end/<task_id>
MyFlow/123/my_step → MyFlow/123/my_step/<task_id> (if single task)
MyFlow/123/my_step/456 → Used as-is

Environment Override

You can override the task’s original environment using either:

--override-env: Specify a named environment alias
--override-env-from-pathspec: Use the environment from another task

This is useful when:

The original environment has issues
You want to test with a different set of dependencies
You need additional debugging tools not in the original environment

Inspect Mode

When --inspect is enabled, the generated debug script loads the task’s artifacts and final state, allowing you to:

Examine output artifacts
Inspect self attributes
Analyze the task’s final state without re-running the code

Examples

Debug a specific task:

metaflow debug task --metaflow-root-dir ./debug-session MyFlow/123/train/456

Output:

Python executable path for debugging is set to /path/to/conda/env/bin/python
Generating debug scripts for task MyFlow/123/train/456 using local backend
Debug scripts and notebook generated at ./debug-session; launch debug_notebook.ipynb to debug the task

Debug with latest run:

metaflow debug task --metaflow-root-dir ./debug MyFlow

This will automatically use the latest successful run’s end step. Debug with custom environment:

metaflow debug task \
  --metaflow-root-dir ./debug \
  --override-env myteam/debug-env:latest \
  MyFlow/123/train/456

Debug with environment from another task:

metaflow debug task \
  --metaflow-root-dir ./debug \
  --override-env-from-pathspec MyFlow/100/train/123 \
  MyFlow/200/train/456

Uses the environment from run 100’s train step to debug run 200’s train step. Inspect mode (examine finished task):

metaflow debug task \
  --metaflow-root-dir ./inspect-session \
  --inspect \
  MyFlow/123/train/456

The generated notebook will load the task’s artifacts and self attributes without re-executing the step code.

Generated Files

The command creates the following in --metaflow-root-dir:

debug-session/
├── debug_notebook.ipynb          # Jupyter notebook for debugging
├── FlowName_RunId_StepName_TaskId_debug.py  # Python debug script
├── _escape_trampolines/          # Import wrappers for debugging
├── __conda_python                # Python executable symlink
├── metaflow_setup.sh             # Environment setup script
└── [code package contents]       # Flow code and dependencies

Using the Debug Notebook

Once generated, launch Jupyter and open debug_notebook.ipynb:

cd debug-session
jupyter notebook debug_notebook.ipynb

The notebook includes:

Pre-loaded task context (self, inputs, artifacts)
Step code ready to execute
Access to all task metadata
Proper environment configuration

Using the Debug Script

Alternatively, run the debug script directly:

cd debug-session
python FlowName_RunId_StepName_TaskId_debug.py

Requirements

The task must have been executed remotely (must have a code package)
The task’s environment must be resolvable
Jupyter must be installed if using notebooks

Error Handling

Task has no code package:

Task MyFlow/123/start/456 does not have code-package. 
`debug task` command only supports tasks that were executed remotely.

Solution: Only remote tasks (run with --with batch or deployed to Argo/Step Functions) can be debugged. Step has multiple tasks:

Step MyFlow/123/train does not refer to a single task -- 
please specify the task unambiguously

Solution: Add the task ID to the pathspec: MyFlow/123/train/456 Environment not found:

Environment 'myenv' does not refer to a known environment

Solution: Ensure the environment exists and is accessible. Use metaflow environment get to fetch it first.

Advanced Usage

Kernel Configuration

The command automatically configures a Jupyter kernel with:

Correct Python executable
PYTHONPATH set to include escape trampolines
All environment variables from the task’s Metaflow version

You can verify the kernel configuration:

jupyter kernelspec list

Environment Variables

The generated metaflow_setup.sh script contains all necessary environment variables:

source ./debug-session/metaflow_setup.sh
python ./debug-session/FlowName_RunId_StepName_TaskId_debug.py

Debugging Decorators

The debug environment preserves all decorators from the original task:

@conda / @pypi environments
@resources settings
@timeout values
Custom decorators

However, only the environment is replicated locally. Resource constraints and timeouts are not enforced.

metaflow environment create - Create environments for debugging
metaflow environment show - Inspect task environments

Decorators

CLI Commands

metaflow debug task

Synopsis

Global Options

Commands