@named_env

The @named_env decorator allows you to reference a pre-resolved, named environment for a specific step. Named environments are like Docker tags - they’re aliases for fully resolved environments that can be shared across flows and teams.

Overview

Use @named_env to reference environments that have been resolved and named separately. This enables environment reuse, sharing, and consistent versioning across projects.

from metaflow import FlowSpec, step, named_env

class MyFlow(FlowSpec):
    
    @named_env(name="team/ml-base-env:v2")
    @step
    def start(self):
        import pandas as pd
        import numpy as np
        # Uses the pre-resolved ml-base-env
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

When to Use

Shared environments: Use environments resolved and maintained by your team
Versioned environments: Pin to specific environment versions (e.g., v1, v2)
Reuse across flows: Multiple flows can use the same named environment
Reproduce past runs: Reference environments from previous executions using pathspecs
Latest environments: Use mutable tags like :latest for always-current dependencies
Environment isolation: Separate environment management from flow code

Parameters

name

str

default:"None"

Reference to a named environment alias. The environment must have been previously resolved and aliased.Name format: namespace/environment-name[:tag]

The tag is optional; defaults to :latest if not specified
Tags can contain environment variables using @{VAR_NAME} syntax
Mutable tags: :latest, :candidate, :stable can be updated
Immutable tags: All other tags are permanent once created

# Use specific version
@named_env(name="team/ml-env:v1")

# Use latest version (mutable)
@named_env(name="team/ml-env:latest")
@named_env(name="team/ml-env")  # Same as :latest

# Use environment variable
@named_env(name="team/ml-env:@{ENV_VERSION}")

Note: Cannot specify both name and pathspec in the same decorator.

pathspec

str

default:"None"

Reference the environment from a previously executed step using its pathspec.Pathspec format: FlowName/RunId/StepName

Can use environment variables with @{VAR_NAME} syntax
Useful for reproducing exact environments from past runs
Automatically fetches the environment used by that step

# Use environment from a specific run
@named_env(pathspec="MyFlow/123/train")

# Use environment variable for run ID
@named_env(pathspec="MyFlow/@{RUN_ID}/train")

Note: Cannot specify both name and pathspec in the same decorator.

fetch_at_exec

bool

default:"False"

If True, fetches the environment when the task executes (at runtime) rather than at deploy time or flow start.This is useful for:

Using the :latest version at execution time, not deployment time
Dynamic environments that update frequently
Production deployments that should pick up environment updates

Requires either name or pathspec to be specified.

@named_env(name="team/ml-env:latest", fetch_at_exec=True)
@step
def train(self):
    # Always uses the latest environment when task runs
    # Not the latest when flow was deployed
    pass

disabled

bool

default:"False"

If set to True, disables the managed environment for this step and uses the external environment instead.

@named_env(disabled=True)
@step
def use_system_env(self):
    # Runs in system environment
    pass

Usage Examples

Using a Named Environment

Reference a pre-resolved environment:

from metaflow import FlowSpec, step, named_env

class MLFlow(FlowSpec):
    
    @named_env(name="ml-team/base-env:v1")
    @step
    def start(self):
        import pandas as pd
        import numpy as np
        import sklearn
        # All packages from ml-team/base-env:v1
        self.next(self.train)
    
    @named_env(name="ml-team/gpu-env:v2")
    @step
    def train(self):
        import torch
        # Uses different environment with GPU support
        self.next(self.end)
    
    @step
    def end(self):
        pass

Creating a Named Environment

Resolve and name an environment using CLI:

# From requirements.txt
metaflow environment resolve \
  --python ">=3.8,<3.9" \
  --alias team/ml-env:v1 \
  -r requirements.txt

# From environment.yml
metaflow environment resolve \
  --python ">=3.8,<3.9" \
  --alias team/ml-env:v1 \
  -f environment.yml

# From flow environment
python myflow.py --environment=conda environment resolve \
  --alias team/ml-env:v1

Or from within a flow:

# First run to create the environment
@conda(libraries={"pandas": "1.5.0", "numpy": "1.21.5"})
@step
def prepare_env(self):
    pass

# Then alias it
# metaflow environment alias --pathspec MyFlow/123/prepare_env --alias team/ml-env:v1

Using Latest Version

Use mutable :latest tag for always-current environments:

@named_env(name="team/ml-env:latest")
@step
def train(self):
    # Uses the latest version of team/ml-env
    # Each time you resolve the environment, :latest updates
    import pandas as pd
    self.next(self.end)

# Or omit :latest (it's the default)
@named_env(name="team/ml-env")
@step
def train_v2(self):
    # Same as team/ml-env:latest
    pass

Fetch at Execution Time

Always use the latest environment at runtime:

from metaflow import FlowSpec, step, named_env

class ProductionFlow(FlowSpec):
    
    @named_env(
        name="prod/data-pipeline:latest",
        fetch_at_exec=True
    )
    @step
    def process(self):
        # Uses the latest environment *when this task runs*
        # Not the latest when flow was deployed
        import pandas as pd
        self.next(self.end)
    
    @step
    def end(self):
        pass

Without fetch_at_exec, the environment is fetched when:

You run the flow locally
You deploy to a scheduler (Argo/Airflow/Step Functions)

With fetch_at_exec=True, the environment is fetched when:

The specific task starts executing

Using Pathspecs

Reuse the environment from a previous run:

@named_env(pathspec="TrainingFlow/142/train")
@step
def reproduce_training(self):
    # Uses the exact environment from TrainingFlow run 142, step train
    import torch
    # Train with identical dependencies
    self.next(self.end)

With environment variables:

import os

class ReproducibleFlow(FlowSpec):
    
    @named_env(pathspec="TrainingFlow/@{ORIGINAL_RUN_ID}/train")
    @step
    def start(self):
        # Use environment variable to specify run
        pass

Run with:

ORIGINAL_RUN_ID=142 python flow.py --environment=conda run

Extending Named Environments

Add packages to a named environment:

from metaflow import FlowSpec, step, named_env, conda, pypi

class ExtendedFlow(FlowSpec):
    
    @named_env(name="team/base-env:v1")
    @conda(libraries={"xgboost": "1.7.0"})
    @step
    def train_xgboost(self):
        # base-env:v1 + xgboost
        import xgboost as xgb
        self.next(self.train_pytorch)
    
    @named_env(name="team/base-env:v1")
    @pypi(packages={"fastapi": "0.95.0"})
    @step
    def train_pytorch(self):
        # base-env:v1 + fastapi
        from fastapi import FastAPI
        self.next(self.end)
    
    @step
    def end(self):
        pass

Environment Naming Convention

Recommended format: team/project/env-name:tag

# Good naming conventions
@named_env(name="ml-team/recommendation/base:v1")
@named_env(name="data-eng/etl/spark-env:v2")
@named_env(name="research/nlp/transformers:latest")

# Mutable tags (can be updated)
@named_env(name="team/env:latest")
@named_env(name="team/env:candidate")
@named_env(name="team/env:stable")

# Immutable tags (permanent)
@named_env(name="team/env:v1")
@named_env(name="team/env:2023-03-15")
@named_env(name="team/env:prod-release-1.2")

Combining with Flow-Level Decorator

Step-level @named_env can override @named_env_base:

from metaflow import FlowSpec, step, named_env_base, named_env

@named_env_base(name="team/default-env:v1")
class MyFlow(FlowSpec):
    
    @step
    def use_base(self):
        # Uses team/default-env:v1
        self.next(self.use_different)
    
    @named_env(name="team/special-env:v2")
    @step
    def use_different(self):
        # Overrides base, uses team/special-env:v2
        self.next(self.end)
    
    @step
    def end(self):
        pass

Viewing Named Environments

List Available Environments

# Get details about a named environment
metaflow environment get team/ml-env:v1

# Show environment for a specific step
metaflow environment show --pathspec MyFlow/142/train

Inspect Environment Contents

# See all packages in named environment
metaflow environment show team/ml-env:v1

Interaction with Other Decorators

Extending with @conda and @pypi

Add packages to a named environment:

@named_env(name="team/base:v1")
@conda(libraries={"pytorch": "2.0.0"})
@pypi(packages={"transformers": "4.28.0"})
@step
def train(self):
    # base:v1 + pytorch (Conda) + transformers (PyPI)
    import torch
    from transformers import AutoModel
    self.next(self.end)

With @named_env_base

Flow-level base with step-level overrides:

@named_env_base(name="team/default:v1")
class MyFlow(FlowSpec):
    
    @step
    def use_default(self):
        # Uses team/default:v1
        pass
    
    @named_env(name="team/special:v2")
    @step
    def use_special(self):
        # Overrides, uses team/special:v2
        pass

Cannot Mix with Definition Decorators

At flow level, cannot combine with @conda_base or @pypi_base:

# This is an error:
@conda_base(libraries={"pandas": "1.5.0"})
@named_env_base(name="team/env")  # ERROR
class BadFlow(FlowSpec):
    pass

At step level, @named_env can be extended:

# This is OK:
@named_env(name="team/env")
@conda(libraries={"pandas": "1.5.0"})
@step
def process(self):
    # Extends team/env with pandas
    pass

Mutable vs Immutable Tags

Mutable Tags

These can be updated to point to different environments:

:latest - Points to the most recently resolved environment
:candidate - For testing/staging environments
:stable - For production-ready environments

# Team can update what :latest points to
@named_env(name="team/ml-env:latest")

Immutable Tags

Once created, these always point to the same environment:

:v1, :v2, :v2.1 - Version tags
:2023-03-15 - Date tags
:prod-release-1.0 - Release tags

# Always uses the exact same environment
@named_env(name="team/ml-env:v1")

Named environments are automatically shared:

Resolved once: One team member resolves and names the environment
Cached in cloud: Stored in S3/GCS/Azure
Used by all: Other team members automatically fetch and use it

# Team member 1: Create and name the environment
# (Runs once)
metaflow environment resolve \
  --alias team/shared-env:v1 \
  -r requirements.txt

# Team members 2, 3, 4, ...: Use the environment
# (Automatically fetches from cloud storage)
@named_env(name="team/shared-env:v1")
@step
def process(self):
    # Uses the shared environment
    pass

@named_env_base - Flow-level named environment reference
@conda - Step-level Conda dependencies
@conda_base - Flow-level Conda dependencies
@pypi - Step-level PyPI dependencies
@pypi_base - Flow-level PyPI dependencies

Requirements

Must use --environment=conda when running the flow
Named environment must exist (be previously resolved and aliased)
Requires Conda/Mamba/Micromamba installed

Notes

Named environments are immutable (except for mutable tags)
Environment contents are locked and reproducible
Can be shared across teams and projects
Environments are cached in cloud storage (S3/GCS/Azure)
Pathspecs provide exact reproducibility of past runs
Fetch-at-exec useful for production deployments with mutable tags

Decorators

CLI Commands

Overview

When to Use

Parameters

Usage Examples

Using a Named Environment

Creating a Named Environment

Using Latest Version

Fetch at Execution Time

Using Pathspecs

Extending Named Environments

Environment Naming Convention

Combining with Flow-Level Decorator

Viewing Named Environments

List Available Environments

Inspect Environment Contents

Interaction with Other Decorators

Extending with @conda and @pypi

With @named_env_base

Cannot Mix with Definition Decorators

Mutable vs Immutable Tags

Mutable Tags

Immutable Tags

Requirements

Notes

Build docs developers (and LLMs) love

Decorators

CLI Commands

​Overview

​When to Use

​Parameters

​Usage Examples

​Using a Named Environment

​Creating a Named Environment

​Using Latest Version

​Fetch at Execution Time

​Using Pathspecs

​Extending Named Environments

​Environment Naming Convention

​Combining with Flow-Level Decorator

​Viewing Named Environments

​List Available Environments

​Inspect Environment Contents

​Interaction with Other Decorators

​Extending with @conda and @pypi

​With @named_env_base

​Cannot Mix with Definition Decorators

​Mutable vs Immutable Tags

​Mutable Tags

​Immutable Tags

​Environment Sharing

​Related Decorators

​Requirements

​Notes

Build docs developers (and LLMs) love

Overview

When to Use

Parameters

Usage Examples

Using a Named Environment

Creating a Named Environment

Using Latest Version

Fetch at Execution Time

Using Pathspecs

Extending Named Environments

Environment Naming Convention

Combining with Flow-Level Decorator

Viewing Named Environments

List Available Environments

Inspect Environment Contents

Interaction with Other Decorators

Extending with @conda and @pypi

With @named_env_base

Cannot Mix with Definition Decorators

Mutable vs Immutable Tags

Mutable Tags

Immutable Tags

Environment Sharing

Related Decorators

Requirements

Notes