@named_env_base

The @named_env_base decorator allows you to reference a pre-resolved, named environment for all steps in your flow by default. Named environments are like Docker tags - they’re aliases for fully resolved environments that can be shared across flows and teams.

Overview

Use @named_env_base to set a default named environment for your entire flow. Individual steps can override this with their own @named_env decorator or extend it with @conda or @pypi.

from metaflow import FlowSpec, step, named_env_base, conda

@named_env_base(name="team/ml-base-env:v2")
class MyFlow(FlowSpec):
    
    @step
    def start(self):
        import pandas as pd
        import numpy as np
        # Uses ml-base-env:v2
        self.next(self.train)
    
    @conda(libraries={"xgboost": "1.7.0"})
    @step
    def train(self):
        import pandas as pd
        import xgboost as xgb
        # ml-base-env:v2 + xgboost
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

When to Use

Team-standard environments: Use your team’s standard environment across all steps
Reduce repetition: Avoid specifying the same environment on every step
Versioned base: Pin all steps to a specific environment version
Shared environments: Use environments maintained by your team or organization
Production stability: Lock production flows to tested environment versions
Latest by default: Use :latest tag to always use current environment

Parameters

name

str

default:"None"

Reference to a named environment alias. The environment must have been previously resolved and aliased.Name format: namespace/environment-name[:tag]

The tag is optional; defaults to :latest if not specified
Tags can contain environment variables using @{VAR_NAME} syntax
Mutable tags: :latest, :candidate, :stable can be updated
Immutable tags: All other tags are permanent once created

# Use specific version
@named_env_base(name="team/ml-env:v1")

# Use latest version (mutable)
@named_env_base(name="team/ml-env:latest")
@named_env_base(name="team/ml-env")  # Same as :latest

# Use environment variable
@named_env_base(name="team/ml-env:@{ENV_VERSION}")

Note: Cannot specify both name and pathspec in the same decorator.

pathspec

str

default:"None"

Reference the environment from a previously executed step using its pathspec.Pathspec format: FlowName/RunId/StepName

Can use environment variables with @{VAR_NAME} syntax
Useful for reproducing exact environments from past runs
Automatically fetches the environment used by that step

# Use environment from a specific run
@named_env_base(pathspec="MyFlow/123/train")

# Use environment variable for run ID
@named_env_base(pathspec="MyFlow/@{RUN_ID}/train")

Note: Cannot specify both name and pathspec in the same decorator.

fetch_at_exec

bool

default:"False"

If True, fetches the environment when each task executes (at runtime) rather than at deploy time or flow start.This is useful for:

Using the :latest version at execution time, not deployment time
Dynamic environments that update frequently
Production deployments that should pick up environment updates

Requires either name or pathspec to be specified.

@named_env_base(name="team/ml-env:latest", fetch_at_exec=True)
class ProductionFlow(FlowSpec):
    @step
    def process(self):
        # Always uses the latest environment when task runs
        pass

disabled

bool

default:"False"

If set to True, disables the managed environment for all steps by default. Individual steps can still enable environments using @named_env, @conda, or @pypi.

@named_env_base(disabled=True)
class SelectiveFlow(FlowSpec):
    @step
    def use_system(self):
        # Runs in system environment
        pass
    
    @named_env(name="team/special-env")
    @step
    def use_managed(self):
        # Explicitly enables named environment
        pass

Usage Examples

Basic Flow-Wide Named Environment

Set a named environment for all steps:

from metaflow import FlowSpec, step, named_env_base

@named_env_base(name="ml-team/production-env:v1")
class ProductionPipeline(FlowSpec):
    
    @step
    def start(self):
        import pandas as pd
        import numpy as np
        # Uses production-env:v1
        self.next(self.process)
    
    @step
    def process(self):
        import sklearn
        # Still uses production-env:v1
        self.next(self.end)
    
    @step
    def end(self):
        # Still uses production-env:v1
        pass

Extending Base Environment

Add step-specific packages to the base environment:

@named_env_base(name="team/base-ml:v2")
class ExtendedFlow(FlowSpec):
    
    @step
    def start(self):
        import pandas as pd
        # Only base-ml:v2 packages
        self.next(self.train_xgb, self.train_pytorch)
    
    @conda(libraries={"xgboost": "1.7.0"})
    @step
    def train_xgb(self):
        import pandas as pd
        import xgboost as xgb
        # base-ml:v2 + xgboost
        self.next(self.join)
    
    @conda(libraries={"pytorch": "2.0.0"})
    @step
    def train_pytorch(self):
        import pandas as pd
        import torch
        # base-ml:v2 + pytorch
        self.next(self.join)
    
    @step
    def join(self, inputs):
        self.next(self.end)
    
    @step
    def end(self):
        pass

Overriding Base Environment

Use a different named environment for specific steps:

@named_env_base(name="team/default-env:v1")
class OverrideFlow(FlowSpec):
    
    @step
    def start(self):
        # Uses team/default-env:v1
        self.next(self.special_step)
    
    @named_env(name="team/gpu-env:v2")
    @step
    def special_step(self):
        import torch
        # Overrides base, uses team/gpu-env:v2
        self.next(self.end)
    
    @step
    def end(self):
        # Back to team/default-env:v1
        pass

Using Latest Environment

Always use the latest version of an environment:

@named_env_base(name="team/data-pipeline:latest")
class AlwaysCurrentFlow(FlowSpec):
    
    @step
    def start(self):
        # Uses whatever team/data-pipeline:latest points to
        # Updates when :latest is updated
        import pandas as pd
        self.next(self.end)
    
    @step
    def end(self):
        pass

# Equivalent (omitting :latest)
@named_env_base(name="team/data-pipeline")
class SameFlow(FlowSpec):
    pass

Fetch at Execution Time

Fetch the latest environment when tasks execute, not when deployed:

@named_env_base(
    name="prod/data-pipeline:latest",
    fetch_at_exec=True
)
class DynamicProdFlow(FlowSpec):
    
    @step
    def start(self):
        # Uses the :latest environment *when this task runs*
        # Not the :latest when the flow was deployed
        import pandas as pd
        self.next(self.end)
    
    @step
    def end(self):
        pass

Use case: Deploy flow once, but each execution picks up latest environment updates.

Using Pathspecs

Reproduce environments from previous runs:

@named_env_base(pathspec="ProductionFlow/856/train")
class ReproduceFlow(FlowSpec):
    
    @step
    def start(self):
        # Uses the exact environment from ProductionFlow run 856, step train
        import torch
        self.next(self.end)
    
    @step
    def end(self):
        pass

With environment variables:

import os

@named_env_base(pathspec="ProductionFlow/@{PROD_RUN_ID}/train")
class DebugFlow(FlowSpec):
    
    @step
    def start(self):
        # Environment variable substitution
        pass

Run with:

PROD_RUN_ID=856 python flow.py --environment=conda run

Environment Versioning Strategy

Maintain multiple environment versions:

# Development: Use latest
@named_env_base(name="team/pipeline:latest")
class DevFlow(FlowSpec):
    pass

# Staging: Use candidate
@named_env_base(name="team/pipeline:candidate")
class StagingFlow(FlowSpec):
    pass

# Production: Use stable version
@named_env_base(name="team/pipeline:v1.2.0")
class ProdFlow(FlowSpec):
    pass

Combining Multiple Decorators

Extend named base with additional packages:

from metaflow import FlowSpec, step, named_env_base, conda, pypi

@named_env_base(name="team/core:v1")
class HybridFlow(FlowSpec):
    
    @step
    def start(self):
        # Just team/core:v1
        self.next(self.process)
    
    @conda(libraries={"pytorch": "2.0.0"})
    @pypi(packages={"transformers": "4.28.0"})
    @step
    def process(self):
        # team/core:v1 + pytorch (Conda) + transformers (PyPI)
        import torch
        from transformers import AutoModel
        self.next(self.end)
    
    @step
    def end(self):
        pass

Disabling by Default

Disable environments by default, enable selectively:

@named_env_base(disabled=True)
class SelectiveFlow(FlowSpec):
    
    @step
    def system_step(self):
        # Runs in system Python
        import subprocess
        subprocess.run(["system-command"])
        self.next(self.managed_step)
    
    @named_env(name="team/ml-env:v1")
    @step
    def managed_step(self):
        # Explicitly enables named environment
        import pandas as pd
        self.next(self.end)
    
    @step
    def end(self):
        # Back to system Python
        pass

Creating Named Environments

Before using @named_env_base, create and name the environment:

From requirements.txt

metaflow environment resolve \
  --python ">=3.8,<3.9" \
  --alias team/my-env:v1 \
  -r requirements.txt

From environment.yml

metaflow environment resolve \
  --python ">=3.8,<3.9" \
  --alias team/my-env:v1 \
  -f environment.yml

From a flow’s environment

python myflow.py --environment=conda environment resolve \
  --alias team/my-env:v1

From existing pathspec

metaflow environment alias \
  --pathspec MyFlow/123/train \
  --alias team/my-env:v1

Merge Behavior

When combining @named_env_base with step-level decorators:

Extending Base Environment

@named_env_base(name="team/base:v1")
class ExtendFlow(FlowSpec):
    
    @conda(libraries={"xgboost": "1.7.0"})
    @pypi(packages={"fastapi": "0.95.0"})
    @step
    def start(self):
        # base:v1 + xgboost + fastapi
        # All packages from base:v1 remain
        pass

Overriding Base Environment

@named_env_base(name="team/default:v1")
class OverrideFlow(FlowSpec):
    
    @step
    def use_base(self):
        # Uses team/default:v1
        pass
    
    @named_env(name="team/special:v2")
    @step
    def use_different(self):
        # Completely replaces base
        # Uses team/special:v2 instead
        pass

Disabling for Specific Steps

@named_env_base(name="team/standard:v1")
class DisableFlow(FlowSpec):
    
    @step
    def use_base(self):
        # Uses team/standard:v1
        pass
    
    @named_env(disabled=True)
    @step
    def use_system(self):
        # Disables managed environment
        # Uses system Python
        pass

Interaction with Other Decorators

Cannot Combine with Definition Decorators

At flow level, @named_env_base conflicts with @conda_base and @pypi_base:

# This is an ERROR:
@conda_base(libraries={"pandas": "1.5.0"})
@named_env_base(name="team/env")  # ERROR: Conflicting decorators
class BadFlow(FlowSpec):
    pass

# This is also an ERROR:
@pypi_base(packages={"requests": "2.28.0"})
@named_env_base(name="team/env")  # ERROR: Conflicting decorators
class AlsoBadFlow(FlowSpec):
    pass

Reason: Named environments are already resolved. You can’t add requirements at the flow level.

Can Extend at Step Level

Step-level decorators can extend the base named environment:

@named_env_base(name="team/base:v1")
class GoodFlow(FlowSpec):
    
    @conda(libraries={"xgboost": "1.7.0"})
    @step
    def train(self):
        # This is OK: extends base:v1 with xgboost
        pass
    
    @pypi(packages={"fastapi": "0.95.0"})
    @step
    def serve(self):
        # This is OK: extends base:v1 with fastapi
        pass

Mutable vs Immutable Tags

Mutable Tags

These can be updated to point to different environments:

:latest - Points to the most recently resolved environment
:candidate - For testing/staging environments
:stable - For production-ready environments

@named_env_base(name="team/ml-env:latest")
class LatestFlow(FlowSpec):
    # Team can update what :latest points to
    # This flow will use the new environment next time
    pass

Immutable Tags

Once created, these always point to the same environment:

@named_env_base(name="team/ml-env:v1")
class StableFlow(FlowSpec):
    # Always uses the exact same environment
    # v1 never changes
    pass

Named environments are automatically shared across your team:

One person creates: A team member resolves and names the environment
Others use it: Everyone else references it by name
Automatic sync: Environments are cached in cloud storage (S3/GCS/Azure)

# Team member 1: Create the environment (one time)
metaflow environment resolve \
  --alias team/shared-env:v1 \
  -r requirements.txt

# Team members 2, 3, 4, ...: Use the environment
@named_env_base(name="team/shared-env:v1")
class SharedFlow(FlowSpec):
    # Automatically fetches from cloud storage
    pass

Production Deployment Pattern

Recommended pattern for production:

# Development: Use latest
@named_env_base(name="team/api:latest")
class DevFlow(FlowSpec):
    pass

# Testing: Use candidate
@named_env_base(name="team/api:candidate")
class TestFlow(FlowSpec):
    pass

# Production: Use immutable version
@named_env_base(name="team/api:v2.1.0")
class ProdFlow(FlowSpec):
    pass

Promotion workflow:

Develop with :latest
Promote to :candidate for testing
Release as versioned tag (e.g., :v2.1.0) for production

@named_env - Step-level named environment reference
@conda_base - Flow-level Conda dependencies (conflicts with this)
@pypi_base - Flow-level PyPI dependencies (conflicts with this)
@conda - Step-level Conda dependencies (extends this)
@pypi - Step-level PyPI dependencies (extends this)

Requirements

Must use --environment=conda when running the flow
Named environment must exist (be previously resolved and aliased)
Requires Conda/Mamba/Micromamba installed
Applied at the class level (before the flow class definition)

Notes

Named environments are fully resolved and locked
Cannot combine with @conda_base or @pypi_base at flow level
Can be extended at step level with @conda or @pypi
Mutable tags (:latest, :candidate, :stable) can be updated
Immutable tags (versions, dates) are permanent
Environments are cached in cloud storage for sharing
fetch_at_exec useful for production flows using mutable tags

Decorators

CLI Commands

Overview

When to Use

Parameters

Usage Examples

Basic Flow-Wide Named Environment

Extending Base Environment

Overriding Base Environment

Using Latest Environment

Fetch at Execution Time

Using Pathspecs

Environment Versioning Strategy

Combining Multiple Decorators

Disabling by Default

Creating Named Environments

From requirements.txt

From environment.yml

From a flow’s environment

From existing pathspec

Merge Behavior

Extending Base Environment

Overriding Base Environment

Disabling for Specific Steps

Interaction with Other Decorators

Cannot Combine with Definition Decorators

Can Extend at Step Level

Mutable vs Immutable Tags

Mutable Tags

Immutable Tags

Production Deployment Pattern

Requirements

Notes

Build docs developers (and LLMs) love

Decorators

CLI Commands

​Overview

​When to Use

​Parameters

​Usage Examples

​Basic Flow-Wide Named Environment

​Extending Base Environment

​Overriding Base Environment

​Using Latest Environment

​Fetch at Execution Time

​Using Pathspecs

​Environment Versioning Strategy

​Combining Multiple Decorators

​Disabling by Default

​Creating Named Environments

​From requirements.txt

​From environment.yml

​From a flow’s environment

​From existing pathspec

​Merge Behavior

​Extending Base Environment

​Overriding Base Environment

​Disabling for Specific Steps

​Interaction with Other Decorators

​Cannot Combine with Definition Decorators

​Can Extend at Step Level

​Mutable vs Immutable Tags

​Mutable Tags

​Immutable Tags

​Environment Sharing

​Production Deployment Pattern

​Related Decorators

​Requirements

​Notes

Build docs developers (and LLMs) love

Overview

When to Use

Parameters

Usage Examples

Basic Flow-Wide Named Environment

Extending Base Environment

Overriding Base Environment

Using Latest Environment

Fetch at Execution Time

Using Pathspecs

Environment Versioning Strategy

Combining Multiple Decorators

Disabling by Default

Creating Named Environments

From requirements.txt

From environment.yml

From a flow’s environment

From existing pathspec

Merge Behavior

Extending Base Environment

Overriding Base Environment

Disabling for Specific Steps

Interaction with Other Decorators

Cannot Combine with Definition Decorators

Can Extend at Step Level

Mutable vs Immutable Tags

Mutable Tags

Immutable Tags

Environment Sharing

Production Deployment Pattern

Related Decorators

Requirements

Notes