Conda v2 Decorators - Metaflow Netflix Extensions

Conda v2 provides six decorators for specifying dependencies in your Metaflow flows. Each decorator comes in two variants: step-level and flow-level (base).

Available Decorators

@conda

Conda packages (step-level)

@conda_base

Conda packages (flow-level)

@pypi

PyPI packages (step-level)

@pypi_base

PyPI packages (flow-level)

@named_env

Named environment (step-level)

@named_env_base

Named environment (flow-level)

@conda Decorator

Specifies Conda packages for a step.

Parameters

libraries

Type: Dict[str, str]
Default: {} Libraries to use for this step. The key is the package name and the value is the version constraint.

@conda(libraries={
    "pandas": "1.4.0",           # Exact version
    "numpy": ">=1.20,<1.24",     # Range
    "scikit-learn": "",          # Latest version
    "conda-forge::pytorch": "2.0"  # Specific channel
})

You can specify a channel for a specific package using channel::package syntax (e.g., conda-forge::pytorch).

channels

Type: List[str]
Default: [] Additional Conda channels to search for packages.

@conda(
    libraries={"pytorch": "2.0"},
    channels=["pytorch", "conda-forge"]
)

python

Type: str
Default: None (current version) Python version constraint to use.

@conda(python="3.8.10")       # Exact version
@conda(python=">=3.8,<3.9")   # Range
@conda(python="<3.9")         # Upper bound

disabled

Type: bool
Default: False If set to True, this step will use the external environment instead of a Conda environment.

@conda(disabled=True)
@step
def my_step(self):
    # Runs in system environment
    pass

Deprecated Parameters

These parameters are deprecated and will be removed in a future release. Use the specified alternatives instead.

pip_packages: Use @pypi(packages=) instead
pip_sources: Use @pypi(extra_indices=) instead
name: Use @named_env(name=) instead
pathspec: Use @named_env(pathspec=) instead
fetch_at_exec: Use @named_env(fetch_at_exec=) instead

Example

from metaflow import FlowSpec, step, conda

class MyFlow(FlowSpec):
    
    @conda(
        libraries={
            "pandas": "1.5.0",
            "numpy": ">=1.21,<1.24",
            "scikit-learn": "1.2.0"
        },
        channels=["conda-forge"],
        python=">=3.8,<3.9"
    )
    @step
    def start(self):
        import pandas as pd
        import numpy as np
        from sklearn import __version__
        print(f"pandas: {pd.__version__}")
        print(f"numpy: {np.__version__}")
        print(f"sklearn: {__version__}")
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

@conda_base Decorator

Specifies Conda packages for all steps of the flow. Step-level @conda decorators override these settings.

Parameters

Same as @conda decorator.

Example

from metaflow import FlowSpec, step, conda, conda_base

@conda_base(
    libraries={"numpy": "1.21.5"},
    python=">=3.8,<3.9"
)
class MyFlow(FlowSpec):
    
    @step
    def start(self):
        # Uses numpy==1.21.5 from @conda_base
        import numpy as np
        print(f"numpy: {np.__version__}")
        self.next(self.process)
    
    @conda(libraries={"numpy": "1.21.6"})
    @step
    def process(self):
        # Overrides with numpy==1.21.6
        import numpy as np
        print(f"numpy: {np.__version__}")
        self.next(self.end)
    
    @conda(disabled=True)
    @step
    def end(self):
        # Runs outside conda environment
        print("System environment")

if __name__ == "__main__":
    MyFlow()

@pypi Decorator

Specifies PyPI packages for a step.

Parameters

packages

Type: Dict[str, str]
Default: {} PyPI packages to use for this step.

@pypi(packages={
    "pandas": "1.4.0",
    "requests": ">=2.25.0",
    "mypackage": ""
})

extra_indices

Type: List[str]
Default: [] Additional PyPI index URLs to search.

@pypi(
    packages={"my-private-pkg": "1.0.0"},
    extra_indices=["https://pypi.mycompany.com/simple"]
)

python

Type: str
Default: None Python version constraint.

@pypi(python=">=3.8,<3.9")

disabled

Type: bool
Default: False Disable Conda environment for this step.

Example

from metaflow import FlowSpec, step, pypi

class MyFlow(FlowSpec):
    
    @pypi(
        packages={
            "pandas": "1.5.0",
            "requests": ">=2.28.0"
        },
        python=">=3.8,<3.9"
    )
    @step
    def start(self):
        import pandas as pd
        import requests
        print(f"pandas: {pd.__version__}")
        print(f"requests: {requests.__version__}")
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

@pypi_base Decorator

Specifies PyPI packages for all steps of the flow. Step-level @pypi decorators override these settings.

Parameters

Same as @pypi decorator.

Example

from metaflow import FlowSpec, step, pypi, pypi_base

@pypi_base(
    packages={"requests": "2.28.0"},
    python=">=3.8,<3.9"
)
class MyFlow(FlowSpec):
    
    @step
    def start(self):
        # Uses requests from @pypi_base
        import requests
        print(f"requests: {requests.__version__}")
        self.next(self.end)
    
    @pypi(packages={"requests": "2.29.0"})
    @step
    def end(self):
        # Overrides with requests==2.29.0
        import requests
        print(f"requests: {requests.__version__}")

if __name__ == "__main__":
    MyFlow()

@named_env Decorator

References a named environment to use for a step.

Parameters

name

Type: str
Default: None Name or alias of the environment to use. Can use @{} for environment variable substitution.

@named_env(name="mlp/metaflow/my_env:v1")
@named_env(name="@{MY_ENV_VAR}")

pathspec

Type: str
Default: None Pathspec of an existing step to use its environment. Can use @{} for environment variable substitution.

@named_env(pathspec="MyFlow/123/train_step")
@named_env(pathspec="@{FLOW_NAME}/@{RUN_ID}/start")

You cannot specify both name and pathspec in the same decorator.

fetch_at_exec

Type: bool
Default: False If True, fetches the environment when the task executes (instead of at flow start or deploy time). Useful for always using the latest version of a mutable named environment.

@named_env(
    name="mlp/metaflow/my_env:latest",
    fetch_at_exec=True
)

This is useful when you want a step to always use the latest named environment when it runs, rather than the version that was current at deploy time.

disabled

Type: bool
Default: False Disable Conda environment for this step.

Example

from metaflow import FlowSpec, step, named_env

class MyFlow(FlowSpec):
    
    @named_env(name="mlp/metaflow/data_science_env:v1")
    @step
    def start(self):
        import numpy as np
        import pandas as pd
        print(f"Using shared environment")
        self.next(self.train)
    
    @named_env(pathspec="TrainingFlow/456/train")
    @step
    def train(self):
        # Uses the exact environment from another flow's step
        print("Training with same env as TrainingFlow")
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    MyFlow()

@named_env_base Decorator

References a named environment for all steps of the flow. Step-level @named_env decorators override this.

Parameters

Same as @named_env decorator.

@sys_packages Decorator

Specifies system virtual packages for a step. This is an advanced decorator for controlling the resolution environment.

Parameters

packages

Type: Dict[str, str]
Default: {} System virtual packages to use. Supported keys are __cuda and __glibc.

@sys_packages(packages={
    "__cuda": "11.8",
    "__glibc": "2.27"
})

This is an advanced feature. Only use if you need to override default system packages for environment resolution.

disabled

Type: bool
Default: False

Example

from metaflow import FlowSpec, step, conda, sys_packages

class GPUFlow(FlowSpec):
    
    @sys_packages(packages={"__cuda": "11.8"})
    @conda(libraries={"pytorch": "2.0"})
    @step
    def train(self):
        import torch
        print(f"PyTorch version: {torch.__version__}")
        print(f"CUDA available: {torch.cuda.is_available()}")
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    GPUFlow()

@sys_packages_base Decorator

Specifies system virtual packages for all steps of the flow.

Parameters

Same as @sys_packages decorator.

Mixing Decorators

You can mix and match decorators on the same step:

from metaflow import FlowSpec, step, conda, pypi, named_env_base

@named_env_base(name="team/base_env:v1")
class MyFlow(FlowSpec):
    
    @conda(libraries={"numpy": "1.24.0"})
    @pypi(packages={"requests": "2.28.0"})
    @step
    def start(self):
        # Uses base_env:v1 + numpy from conda + requests from pypi
        import numpy as np
        import requests
        print(f"numpy: {np.__version__}")
        print(f"requests: {requests.__version__}")
        self.next(self.end)
    
    @step
    def end(self):
        # Uses only base_env:v1
        pass

if __name__ == "__main__":
    MyFlow()

Decorator Precedence

When multiple decorators are present:

Flow-level decorators are merged

All *_base decorators are combined. They must be compatible (e.g., same Python version).

Step-level decorators are merged

All step decorators (@conda, @pypi, @named_env, etc.) are combined.

Step-level overrides flow-level

Step-level requirements override flow-level requirements for the same packages.

If you specify python=3.7 at the flow level and python=3.8 at the step level, the step will use Python 3.8.

Environment Modes

Conda v2 supports three environment modes:

Pure Conda
Pure PyPI
Mixed

Use only @conda or @conda_base decorators:

@conda(libraries={"pandas": "1.5.0"})

Uses mamba, micromamba, or conda for resolution.

Use only @pypi or @pypi_base decorators:

@pypi(packages={"pandas": "1.5.0"})

Uses pip or uv for resolution.

Use both @conda and @pypi decorators:

@conda(libraries={"cudatoolkit": "11.8"})
@pypi(packages={"tensorflow": "2.11.0"})

Uses conda-lock with poetry for resolution. More flexible but may be slower.

Get Started

Conda v2

Debug Extension

Guides

​Available Decorators

@conda

@conda_base

@pypi

@pypi_base

@named_env

@named_env_base

​@conda Decorator

​Parameters

​libraries

​channels

​python

​disabled

​Deprecated Parameters

​Example

​@conda_base Decorator

​Parameters

​Example

​@pypi Decorator

​Parameters

​packages

​extra_indices

​python

​disabled

​Example

​@pypi_base Decorator

​Parameters

​Example

​@named_env Decorator

​Parameters

​name

​pathspec

​fetch_at_exec

​disabled

​Example

​@named_env_base Decorator

​Parameters

​@sys_packages Decorator

​Parameters

​packages

​disabled

​Example

​@sys_packages_base Decorator

​Parameters

​Mixing Decorators

​Decorator Precedence

​Environment Modes

​Next Steps

Resolving Environments

Named Environments

Build docs developers (and LLMs) love

Available Decorators

@conda Decorator

Parameters

libraries

channels

python

disabled

Deprecated Parameters

Example

@conda_base Decorator

Parameters

Example

@pypi Decorator

Parameters

packages

extra_indices

python

disabled

Example

@pypi_base Decorator

Parameters

Example

@named_env Decorator

Parameters

name

pathspec

fetch_at_exec

disabled

Example

@named_env_base Decorator

Parameters

@sys_packages Decorator

Parameters

packages

disabled

Example

@sys_packages_base Decorator

Parameters

Mixing Decorators

Decorator Precedence

Environment Modes

Next Steps