Getting Started with Conda v2

This guide will walk you through your first steps with Conda v2, from basic usage to creating reproducible environments.

Prerequisites

Installation

Install the Metaflow Netflix Extensions package alongside Metaflow:

pip install metaflow metaflow-extensions-netflix

This extension requires Metaflow v2.8.3 or later.

Required Tools

Your local Conda environment needs:

conda (required)
mamba >= 1.4.0 (optional but recommended)
micromamba >= 1.4.0 (strongly recommended)

Pure PyPI Support
Mixed Package Support
Package Format Support

For environments containing only pip packages:

pip install pip>=23.0

For environments with both pip and conda packages:

conda install conda-lock>=2.1.0

For .tar.bz2 and .conda package format support:

# Option 1: conda-package-handling
conda install conda-package-handling>=1.9.0

# Option 2: micromamba (recommended)
conda install micromamba>=1.4.0

Your First Conda v2 Flow

Basic Conda Packages

Here’s a simple flow using different versions of pandas in different steps:

simplecondaflow.py

from metaflow import FlowSpec, step, conda

class CondaTestFlow(FlowSpec):
    
    @conda(libraries={"pandas": "1.4.0"}, python=">=3.8,<3.9")
    @step
    def start(self):
        import pandas as pd
        assert pd.__version__ == "1.4.0"
        print("I am in start and Pandas version is %s" % pd.__version__)
        self.next(self.end)
        
    @conda(libraries={"pandas": "1.5.0"}, python=">=3.8,<3.9")
    @step
    def end(self):
        import pandas as pd
        assert pd.__version__ == "1.5.0"
        print("I am in end and Pandas version is %s" % pd.__version__)

if __name__ == "__main__":
    CondaTestFlow()

You must use --environment=conda when running flows that use Conda decorators.

Run the flow:

python simplecondaflow.py --environment=conda run

Output:

Workflow starting (run-id 142), see it in the UI at https://mfui.net/CondaTestFlow/142
    Using existing Conda environment 42a4ed94b63f12e1fe9dd29de21bf9ec6e271b1c (a3b104c4ce2215351a2b94076ef7827de3ad890a)
    [142/start/145426383 (pid 21786)] Task is starting.
    [142/start/145426383 (pid 21786)] I am in start and Pandas version is 1.4.0
    [142/start/145426383 (pid 21786)] Task finished successfully.
    Using existing Conda environment 3e07a415e7766b8ed359f00e5f48e35ec79ac056 (41a06733cd332951fa100475c3a05c6916272899)
    [142/end/145426398 (pid 21837)] Task is starting.
    [142/end/145426398 (pid 21837)] I am in end and Pandas version is 1.5.0
    [142/end/145426398 (pid 21837)] Task finished successfully.
    Done! See the run in the UI at https://mfui.net/CondaTestFlow/142

Using PyPI Packages

You can also use pure PyPI packages with the @pypi decorator:

simplecondaflow-pypi.py

from metaflow import FlowSpec, step, pypi

class CondaTestFlowPypi(FlowSpec):
    
    @pypi(packages={"pandas": "1.4.0"}, python=">=3.8,<3.9")
    @step
    def start(self):
        import pandas as pd
        assert pd.__version__ == "1.4.0"
        print("I am in start and Pandas version is %s" % pd.__version__)
        self.next(self.end)
        
    @pypi(packages={"pandas": "1.5.0"}, python=">=3.8,<3.9")
    @step
    def end(self):
        import pandas as pd
        assert pd.__version__ == "1.5.0"
        print("I am in end and Pandas version is %s" % pd.__version__)

if __name__ == "__main__":
    CondaTestFlowPypi()

python simplecondaflow-pypi.py --environment=conda run

Using Flow-Level Decorators

Use flow-level decorators to specify common dependencies for all steps:

from metaflow import FlowSpec, step, conda, conda_base

@conda_base(libraries={'numpy':'1.21.5'}, python='>3.8,<3.9')
class CondaTestFlow(FlowSpec):
    @step
    def a(self):
        import numpy as np
        print(f"Step A: numpy {np.__version__}")
        self.next(self.b)

    @conda(libraries={'numpy':'1.21.6'})
    @step
    def b(self):
        import numpy as np
        print(f"Step B: numpy {np.__version__}")
        self.next(self.c)

    @conda(disabled=True)
    @step
    def c(self):
        # This step runs outside the conda environment
        print("Step C: using system environment")

if __name__ == "__main__":
    CondaTestFlow()

In this example:

Step A executes with numpy==1.21.5 and python>3.8,<3.9 (from @conda_base)
Step B executes with numpy==1.21.6 and python>3.8,<3.9 (overrides numpy)
Step C executes outside the conda environment (system environment)

Step-level decorators override flow-level decorators. Use this pattern to set common dependencies at the flow level and override them for specific steps.

Using Requirements Files

requirements.txt Format

Create a requirements.txt file:

req_numpy.txt

numpy==1.21.5

Resolve the environment:

metaflow environment resolve --python ">=3.8,<3.9" -r req_numpy.txt

environment.yml Format

Create an environment.yml file:

env_numpy.yml

dependencies:
  - numpy=1.21.5

Resolve the environment:

metaflow environment resolve --python ">=3.8,<3.9" -f env_numpy.yml

Pure PyPI
Pure Conda
Mixed

Use requirements.txt for pure PyPI environments:

pandas>=1.0.0
numpy==1.21.5
scikit-learn<1.2

Use environment.yml for pure Conda environments:

dependencies:
  - pandas>=1.0.0
  - numpy=1.21.5
  - scikit-learn<1.2

Use environment.yml with pip section for mixed environments:

dependencies:
  - pandas>=1.0.0
  - pip:
    - tensorflow==2.7.4
    - apache-airflow[aiobotocore]

Environment Resolution

When you run a flow, Metaflow automatically resolves environments before execution:

Automatic Resolution

Environments are resolved automatically when you:

Run a flow with --environment=conda

Deploy to Argo/Airflow/StepFunctions

Resolution Location

Resolution always happens on your local machine, not on remote nodes.

Caching

Resolved environments are cached:

Locally: In a special file (conda_v2.cnd)

Remotely: In S3/Azure/GCS for sharing

Default Environments

For each req_id, Metaflow uses a default environment (the latest locally resolved). Environments are not re-resolved on subsequent runs unless forced.

Forcing Re-Resolution

To force re-resolution of environments:

python simplecondaflow.py --environment=conda environment resolve --force

By default, Metaflow will reuse previously resolved environments. Use --force only when you need to pick up new package versions or resolve for a new architecture.

Viewing Environment Information

Inspect what’s in a resolved environment:

metaflow environment show --pathspec CondaTestFlow/118/start

This shows:

Environment IDs (req_id and full_id)
Resolution date and user
All conda packages installed
All PyPI packages installed
Local environment location

Next Steps

Decorators

Learn about all decorator parameters

Named Environments

Create and share named environments

CLI Reference

Explore all CLI commands

Configuration

Configure Conda v2 options

Get Started

Conda v2

Debug Extension

Guides

Prerequisites

Installation

Required Tools

Your First Conda v2 Flow

Basic Conda Packages

Using PyPI Packages

Using Flow-Level Decorators

Using Requirements Files

requirements.txt Format

environment.yml Format

Environment Resolution

Forcing Re-Resolution

Viewing Environment Information

Next Steps

Decorators

Named Environments

CLI Reference

Configuration

Build docs developers (and LLMs) love

Get Started

Conda v2

Debug Extension

Guides

​Prerequisites

​Installation

​Required Tools

​Your First Conda v2 Flow

​Basic Conda Packages

​Using PyPI Packages

​Using Flow-Level Decorators

​Using Requirements Files

​requirements.txt Format

​environment.yml Format

​Environment Resolution

​Forcing Re-Resolution

​Viewing Environment Information

​Next Steps

Decorators

Named Environments

CLI Reference

Configuration

Build docs developers (and LLMs) love

Prerequisites

Installation

Required Tools

Your First Conda v2 Flow

Basic Conda Packages

Using PyPI Packages

Using Flow-Level Decorators

Using Requirements Files

requirements.txt Format

environment.yml Format

Environment Resolution

Forcing Re-Resolution

Viewing Environment Information

Next Steps