Migration Guide - Metaflow Netflix Extensions

The Netflix Extensions Conda implementation provides significant improvements over the standard Metaflow Conda decorator. This guide helps you migrate existing flows and understand breaking changes.

Overview of Changes

What Changed in v1.0.0

Version 1.0.0 introduced breaking changes to improve usability and consistency. Some arguments have been renamed or moved.

Deprecated Arguments
New Decorators
Resolution Timing

These arguments have been moved or renamed:

Old Argument	New Location	Decorator
`pip_packages`	`packages` in `@pypi`	`@conda`
`pip_sources`	`extra_indices` in `@pypi`	`@conda`
`sources`	`channels`	`@conda`
`name`	Use `@named_env`	`@conda`
`pathspec`	Use `@named_env`	`@conda`
`fetch_at_exec`	Removed	`@conda`

All previous options still work but will be removed in a future release. Migrate to the new syntax.

Six decorators now exist:

@conda - Step-level Conda packages
@conda_base - Flow-level Conda packages
@pypi - Step-level PyPI packages
@pypi_base - Flow-level PyPI packages
@named_env - Step-level named environments
@named_env_base - Flow-level named environments

This separation provides clearer semantics.

Migrating from Standard Conda

Basic Migration

Before (standard Conda):

from metaflow import FlowSpec, step, conda

class OldFlow(FlowSpec):
    
    @conda(libraries={'pandas': '1.4.0'})
    @step
    def start(self):
        import pandas
        self.next(self.end)
    
    @step
    def end(self):
        pass

After (Netflix Extensions):

from metaflow import FlowSpec, step, conda

class NewFlow(FlowSpec):
    
    @conda(libraries={'pandas': '1.4.0'})
    @step
    def start(self):
        import pandas
        self.next(self.end)
    
    @step
    def end(self):
        pass

For pure Conda packages, the syntax is identical! The Netflix Extensions are backward compatible.

Migrating PyPI Packages

Before (standard Conda):

@conda(
    libraries={'pandas': '1.4.0'},
    pip_packages=['tensorflow==2.10.0'],
    pip_sources=['https://my-pypi.com/simple']
)
@step
def train(self):
    import pandas, tensorflow

After (Netflix Extensions):

@conda(libraries={'pandas': '1.4.0'})
@pypi(
    packages={'tensorflow': '2.10.0'},
    extra_indices=['https://my-pypi.com/simple']
)
@step
def train(self):
    import pandas, tensorflow

Separate Decorators

Split pip_packages into a separate @pypi decorator.

Rename Arguments

pip_packages → packages (dict, not list)
pip_sources → extra_indices

Use Dict Format

# Before: list
pip_packages=['tensorflow==2.10.0']

# After: dict
packages={'tensorflow': '2.10.0'}

Migrating Channels

Before:

@conda(
    libraries={'numpy': '1.21.5'},
    sources=['conda-forge', 'defaults']
)

After:

@conda(
    libraries={'numpy': '1.21.5'},
    channels=['conda-forge', 'defaults']
)

Simple rename: sources → channels.

Using Flow-Level Decorators

The Netflix Extensions introduce flow-level decorators for shared dependencies: Before:

class OldFlow(FlowSpec):
    
    @conda(libraries={'pandas': '1.4.0', 'numpy': '1.21.5'})
    @step
    def start(self):
        import pandas, numpy
        self.next(self.train)
    
    @conda(libraries={'pandas': '1.4.0', 'numpy': '1.21.5', 'scikit-learn': '1.2.0'})
    @step
    def train(self):
        import pandas, numpy, sklearn
        self.next(self.end)
    
    @conda(libraries={'pandas': '1.4.0', 'numpy': '1.21.5'})
    @step
    def end(self):
        import pandas, numpy

After:

@conda_base(libraries={'pandas': '1.4.0', 'numpy': '1.21.5'})
class NewFlow(FlowSpec):
    
    @step
    def start(self):
        # Has pandas and numpy from conda_base
        import pandas, numpy
        self.next(self.train)
    
    @conda(libraries={'scikit-learn': '1.2.0'})
    @step
    def train(self):
        # Has pandas, numpy (from base) AND scikit-learn
        import pandas, numpy, sklearn
        self.next(self.end)
    
    @step
    def end(self):
        # Has pandas and numpy from conda_base
        import pandas, numpy

Benefits:

DRY (Don’t Repeat Yourself)
Easier maintenance
Clear flow-level vs step-level dependencies

Migrating Named Environments

Before:

@conda(
    libraries={'pandas': '1.4.0'},
    name='my-team-env'
)
@step
def start(self):
    import pandas

After:

@named_env(name='my-team-env')
@step
def start(self):
    import pandas

Named environments now use a dedicated decorator.

Using Pathspecs

Before:

@conda(pathspec='MyFlow/123/train')
@step
def debug(self):
    import pandas

After:

@named_env(pathspec='MyFlow/123/train')
@step  
def debug(self):
    import pandas

Environment Resolution Behavior

What to Expect

The first time you run with Netflix Extensions, environments will be re-resolved even if you previously ran the flow with standard Conda.

Why?

Standard Conda and Netflix Extensions use different caching formats
Netflix Extensions uses a new metadata file (conda_v2.cnd)
Package cache locations differ (if configured separately)

First Run

python myflow.py --environment=conda run

Output:

Resolving 2 environments ... done in 27 seconds.
Creating Conda environment 42a4ed94b63f12e1fe9dd29de21bf9ec6e271b1c ...

Subsequent Runs

python myflow.py --environment=conda run

Output:

Using existing Conda environment 42a4ed94b63f12e1fe9dd29de21bf9ec6e271b1c

After the first resolution, environments are cached and reused, making subsequent runs much faster.

Configuration Changes

Storage Locations

For safety, do not point Netflix Extensions to the same S3/Azure/GS prefix as standard Conda.

Standard Conda:

METAFLOW_CONDA_PACKAGE_S3ROOT = 's3://my-bucket/conda/'

Netflix Extensions:

# Use a different prefix
METAFLOW_CONDA_S3ROOT = 's3://my-bucket/conda-v2/'
# Or Azure
METAFLOW_CONDA_AZUREROOT = 'azure://my-container/conda-v2/'
# Or Google Cloud
METAFLOW_CONDA_GSROOT = 'gs://my-bucket/conda-v2/'

New Configuration Options

Netflix Extensions add several new config options:

~/.metaflowconfig/config.json

{
  # Storage root (S3/Azure/GS)
  "METAFLOW_CONDA_S3ROOT": "s3://my-bucket/conda-v2/",
  
  # Resolvers
  "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba",  # or "conda", "micromamba"
  "METAFLOW_CONDA_PYPI_DEPENDENCY_RESOLVER": "pip",  # or null to disable
  "METAFLOW_CONDA_MIXED_DEPENDENCY_RESOLVER": "conda-lock",  # or "none"
  
  # Performance
  "METAFLOW_CONDA_PREFERRED_FORMAT": ".conda",  # or ".tar.bz2" or "none"
  
  # Remote environments
  "METAFLOW_CONDA_USE_REMOTE_LATEST": ":username:"  # or ":none:", ":any:", "user1,user2"
}

See Configuration for all options.

Handling Errors

Resolution Failures

Error:

CondaException: Cannot resolve environment across architectures as it depends on 
local files or non-wheel packages

Solution: Netflix Extensions is stricter about cross-platform resolution. See Cross-Platform Resolution for workarounds.

Mixed Mode Errors

Error:

CondaException: Local PYPI packages are not supported in MIXED mode

Solution: Mixed mode (Conda + PyPI) doesn’t support packages from Git or local directories. Use pure PyPI mode or wheels only.

Missing Resolvers

Error:

CondaException: Could not find conda binary for conda-lock

Solution: Install required resolvers:

conda install conda-lock>=2.1.0  # For mixed mode
conda install pip>=23.0  # For PyPI-only mode

Testing Your Migration

Test Locally First

Run flows locally with --environment=conda before deploying:

python myflow.py --environment=conda run

Force Re-resolution

Test that resolution works:

python myflow.py --environment=conda environment resolve --force

Check Environment

Inspect resolved environments:

python myflow.py --environment=conda environment show --pathspec MyFlow/latest/start

Test Remote Execution

Try Batch/Kubernetes:

python myflow.py --environment=conda --with batch run

Rollback

If you encounter issues, you can temporarily rollback:

# Uninstall Netflix Extensions
pip uninstall metaflow-netflix-extensions

# Your code will use standard Metaflow Conda
python myflow.py --environment=conda run

Uninstalling the Netflix Extensions reverts to standard Metaflow behavior with no conflicts (assuming separate storage prefixes).

Migration Checklist

Before Migration

Read this guide fully
Back up existing flows
Document current dependencies
Test in non-production first

During Migration

Update decorator syntax
Separate PyPI packages to @pypi
Use flow-level decorators where appropriate
Update configuration

After Migration

Test local execution
Test remote execution
Verify environment resolution
Document changes for team

Optional Enhancements

Create named environments
Set up remote environment sharing
Configure preferred package format
Enable remote resolution

Getting Help

If you encounter issues during migration:

Enable debug logging:

METAFLOW_DEBUG_CONDA=1 python myflow.py --environment=conda run

Check the Metaflow Slack: Join slack.outerbounds.co
Open an issue: GitHub Issues

Summary of Benefits

After migrating, you’ll have:

Faster Resolution

Parallel resolution and better caching

Mixed Environments

Combine Conda and PyPI packages

Named Environments

Shareable, aliased environments

Cross-Platform

Resolve on Mac, run on Linux

Better Inspection

CLI tools to inspect any environment

Team Sharing

Remote resolution and environment reuse

Get Started

Conda v2

Debug Extension

Guides

​Overview of Changes

​What Changed in v1.0.0

​Migrating from Standard Conda

​Basic Migration

​Migrating PyPI Packages

​Migrating Channels

​Using Flow-Level Decorators

​Migrating Named Environments

​Using Pathspecs

​Environment Resolution Behavior

​What to Expect

​First Run

​Subsequent Runs

​Configuration Changes

​Storage Locations

​New Configuration Options

​Handling Errors

​Resolution Failures

​Mixed Mode Errors

​Missing Resolvers

​Testing Your Migration

​Rollback

​Migration Checklist

Before Migration

During Migration

After Migration

Optional Enhancements

​Getting Help

​Summary of Benefits

Faster Resolution

Mixed Environments

Named Environments

Cross-Platform

Better Inspection

Team Sharing

Build docs developers (and LLMs) love

Overview of Changes

What Changed in v1.0.0

Migrating from Standard Conda

Basic Migration

Migrating PyPI Packages

Migrating Channels

Using Flow-Level Decorators

Migrating Named Environments

Using Pathspecs

Environment Resolution Behavior

What to Expect

First Run

Subsequent Runs

Configuration Changes

Storage Locations

New Configuration Options

Handling Errors

Resolution Failures

Mixed Mode Errors

Missing Resolvers

Testing Your Migration

Rollback

Migration Checklist

Getting Help

Summary of Benefits