Troubleshooting - Metaflow Netflix Extensions

This guide covers common issues, debugging techniques, and known limitations of Conda v2.

Common Issues

Lock File Issues

Symptom: Metaflow hangs with a message about waiting for a lock file.

Waiting for lock file /path/to/.conda_lock to be removed...

Cause: Conda doesn’t like multiple processes modifying its data simultaneously. Lock files prevent conflicts but can get stuck. Solution:

Wait for timeout

Metaflow will wait up to METAFLOW_CONDA_LOCK_TIMEOUT seconds (default: 3600) before giving up.

Remove stuck lock

If you’re certain no other process is using conda:

rm /path/to/.conda_lock

Adjust timeout

Increase the timeout if needed:

export METAFLOW_CONDA_LOCK_TIMEOUT="7200"  # 2 hours

Environment Not Found

Symptom: Error about environment not existing.

Environment 'my_env:v1' does not refer to a known environment

Possible Causes:

Environment only exists remotely

You’re using --local-only but the environment hasn’t been fetched locally.Solution: Remove --local-only or fetch the environment first:

metaflow environment get my_env:v1

Wrong architecture

Environment was resolved for a different architecture.Solution: Specify the correct architecture:

metaflow environment show --arch linux-64 my_env:v1

Typo in name

The environment name is incorrect.Solution: List available environments or check the exact name.

Resolution Failures

Symptom: Resolution fails with package conflicts.

Could not resolve environment: conflicts found

Solutions:

Loosen Constraints
Different Channels
Pure PyPI

Try relaxing version constraints:

# Instead of:
@conda(libraries={"pandas": "1.4.0"})

# Try:
@conda(libraries={"pandas": ">=1.4,<2.0"})

Try different conda channels:

@conda(
    libraries={"pytorch": "2.0"},
    channels=["pytorch", "conda-forge"]
)

If mixing conda and pip, try pure PyPI:

@pypi(packages={
    "pandas": "1.4.0",
    "numpy": "1.21.0"
})

Cross-Architecture Resolution Failures

Symptom: Cannot resolve when developing on Mac for Linux deployment.

Cannot resolve packages from repositories for linux-64 on osx-arm64

Cause: PyPI packages from Git repositories or local directories require building on target architecture. Solution:

Use wheel packages only

Ensure all PyPI packages are available as wheels:

pandas==1.5.0   # Available as wheel
numpy==1.21.5   # Available as wheel

Avoid Git repositories

Don’t use packages from Git in cross-architecture scenarios:

# This won't work cross-architecture:
# mypackage @ git+https://github.com/user/repo.git

# Build a wheel instead:
mypackage==1.0.0

Resolve on target architecture

Resolve on a Linux machine for Linux deployment.

Mixed Mode Limitations

Symptom: Cannot use certain PyPI packages in mixed conda+pip mode.

Package from Git repository not supported in mixed mode

Cause: Mixed mode (conda-lock) doesn’t support:

Git repositories
Local directories
Editable installs
Packages requiring compilation

Solution:

Pure PyPI
Conda Only

Use pure PyPI mode if you need Git packages:

@pypi(packages={
    "tensorflow": "2.11.0"
})

requirements.txt

tensorflow==2.11.0
my-git-package @ git+https://github.com/user/repo.git

Use conda packages when available:

@conda(libraries={
    "tensorflow": "2.11.0"
})

Package Format Issues

Symptom: Environment creation fails with mixed .conda and .tar.bz2 packages.

Environment creation failed: conda offline mode error

Cause: Due to a conda bug, environments with both .conda and .tar.bz2 packages fail in offline mode. Solution:

Install micromamba

Metaflow uses micromamba when available, which doesn’t have this issue:

conda install micromamba>=1.4.0

Use preferred format

Set a preferred format to avoid mixing:

export METAFLOW_CONDA_PREFERRED_FORMAT=".conda"

Metaflow will automatically transmute packages to this format.

Debugging Techniques

Enable Debug Output

Get detailed information about what Conda v2 is doing:

export METAFLOW_DEBUG_CONDA="1"
python myflow.py --environment=conda run

This shows:

Resolution steps
Package downloads
Environment creation
Cache operations

Inspect Resolved Environment

See exactly what packages are in an environment:

metaflow environment show --pathspec MyFlow/123/train

Look for:

Package versions
Resolution date/user
Architecture
Local presence

Check Local Cache

Inspect the local cache file:

# Location depends on CONDA_LOCAL_PATH
cat ~/.metaflow/conda_v2.cnd | jq

Test Resolution Manually

Test resolution without running the flow:

metaflow environment resolve \
  --dry-run \
  --python ">=3.8,<3.9" \
  -r requirements.txt

Force Re-Resolution

Force resolution to pick up new packages:

python myflow.py --environment=conda environment resolve --force

Known Issues

Micromamba Transmutation

Issue: Micromamba cannot transmute packages across architectures due to mamba#2328. Workaround: Install conda-package-handling for cross-architecture transmutation:

conda install conda-package-handling>=1.9.0

Conda Offline Mode

Issue: Conda/mamba fail to create environments with mixed package formats in offline mode due to conda#11775. Workaround: Use micromamba, which doesn’t have this issue:

conda install micromamba>=1.4.0

Environment Markers Not Supported

Issue: PEP 508 environment markers in requirements.txt are not supported.

# NOT SUPPORTED:
pandas>=1.0; python_version >= "3.8"
typing-extensions; python_version < "3.8"

Workaround: Use separate requirements files or conditional decorators:

import sys

if sys.version_info >= (3, 8):
    @pypi(packages={"pandas": ">=1.0"})
    @step
    def my_step(self):
        pass

Editable Installs Not Supported

Issue: Editable installs (pip install -e) are not supported.

# NOT SUPPORTED:
-e /path/to/local/package

Workaround: Build and install as a regular package, or include the code directly.

Migration from Conda Beta

If you used Conda v2 during beta, some environments may be re-resolved when you upgrade to v1.0.0. This is due to changes in how environments are cached.

Changes from Beta

Argument Changes

Some decorator arguments have been deprecated:

pip_packages → Use @pypi(packages=)
pip_sources → Use @pypi(extra_indices=)
sources → Use @pypi(extra_indices=)
name → Use @named_env(name=)
pathspec → Use @named_env(pathspec=)
fetch_at_exec → Use @named_env(fetch_at_exec=)

Cache Location Changes

Cache directories have changed. Set these to new values if using both versions:

export METAFLOW_CONDA_MAGIC_FILE_V2="conda_v2_new.cnd"
export METAFLOW_CONDA_PACKAGES_DIRNAME="packages_v2"
export METAFLOW_CONDA_ENVS_DIRNAME="envs_v2"

Migration Steps

Update code

Replace deprecated arguments:

# Old (beta):
@conda(pip_packages={"requests": "2.28.0"})

# New:
@pypi(packages={"requests": "2.28.0"})

Re-resolve environments

Force re-resolution with the new version:

python myflow.py --environment=conda environment resolve --force

Test thoroughly

Verify that environments resolve and work correctly:

python myflow.py --environment=conda run

Getting Help

Check Debug Output

Always include debug output when reporting issues:

export METAFLOW_DEBUG_CONDA="1"
python myflow.py --environment=conda run > debug.log 2>&1

Community Support

Get help from the Metaflow community:

Slack

Join the Metaflow community Slack

GitHub Issues

Report bugs or request features

Information to Include

When reporting issues, include:

Metaflow version: metaflow version
Extension version: pip show metaflow-extensions-netflix
Operating system and architecture
Debug output: METAFLOW_DEBUG_CONDA=1
Requirements file or decorator code
Full error message and stack trace

Performance Tips

Use .conda Format

Significant speedup with .conda packages:

export METAFLOW_CONDA_PREFERRED_FORMAT=".conda"

Use mamba or micromamba

Faster resolution than conda:

export METAFLOW_CONDA_DEPENDENCY_RESOLVER="mamba"

Cache Environments

Ensure caching is enabled:

# Verify cache location is set
echo $METAFLOW_CONDA_S3ROOT

# Should see packages being cached
export METAFLOW_DEBUG_CONDA="1"
python myflow.py --environment=conda run

Use Named Environments

Share pre-resolved environments:

# Resolve once
metaflow environment resolve --alias team/base:v1 -r requirements.txt

# Everyone uses the same environment
@named_env(name="team/base:v1")

Enable Remote Latest

Avoid re-resolving on each machine:

export METAFLOW_CONDA_USE_REMOTE_LATEST=":username:"

Get Started

Conda v2

Debug Extension

Guides

​Common Issues

​Lock File Issues

​Environment Not Found

​Resolution Failures

​Cross-Architecture Resolution Failures

​Mixed Mode Limitations

​Package Format Issues

​Debugging Techniques

​Enable Debug Output

​Inspect Resolved Environment

​Check Local Cache

​Test Resolution Manually

​Force Re-Resolution

​Known Issues

​Micromamba Transmutation

​Conda Offline Mode

​Environment Markers Not Supported

​Editable Installs Not Supported

​Migration from Conda Beta

​Changes from Beta

​Migration Steps

​Getting Help

​Check Debug Output

​Community Support

Slack

GitHub Issues

​Information to Include

​Performance Tips

​Use .conda Format

​Use mamba or micromamba

​Cache Environments

​Use Named Environments

​Enable Remote Latest

​Next Steps

Configuration

Overview

Build docs developers (and LLMs) love

Common Issues

Lock File Issues

Environment Not Found

Resolution Failures

Cross-Architecture Resolution Failures

Mixed Mode Limitations

Package Format Issues

Debugging Techniques

Enable Debug Output

Inspect Resolved Environment

Check Local Cache

Test Resolution Manually

Force Re-Resolution

Known Issues

Micromamba Transmutation

Conda Offline Mode

Environment Markers Not Supported

Editable Installs Not Supported

Migration from Conda Beta

Changes from Beta

Migration Steps

Getting Help

Check Debug Output

Community Support

Information to Include

Performance Tips

Use .conda Format

Use mamba or micromamba

Cache Environments

Use Named Environments

Enable Remote Latest

Next Steps