Configuration - Metaflow Netflix Extensions

Conda v2 provides extensive configuration options to customize its behavior. All configuration options are set through environment variables prefixed with METAFLOW_.

Storage Configuration

Configure where Conda v2 stores cached packages and environments.

CONDA_S3ROOT

Type: String (S3 URL)
Default: ${DATASTORE_SYSROOT_S3}/conda_env S3 directory containing all cached packages, environments, and conda distributions.

export METAFLOW_CONDA_S3ROOT="s3://my-bucket/conda-cache"

For safety, do not point this to the same prefix as the standard Metaflow Conda implementation.

CONDA_AZUREROOT

Type: String (Azure Blob URL)
Default: ${DATASTORE_SYSROOT_AZURE}/conda_env Azure Blob Storage directory for cached environments.

export METAFLOW_CONDA_AZUREROOT="az://my-container/conda-cache"

Azure Setup Required:

Manually create the blob container
Grant Storage Blob Data Contributor role to service principal or user accounts

CONDA_GSROOT

Type: String (GCS URL)
Default: ${DATASTORE_SYSROOT_GS}/conda_env Google Cloud Storage directory for cached environments.

export METAFLOW_CONDA_GSROOT="gs://my-bucket/conda-cache"

CONDA_LOCALROOT

Type: String (file path)
Default: ${DATASTORE_SYSROOT_LOCAL}/conda_env Local filesystem directory for cached environments (when using local datastore).

export METAFLOW_CONDA_LOCALROOT="/data/conda-cache"

CONDA_PACKAGES_DIRNAME

Type: String
Default: "packages" Directory within CONDA_*ROOT for cached packages.

export METAFLOW_CONDA_PACKAGES_DIRNAME="pkgs"

CONDA_ENVS_DIRNAME

Type: String
Default: "envs" Directory within CONDA_*ROOT for cached environments.

export METAFLOW_CONDA_ENVS_DIRNAME="environments"

Resolver Configuration

Configure which tools are used to resolve dependencies.

CONDA_DEPENDENCY_RESOLVER

Type: String
Default: "mamba"
Choices: mamba, conda, micromamba Tool to use for resolving Conda packages.

export METAFLOW_CONDA_DEPENDENCY_RESOLVER="micromamba"

mamba
micromamba
conda

Recommended: Fastest and most reliable resolver for Conda packages.

conda install mamba>=1.4.0
export METAFLOW_CONDA_DEPENDENCY_RESOLVER="mamba"

Fastest: Even faster than mamba, but sometimes less stable.

conda install micromamba>=1.4.0
export METAFLOW_CONDA_DEPENDENCY_RESOLVER="micromamba"

Slowest: Standard conda resolver. Use if others don’t work.

export METAFLOW_CONDA_DEPENDENCY_RESOLVER="conda"

CONDA_PYPI_DEPENDENCY_RESOLVER

Type: String
Default: "uv"
Choices: pip, uv, none Tool to use for resolving pure PyPI environments.

export METAFLOW_CONDA_PYPI_DEPENDENCY_RESOLVER="pip"

Set to "none" to disable pure PyPI environment support.

CONDA_MIXED_DEPENDENCY_RESOLVER

Type: String
Default: "conda-lock"
Choices: conda-lock, none Tool to use for resolving mixed Conda+PyPI environments.

export METAFLOW_CONDA_MIXED_DEPENDENCY_RESOLVER="conda-lock"

Set to "none" to disable mixed environment support.

Performance Configuration

CONDA_PREFERRED_FORMAT

Type: String
Default: "none"
Choices: .tar.bz2, .conda, none Preferred package format. Packages not in this format are automatically transmuted.

export METAFLOW_CONDA_PREFERRED_FORMAT=".conda"

Significant speedup: Using .conda format provides notable performance gains. Requires either micromamba>=1.4.0 or conda-package-handling>=1.9.0.

CONDA_LOCK_TIMEOUT

Type: Integer (seconds)
Default: 3600 Timeout for acquiring locks when creating environments.

export METAFLOW_CONDA_LOCK_TIMEOUT="300"

Conda does not like multiple processes modifying its data simultaneously. Metaflow uses lock files to prevent this. If a lock file is stuck, Metaflow will wait this long before giving up.

Remote Execution Configuration

CONDA_REMOTE_INSTALLER_DIRNAME

Type: String
Default: None Directory within CONDA_*ROOT containing cached micromamba (or similar) executables for remote environments.

export METAFLOW_CONDA_REMOTE_INSTALLER_DIRNAME="conda-remote"

If not set, latest micromamba is downloaded on remote environments when needed.

CONDA_REMOTE_INSTALLER

Type: String
Default: None Architecture-specific installer filename in CONDA_REMOTE_INSTALLER_DIRNAME. Use {arch} for architecture substitution.

export METAFLOW_CONDA_REMOTE_INSTALLER="micromamba-{arch}"

CONDA_REMOTE_COMMANDS

Type: List
Default: ["batch", "kubernetes"] Decorators that indicate remote execution.

export METAFLOW_CONDA_REMOTE_COMMANDS='["batch", "kubernetes", "my_custom_remote"]'

Local Distribution Configuration

CONDA_LOCAL_DIST_DIRNAME

Type: String
Default: None Directory within CONDA_*ROOT containing fully created conda environments for local execution.

export METAFLOW_CONDA_LOCAL_DIST_DIRNAME="conda-local"

If not set, the local machine’s Conda installation is used.

CONDA_LOCAL_DIST

Type: String
Default: None Architecture-specific tarball in CONDA_LOCAL_DIST_DIRNAME. Use {arch} for substitution.

export METAFLOW_CONDA_LOCAL_DIST="conda-{arch}.tgz"

CONDA_LOCAL_PATH

Type: String
Default: None Path to install the local conda distribution. If not exists, Metaflow installs from CONDA_LOCAL_DIST.

export METAFLOW_CONDA_LOCAL_PATH="/usr/local/libexec/metaflow-conda"

Source Configuration

CONDA_DEFAULT_PYPI_SOURCE

Type: String (URL)
Default: None (uses PyPI default) Mirror to use for PyPI packages.

export METAFLOW_CONDA_DEFAULT_PYPI_SOURCE="https://pypi.mycompany.com/simple"

CONDA_SRCS_AUTH_INFO

Type: Dictionary (JSON string)
Default: {} Authentication information for package sources. Key is hostname, value is [username, password].

export METAFLOW_CONDA_SRCS_AUTH_INFO='{"pypi.mycompany.com": ["user", "pass"]}'

Metaflow automatically reads authentication from standard locations (.netrc, pip.conf, etc.). Use this for additional sources.

Environment Resolution Behavior

CONDA_USE_REMOTE_LATEST

Type: String
Default: ":none:"
Choices: :none:, :username:, :any:, or comma-separated usernames Whether to check remote storage for already-resolved environments.

:none:
:username:
:any:
Specific Users

Default: Never use remote environments. Always re-resolve if not known locally.

export METAFLOW_CONDA_USE_REMOTE_LATEST=":none:"

Use remote environments resolved by the same user.

export METAFLOW_CONDA_USE_REMOTE_LATEST=":username:"

Use remote environments resolved by any user.

export METAFLOW_CONDA_USE_REMOTE_LATEST=":any:"

Use remote environments from specific users.

export METAFLOW_CONDA_USE_REMOTE_LATEST="alice,bob,charlie"

System Package Configuration

CONDA_SYS_DEFAULT_PACKAGES

Type: Dictionary (JSON string)
Default: {"linux-64": {"__glibc": "2.27"}} Default system packages when not specified.

export METAFLOW_CONDA_SYS_DEFAULT_PACKAGES='{"linux-64": {"__glibc": "2.27"}}'

CONDA_SYS_DEFAULT_GPU_PACKAGES

Type: Dictionary (JSON string)
Default: {} System packages to add when GPU resources are requested.

export METAFLOW_CONDA_SYS_DEFAULT_GPU_PACKAGES='{"__cuda": "11.8=0"}'

CONDA_BUILDER_ENV_PACKAGES

Type: List
Default: [] (or ["keyrings.google-artifactregistry-auth"] if GOOGLE_APPLICATION_CREDENTIALS is set) Packages needed in intermediate conda environments for resolving PyPI environments.

export METAFLOW_CONDA_BUILDER_ENV_PACKAGES='["keyrings.google-artifactregistry-auth"]'

Architecture Configuration

CONDA_ALL_ARCHS

Type: List
Default: ["linux-64", "osx-64", "osx-arm64"] All architectures supported by Conda.

export METAFLOW_CONDA_ALL_ARCHS='["linux-64", "osx-arm64"]'

Debugging Configuration

DEBUG_CONDA

Type: Boolean
Default: False Enable detailed debug output for Conda operations.

export METAFLOW_DEBUG_CONDA="1"
python myflow.py --environment=conda run

Debug output can help diagnose resolution or environment creation issues. Include this output when reporting bugs.

CONDA_TEST

Type: Boolean
Default: False Enable test mode (used internally for testing).

export METAFLOW_CONDA_TEST="1"

Common Configuration Profiles

Development Profile

Fast resolution, use latest packages:

export METAFLOW_CONDA_DEPENDENCY_RESOLVER="micromamba"
export METAFLOW_CONDA_PYPI_DEPENDENCY_RESOLVER="uv"
export METAFLOW_CONDA_PREFERRED_FORMAT=".conda"
export METAFLOW_CONDA_USE_REMOTE_LATEST=":username:"

Production Profile

Stable resolution, reproducible builds:

export METAFLOW_CONDA_DEPENDENCY_RESOLVER="mamba"
export METAFLOW_CONDA_PYPI_DEPENDENCY_RESOLVER="pip"
export METAFLOW_CONDA_USE_REMOTE_LATEST=":none:"
export METAFLOW_CONDA_PREFERRED_FORMAT=".conda"

Share environments across team:

export METAFLOW_CONDA_USE_REMOTE_LATEST=":any:"
export METAFLOW_CONDA_DEPENDENCY_RESOLVER="mamba"
export METAFLOW_CONDA_PREFERRED_FORMAT=".conda"

Enterprise Profile

Private package sources:

export METAFLOW_CONDA_DEFAULT_PYPI_SOURCE="https://pypi.mycompany.com/simple"
export METAFLOW_CONDA_SRCS_AUTH_INFO='{"pypi.mycompany.com": ["user", "pass"]}'
export METAFLOW_CONDA_DEPENDENCY_RESOLVER="mamba"
export METAFLOW_CONDA_USE_REMOTE_LATEST=":username:"

Configuration File

All configuration can be set in your Metaflow config file or directly in the code:

In Config File

~/.metaflowconfig/config.json

{
  "CONDA_S3ROOT": "s3://my-bucket/conda-cache",
  "CONDA_DEPENDENCY_RESOLVER": "mamba",
  "CONDA_PREFERRED_FORMAT": ".conda",
  "CONDA_USE_REMOTE_LATEST": ":username:"
}

In Code

Prior to Metaflow v2.10, configuration must be set in metaflow_extensions/netflix_ext/config/mfextinit_netflixext.py directly, not via external config or environment variables.

Verification

Verify your configuration:

from metaflow import metaflow_config

print(f"S3 Root: {metaflow_config.CONDA_S3ROOT}")
print(f"Resolver: {metaflow_config.CONDA_DEPENDENCY_RESOLVER}")
print(f"PyPI Resolver: {metaflow_config.CONDA_PYPI_DEPENDENCY_RESOLVER}")
print(f"Mixed Resolver: {metaflow_config.CONDA_MIXED_DEPENDENCY_RESOLVER}")
print(f"Preferred Format: {metaflow_config.CONDA_PREFERRED_FORMAT}")
print(f"Use Remote Latest: {metaflow_config.CONDA_USE_REMOTE_LATEST}")

Get Started

Conda v2

Debug Extension

Guides

​Storage Configuration

​CONDA_S3ROOT

​CONDA_AZUREROOT

​CONDA_GSROOT

​CONDA_LOCALROOT

​CONDA_PACKAGES_DIRNAME

​CONDA_ENVS_DIRNAME

​Resolver Configuration

​CONDA_DEPENDENCY_RESOLVER

​CONDA_PYPI_DEPENDENCY_RESOLVER

​CONDA_MIXED_DEPENDENCY_RESOLVER

​Performance Configuration

​CONDA_PREFERRED_FORMAT

​CONDA_LOCK_TIMEOUT

​Remote Execution Configuration

​CONDA_REMOTE_INSTALLER_DIRNAME

​CONDA_REMOTE_INSTALLER

​CONDA_REMOTE_COMMANDS

​Local Distribution Configuration

​CONDA_LOCAL_DIST_DIRNAME

​CONDA_LOCAL_DIST

​CONDA_LOCAL_PATH

​Source Configuration

​CONDA_DEFAULT_PYPI_SOURCE

​CONDA_SRCS_AUTH_INFO

​Environment Resolution Behavior

​CONDA_USE_REMOTE_LATEST

​System Package Configuration

​CONDA_SYS_DEFAULT_PACKAGES

​CONDA_SYS_DEFAULT_GPU_PACKAGES

​CONDA_BUILDER_ENV_PACKAGES

​Architecture Configuration

​CONDA_ALL_ARCHS

​Debugging Configuration

​DEBUG_CONDA

​CONDA_TEST

​Common Configuration Profiles

​Development Profile

​Production Profile

​Team Sharing Profile

​Enterprise Profile

​Configuration File

​In Config File

​In Code

​Verification

​Next Steps

Troubleshooting

CLI Reference

Build docs developers (and LLMs) love

Storage Configuration

CONDA_S3ROOT

CONDA_AZUREROOT

CONDA_GSROOT

CONDA_LOCALROOT

CONDA_PACKAGES_DIRNAME

CONDA_ENVS_DIRNAME

Resolver Configuration

CONDA_DEPENDENCY_RESOLVER

CONDA_PYPI_DEPENDENCY_RESOLVER

CONDA_MIXED_DEPENDENCY_RESOLVER

Performance Configuration

CONDA_PREFERRED_FORMAT

CONDA_LOCK_TIMEOUT

Remote Execution Configuration

CONDA_REMOTE_INSTALLER_DIRNAME

CONDA_REMOTE_INSTALLER

CONDA_REMOTE_COMMANDS

Local Distribution Configuration

CONDA_LOCAL_DIST_DIRNAME

CONDA_LOCAL_DIST

CONDA_LOCAL_PATH

Source Configuration

CONDA_DEFAULT_PYPI_SOURCE

CONDA_SRCS_AUTH_INFO

Environment Resolution Behavior

CONDA_USE_REMOTE_LATEST

System Package Configuration

CONDA_SYS_DEFAULT_PACKAGES

CONDA_SYS_DEFAULT_GPU_PACKAGES

CONDA_BUILDER_ENV_PACKAGES

Architecture Configuration

CONDA_ALL_ARCHS

Debugging Configuration

DEBUG_CONDA

CONDA_TEST

Common Configuration Profiles

Development Profile

Production Profile

Team Sharing Profile

Enterprise Profile

Configuration File

In Config File

In Code

Verification

Next Steps