Skip to main content
Metaflow allows you to resolve environments on one architecture (e.g., your Mac laptop) and execute them on another (e.g., Linux on AWS Batch). This cross-platform capability is essential for cloud workflows but comes with important restrictions.

​
Supported Architectures

Metaflow currently supports these architectures:

linux-64

Linux x86_64Most common for cloud compute (AWS Batch, ECS)

osx-64

macOS IntelOlder Mac laptops/desktops

osx-arm64

macOS Apple SiliconM1/M2/M3 Macs

win-64

Windows x86_64Windows desktops (limited support)

​
How Cross-Platform Resolution Works

1

Local Resolution

When you run your flow or deploy it, Metaflow resolves environments on your local machine, not on remote nodes.
2

Architecture Detection

Metaflow detects the target architecture:
  • For --with batch: linux-64
  • For local runs: Your current architecture
  • For deployments: Target platform architecture
3

Platform-Specific Resolution

Conda/pip resolvers fetch packages for the target architecture, not your local one.
4

Package Caching

Resolved packages are cached to S3/Azure/GS with architecture-specific paths.
5

Remote Hydration

On the target machine, Metaflow downloads packages and creates the environment.

​
Pure Conda Packages

Pure Conda packages work seamlessly across platforms:
cross_platform_conda.py
from metaflow import FlowSpec, step, conda

class CrossPlatformFlow(FlowSpec):
    
    @conda(libraries={"pandas": "1.5.0", "numpy": "1.21.5"}, python="3.9")
    @step
    def start(self):
        import pandas as pd
        import numpy as np
        print(f"Running on {np.__version__}")
        self.next(self.end)
    
    @step
    def end(self):
        pass

if __name__ == "__main__":
    CrossPlatformFlow()
Run locally on Mac: 
python cross_platform_conda.py --environment=conda run
Run on AWS Batch (Linux): 
python cross_platform_conda.py --environment=conda --with batch run
Metaflow automatically resolves for linux-64 when using --with batch, even if you’re on a Mac.

​
PyPI Wheel Packages

PyPI packages available as wheels also work cross-platform: 
@pypi(packages={
    "pandas": "1.5.0",
    "numpy": "1.23.0",
    "scikit-learn": "1.2.0"
}, python="3.9")
@step
def train(self):
    # Works on Mac or Linux
    from sklearn.ensemble import RandomForestClassifier
    model = RandomForestClassifier()

​
How Wheel Selection Works

When resolving for a different architecture, pip uses platform-specific tags: 
# Resolving on osx-arm64 for linux-64
args = [
    "--only-binary=:all:",  # Only wheels, no source builds
    "--abi", "cp39",
    "--abi", "cp39m",
    "--abi", "abi3",
    "--platform", "manylinux2014_x86_64",
    "--platform", "manylinux_2_17_x86_64",
    # ... more platforms
]
Metaflow automatically generates the correct platform tags based on:
  • Python version
  • Target architecture
  • GLIBC version (for Linux)

​
GLIBC Considerations (Linux Only)

When resolving for Linux, Metaflow considers the GLIBC version: 
# Virtual package __glibc is automatically detected
# For linux-64, you might have:
__glibc == 2.27=0
This ensures compatibility with the system libraries available on the target:
Most AWS Batch environments use Amazon Linux 2:
__glibc: 2.26
Wheels are selected to be compatible with GLIBC 2.26+.

​
Restrictions: Non-Wheel Packages

The most important restriction: Non-wheel packages cannot be resolved cross-platform.

​
What Are Non-Wheel Packages?

Source Distributions

Packages that need compilation:
  • .tar.gz files
  • .zip source archives
  • Packages with C/C++/Rust extensions

VCS Packages

Packages from version control:
  • Git repositories
  • SVN repositories
  • Mercurial repositories

Local Packages

Local file system packages:
  • Local directories
  • File:// URLs
  • Development packages

Why They Fail

These require building on the target platform, but resolution happens on your local machine with different architecture.

​
Example of Failure

requirements_invalid.txt
# ❌ This will FAIL when resolving from Mac for Linux Batch
clip @ git+https://github.com/openai/CLIP.git@main

# ❌ This will FAIL - source-only package
outlier-detector==0.0.3

# ❌ This will FAIL - local directory
my-package @ file:///Users/me/projects/my-package
Error message: 
CondaException: Cannot resolve environment across architectures as it depends on 
local files or non-wheel packages: clip @ git+..., outlier-detector==0.0.3

​
Workarounds

Check if a wheel version exists:
pip download --only-binary=:all: outlier-detector
If not available, consider:
  • Requesting wheel builds from maintainers
  • Using a similar package that has wheels
  • Switching to pure Conda mode

​
Mixed Mode Restrictions

In mixed Conda+PyPI mode, all PyPI packages must be wheels:
env_mixed_invalid.yml
dependencies:
  - python=3.9
  - numpy=1.23.0
  - pip:
    # ❌ FAILS - Git repos not supported in mixed mode at all
    - my-pkg @ git+https://github.com/user/repo.git@main
conda-lock (used for mixed mode) does not support building packages from source. This restriction applies even when resolving on the same architecture.

​
Platform-Specific Packages

Some packages are only available on certain platforms: 
@pypi(packages={
    "pywin32": "305",  # Only available on Windows
})
@step
def windows_only(self):
    # This will fail on Mac or Linux
    import win32api
Handle platform differences: 
import sys

@pypi(packages={
    "pandas": "1.5.0",
})
@step  
def cross_platform(self):
    import pandas as pd
    
    if sys.platform == "darwin":
        print("Running on macOS")
    elif sys.platform == "linux":
        print("Running on Linux")

​
Checking Wheel Availability

Before adding a package, check if wheels exist for your target platforms: 
# Check on PyPI
pip download --only-binary=:all: --platform manylinux2014_x86_64 tensorflow

# List available wheels
pip index versions tensorflow
Look for wheels with platform tags like:
  • manylinux2014_x86_64 - Linux
  • macosx_10_9_x86_64 - macOS Intel
  • macosx_11_0_arm64 - macOS Apple Silicon
  • win_amd64 - Windows

​
Best Practices

1

Default to Wheels

Always prefer packages with wheel distributions for cross-platform compatibility.
2

Test Locally First

Run locally with --environment=conda to catch resolution issues before deploying.
3

Use Named Environments

Resolve once, then share the environment using aliases (see Environment Sharing).
4

Document Platform Requirements

If your flow requires a specific platform, document it clearly:
# This flow must run on linux-64 due to custom C extensions
@pypi(packages={"my-custom-lib": "1.0.0"})

​
Architecture Override

Manually specify target architecture: 
# Resolve for linux-64 even on Mac
metaflow environment resolve \
  --python "3.9" \
  -r requirements.txt \
  --arch linux-64
This is rarely needed as Metaflow automatically detects the correct architecture based on your execution context (--with batch, argo-workflows, etc.).

Build docs developers (and LLMs) love

Get started for freeTalk to us