Troubleshooting

This guide covers the most common issues encountered when running Flyte. Before diving in, collect the following diagnostic information:

# Get the pod status and events
kubectl describe pod <PodName> -n <namespace>

# Get pod logs
kubectl logs <PodName> -n <namespace>

<PodName> is the node execution string shown in the Flyte UI. <namespace> corresponds to the Flyte project-domain, e.g. flytesnacks-development.

The Flyte UI shows node execution IDs like ab5mg9lzgth62h82qprp-n0-0. This is also the pod name in Kubernetes.

Installation and sandbox issues

Cannot connect to the Docker daemon

Error:

Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

This occurs when running Docker Desktop instead of the native Docker engine on Linux. The socket path differs.Fix for Docker Desktop on macOS:

sudo ln -s ~/Library/Containers/com.docker.docker/Data/docker.raw.sock /var/run/docker.sock

Fix for Docker Desktop on Linux:

sudo ln -s ~/.docker/desktop/docker.sock /var/run/docker.sock

Fix for Rancher Desktop on Linux:

sudo ln -s ~/.rd/docker.sock /var/run/docker.sock

If you are using another container runtime, link its socket to /var/run/docker.sock.

Insufficient CPU when starting sandbox

Error:

message: '0/1 nodes are available: 1 Insufficient cpu.
preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.'

This is common on macOS with Docker Desktop.Fix: Open Docker Desktop settings and increase resources to a minimum of 4 CPU cores and 3 GB RAM.

TLS certificate not trusted (x509 error)

Error:

authentication handshake failed: x509: "Kubernetes Ingress Controller Fake Certificate" certificate is not trusted

This occurs when TLS is not properly configured in a flyte-core deployment.Fix: Enable TLS in your values.yaml:

ingress:
  host: example.com
  separateGrpcIngress: true
  separateGrpcIngressAnnotations:
    ingress.kubernetes.io/backend-protocol: "grpc"
  annotations:
    ingress.kubernetes.io/app-root: "/console"
    ingress.kubernetes.io/default-backend-redirect: "/console"
    kubernetes.io/ingress.class: haproxy
  tls:
    enabled: true

Also update your flytectl config to disable insecure mode:

admin:
  endpoint: dns:///example.com
  authType: Pkce
  insecure: false
  insecureSkipVerify: true

Wrong SSL version (OPENSSL_internal:WRONG_VERSION_NUMBER)

Error:

OPENSSL_internal:WRONG_VERSION_NUMBER

For flyte-binary: Verify that the endpoint name in your config.yaml matches the DNS names in the SSL certificate (whether self-signed or CA-issued).For sandbox: Verify the FLYTECTL_CONFIG environment variable points to the correct config file:

export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml

Execution failures

OOMKilled — container terminated with exit code 137

Error:

terminated with exit code (137). Reason [OOMKilled]

The container exceeded its memory limit.Fix 1: For Helm deployments, update task resource defaults in your values.yaml:

inline:
  task_resources:
    defaults:
      cpu: 100m
      memory: 100Mi
      storage: 100Mi
    limits:
      memory: 1Gi

Fix 2: Override resource limits directly in your task code:

from flytekit import Resources, task

@task(limits=Resources(mem="256Mi"))
def your_task(...):
    ...

Fix 3: For EKS deployments, adjust limits in the inline section of eks-production.yaml. Use the most recent Helm charts.

ImagePullBackOff

Error: Kubernetes cannot pull the task container image.Fix 1: If your environment uses a network proxy, pass the proxy configuration when starting the sandbox:

flytectl demo start --env HTTP_PROXY=<your-proxy-IP>

Fix 2: Never use latest as an image tag. Kubernetes changes the pull policy to Always for latest, forcing a pull on every pod start. Use a specific version tag:

@task(container_image="my-registry.example.com/my-image:v1.2.3")
def my_task(...):
    ...

Fix 3: If the registry requires authentication, create a Kubernetes image pull secret and configure it in your pod template.

ModuleNotFoundError in container tasks

Error:

ModuleNotFoundError: No module named 'mymodule'

Cause: The Python module is not on the container’s path.Fix: If using a custom Docker image, ensure:

Your Dockerfile is at the same level as the flyte directory.
An empty __init__.py exists in your project folder.

Expected directory layout:

myflyteapp/
├── Dockerfile
├── docker_build_and_tag.sh
└── flyte/
    ├── __init__.py
    └── workflows/
        ├── __init__.py
        └── example.py

Spark task error: JavaPackage is not callable

Error:

FlyteScopedUserException: 'JavaPackage' object is not callable

Cause: The spark plugin is not enabled in the FlytePropeller configuration.Fix: Add spark to the enabled-plugins list in your config YAML:

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - K8S-ARRAY
      - spark
    default-for-task-types:
      - container: container
      - container_array: K8S-ARRAY

Dynamic workflow: failed + succeeded + running inconsistent state

Error: An execution appears stuck or reports an inconsistent failed + succeeded + running state.Cause: A malformed dynamic workflow was processed by FlytePropeller. This was a known bug fixed in v1.16.4.Fix: Upgrade to Flyte v1.16.4 or later. If you cannot upgrade immediately, use RecoverExecution to resume from the last known good state:

grpcurl -plaintext \
  -d '{"id": {"project": "flytesnacks", "domain": "development", "name": "<execution-id>"}}' \
  localhost:81 flyteidl.service.AdminService/RecoverExecution

Storage and data issues

AccessDenied when writing to S3 (EKS deployment)

Error:

An error occurred (AccessDenied) when calling the PutObject operation

Cause: The Kubernetes service account Flyte uses does not have the correct IAM role annotation for IRSA (IAM Roles for Service Accounts).Fix 1: Verify the service account annotation:

kubectl describe sa <my-flyte-sa> -n <flyte-namespace>

Expected output should include:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/flyte-system-role

Fix 2: If the annotation is missing, add it manually:

kubectl annotate serviceaccount -n <flyte-namespace> <my-flyte-sa> \
  eks.amazonaws.com/role-arn=arn:aws:iam::<account-id>:role/<flyte-iam-role>

Refer to the community-maintained Flyte the Hard Way guide for full EKS IAM configuration.

Cannot access Minio in local sandbox

When running the local sandbox, Minio is available at:

Console UI: http://localhost:30080/minio/login
API: http://localhost:30002
Credentials: username minio, password miniostorage

For debugging, set these environment variables when running tasks locally:

export FLYTE_AWS_ENDPOINT="http://localhost:30002"
export FLYTE_AWS_ACCESS_KEY_ID="minio"
export FLYTE_AWS_SECRET_ACCESS_KEY="miniostorage"

Authentication issues

Unauthenticated errors in flytectl

Error: rpc error: code = UnauthenticatedFix 1: Re-authenticate:

flytectl config init --host flyte.example.com

Fix 2: Verify your config file has the correct auth settings:

admin:
  endpoint: dns:///flyte.example.com
  authType: Pkce        # or ClientSecret for service accounts
  insecure: false

Fix 3: For development/sandbox, you can disable auth entirely:

admin:
  endpoint: dns:///localhost:30080
  insecure: true

Auth config not found after demo start

After running flytectl demo start, the sandbox config is written to ~/.flyte/config-sandbox.yaml. Export it:

export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml

Add this to your shell profile to persist it across sessions.

Getting more help

GitHub Issues

Open a bug report or feature request.

Slack Community

Get real-time help in the #ask-the-community channel.

GitHub Discussions

Ask questions or share ideas with the community.

Documentation

Browse the official Flyte documentation.

Contributing

Troubleshooting

Installation and sandbox issues

Execution failures

Storage and data issues

Authentication issues

Getting more help

GitHub Issues

Slack Community

GitHub Discussions

Documentation

Build docs developers (and LLMs) love

Contributing

​Installation and sandbox issues

​Execution failures

​Storage and data issues

​Authentication issues

​Getting more help

GitHub Issues

Slack Community

GitHub Discussions

Documentation

Build docs developers (and LLMs) love

Installation and sandbox issues

Execution failures

Storage and data issues

Authentication issues

Getting more help