Skip to main content
Flyte’s plugin system lets you run tasks on specialized infrastructure beyond plain Kubernetes pods. Backend plugins intercept task execution and dispatch work to external systems — Spark clusters, Ray clusters, Kubeflow training jobs, cloud query engines, and more.

​
Plugin types

Plugin typeDescriptionExamples
K8s operator pluginsDispatch work to Kubernetes operatorsSpark, Ray, KFPyTorch, KFTensorFlow, Dask, MPI
AWS pluginsNative AWS service integrationsAthena, SageMaker, AWS Batch
GCP pluginsNative GCP service integrationsBigQuery, Dataflow
Web API pluginsREST/gRPC calls to external servicesDatabricks, Snowflake
Agent pluginsAsync agent-based executionAny long-running external job

​
Enabling plugins

Plugins are enabled in two places:
  1. Kubernetes operator — Install the operator that the plugin delegates to (e.g., spark-operator, kuberay-operator, training-operator)
  2. Flyte task plugin config — Register the plugin in FlytePropeller’s tasks.task-plugins config

​
flyte-binary (task plugin config)

configuration:
  inline:
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
          - spark       # Add the plugin ID here
          - ray
          - pytorch
        default-for-task-types:
          - container: container
          - container_array: k8s-array
          - spark: spark
          - ray: ray
          - pytorch: pytorch

​
flyte-core (task plugin config)

configmap:
  enabled_plugins:
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
          - spark
          - ray
          - pytorch
        default-for-task-types:
          container: container
          sidecar: sidecar
          container_array: k8s-array
          spark: spark
          ray: ray
          pytorch: pytorch

​
Available plugins

Spark

Run Apache Spark jobs on Kubernetes using the spark-on-k8s-operator. Supports both Python and Scala Spark tasks.

Ray

Launch distributed Ray clusters on Kubernetes using KubeRay. Supports @task(task_config=RayJobConfig(...)).

KFPyTorch

Run distributed PyTorch training jobs using the Kubeflow training-operator. Supports multi-worker PyTorchJob configs.

KFTensorFlow

Run distributed TensorFlow training using the training-operator’s TFJob resource.

MPI

Run MPI-based distributed training jobs (Horovod, etc.) using the training-operator’s MPIJob.

Dask

Run Dask workflows on Kubernetes using the Dask Kubernetes operator.

AWS Athena

Execute SQL queries against AWS Athena directly from Flyte tasks.

AWS Batch

Submit batch compute jobs to AWS Batch from Flyte workflows.

BigQuery

Execute BigQuery SQL jobs from Flyte tasks using the GCP BigQuery client.

​
Apply and upgrade

After editing your values.yaml, upgrade the Helm release: 
helm upgrade flyte-backend flyteorg/flyte-binary \
  --namespace flyte \
  --values values.yaml
Verify all pods are healthy: 
kubectl get pods -n flyte

​
Check available plugins at runtime

# List enabled plugins
kubectl exec -n flyte deployment/flyte-backend -- \
  flyte-binary propeller list-plugins

Build docs developers (and LLMs) love

Get started for freeTalk to us