Runners Overview

​

What is a Runner?

​

Available Runners

​

Local Execution

​

Distributed Execution

​

Choosing a Runner

​

Development & Testing

• DirectRunner: Best for local development and testing
• PrismRunner: Fast local testing with portable architecture

​

Production Workloads

• DataflowRunner: Fully managed, no cluster management, automatic optimization

• FlinkRunner: Strong streaming capabilities, exactly-once processing
• SparkRunner: Leverage existing Spark infrastructure

​

Key Considerations

​

Setting the Runner

PipelineOptions options = PipelineOptionsFactory.create();
options.setRunner(DataflowRunner.class);

Pipeline p = Pipeline.create(options);

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions([
    '--runner=DataflowRunner',
])

with beam.Pipeline(options=options) as p:
    # Your pipeline code

import (
    "github.com/apache/beam/sdks/v2/go/pkg/beam"
    "github.com/apache/beam/sdks/v2/go/pkg/beam/runners/dataflow"
)

// Runner is specified when executing
dataflow.Execute(context.Background(), p)

# Java
mvn compile exec:java -Dexec.mainClass=com.example.MyPipeline \
  -Dexec.args="--runner=DataflowRunner"

# Python
python my_pipeline.py --runner=DataflowRunner

# Go
go run my_pipeline.go --runner=dataflow

​

Runner Capabilities

​

Common Capabilities

• Bounded/Unbounded PCollections: All runners support bounded data, most support unbounded
• ParDo: Supported by all runners
• GroupByKey: Supported by all runners
• Windowing: Support varies by runner
• State & Timers: Not supported by DirectRunner, supported by distributed runners

​

Next Steps