Skip to main content

​
StreamMetrics

@dataclass
class StreamMetrics:
    latency_ms: float
    throughput_rows_per_s: float
Dataclass for storing streaming performance metrics.
​
latency_ms
float
Average latency in milliseconds per row processed
​
throughput_rows_per_s
float
Number of rows processed per second

​
stream_dataframe

def stream_dataframe(df: pd.DataFrame, chunk_size: int)
Generator function that yields chunks of a DataFrame for streaming processing.
​
df
pd.DataFrame
required
The DataFrame to stream in chunks
​
chunk_size
int
required
Number of rows per chunk
​
return
Generator[pd.DataFrame]
Generator yielding DataFrame chunks of size chunk_size

​
process_stream

def process_stream(
    df: pd.DataFrame,
    chunk_size: int,
    process_fn
) -> tuple[pd.DataFrame, StreamMetrics]
Processes a DataFrame in streaming chunks and returns results with performance metrics.
​
df
pd.DataFrame
required
The DataFrame to process
​
chunk_size
int
required
Number of rows per chunk for streaming
​
process_fn
Callable
required
Function to apply to each chunk. Should accept a DataFrame and return a DataFrame
​
return
tuple[pd.DataFrame, StreamMetrics]
A tuple containing:
  • Concatenated DataFrame with all processed results
  • StreamMetrics object with performance statistics

​
compare_batch_vs_streaming

def compare_batch_vs_streaming(
    df: pd.DataFrame,
    process_fn,
    chunk_size: int
) -> dict[str, float]
Compares performance between batch and streaming processing modes.
​
df
pd.DataFrame
required
The DataFrame to process in both modes
​
process_fn
Callable
required
Function to apply in both batch and streaming modes
​
chunk_size
int
required
Chunk size for streaming mode
​
return
dict[str, float]
Dictionary containing performance comparison metrics:
  • batch_time_s: Total time for batch processing in seconds
  • stream_time_s: Total time for streaming processing in seconds
  • stream_latency_ms_per_row: Average latency per row in milliseconds
  • stream_throughput_rows_per_s: Rows processed per second in streaming mode

Build docs developers (and LLMs) love

Get started for freeTalk to us