Databricks Spark Structured Streaming

by databrickse77e37e8a4daNo license345 starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated today

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

Instructions onlyData & Analytics Software Development

AI-generated overview

Guide to building production Spark Structured Streaming pipelines on Databricks, covering Kafka, joins, state, checkpoints and cost.

What it does: This skill provides a navigational guide to Spark Structured Streaming patterns for production workloads on Databricks. It points to reference documents on Kafka ingestion, Real-Time Mode, Lakebase sinks, stream-stream and stream-static joins, multi-sink writes, merge operations, checkpoints, stateful operations, triggers and cost optimization. It also includes a quick-start code snippet and a production checklist covering checkpoint persistence, cluster sizing, monitoring, exactly-once semantics and watermarks.
When to use it: Use it when building or tuning streaming pipelines, such as ingesting from Kafka into Delta, implementing Real-Time Mode, configuring triggers, handling stateful operations with watermarks, or optimizing streaming cost and performance.
Requirements: Requires the Databricks CLI (>= v1.0.0) and a Databricks environment with Spark Structured Streaming. Ships no scripts; it is instructions plus reference markdown files and image assets.

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

python

from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streamingdf = (spark    .readStream    .format("kafka")    .option("kafka.bootstrap.servers", "broker:9092")    .option("subscribe", "topic")    .load()    .select(from_json(col("value").cast("string"), schema).alias("data"))    .select("data.*"))
df.writeStream \    .format("delta") \    .outputMode("append") \    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \    .trigger(processingTime="30 seconds") \    .start("/delta/target_table")

from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streamingdf = (spark    .readStream    .format("kafka")    .option("kafka.bootstrap.servers", "broker:9092")    .option("subscribe", "topic")    .load()    .select(from_json(col("value").cast("string"), schema).alias("data"))    .select("data.*"))
df.writeStream \    .format("delta") \    .outputMode("append") \    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \    .trigger(processingTime="30 seconds") \    .start("/delta/target_table")

Core Patterns

Pattern	Description	Reference
Kafka Streaming	Kafka to Delta, Kafka to Kafka, Real-Time Mode	See references/kafka-streaming.md [blocked]
Real-Time Mode (RTM)	Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics	See references/real-time-mode.md [blocked]
Lakebase Sink	Write streaming records into Lakebase Postgres with transactional upserts. Native `format("postgresql")` sink (DBR 18.3+) and manual `foreach` sink as a fallback	See references/lakebase-sink-python.md [blocked]
Stream Joins	Stream-stream joins, stream-static joins	See references/stream-stream-joins.md [blocked], references/stream-static-joins.md [blocked]
Multi-Sink Writes	Write to multiple tables, parallel merges	See references/multi-sink-writes.md [blocked]
Merge Operations	MERGE performance, parallel merges, optimizations	See references/merge-operations.md [blocked]

Configuration

Topic	Description	Reference
Checkpoints	Checkpoint management and best practices	See references/checkpoint-best-practices.md [blocked]
Stateful Operations	Watermarks, state stores, RocksDB configuration	See references/stateful-operations.md [blocked]
Trigger & Cost	Trigger selection, cost optimization, RTM	See references/trigger-and-cost-optimization.md [blocked]

Best Practices

Production Checklist

Checkpoint location is persistent (UC volumes, not DBFS)
Unique checkpoint per stream
Fixed-size cluster (no autoscaling for streaming)
Monitoring configured (input rate, lag, batch duration)
Exactly-once verified (txnVersion/txnAppId)
Watermark configured for stateful operations
Left joins for stream-static (not inner)

Source and attribution

Source:databricks/databricks-agent-skillsinplugins/databricks/claude/skills/databricks-spark-structured-streamingat commite77e37e

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Databricks Spark Structured Streaming

Spark Structured Streaming

Quick Start

Core Patterns

Configuration

Best Practices

Production Checklist

Source and attribution

More from databricks/databricks-agent-skills

Databricks Unstructured Pdf Generation

Databricks Setup Local

Databricks Dbsql

Databricks Data Discovery

Databricks Dabs

Databricks App Design

More in Data & Analytics

Gke Cost Analysis

Kql

Google Analytics Data Api Basics

Google Analytics Admin Api Basics

Azure Kusto Irql Graph

Azure Kusto Irql