Databricks Spark Structured Streaming

作者 databrickse77e37e8a4da无许可证345 个星标收录于 2026年10月8日更新于 2026年10月8日仓库今天更新

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

仅含说明Data & Analytics Software Development

AI 生成的概览

在 Databricks 上构建生产级 Spark Structured Streaming 管道的指南，涵盖 Kafka、连接、状态、检查点与成本。

功能: 该技能提供在 Databricks 上用于生产负载的 Spark Structured Streaming 模式导航指南。它指向多份参考文档，涵盖 Kafka 接入、实时模式、Lakebase 接收端、流-流与流-静态连接、多接收端写入、合并操作、检查点、有状态操作、触发器与成本优化。它还包含一段快速入门代码片段和一份生产检查清单，涉及检查点持久化、集群规格、监控、精确一次语义与水位线。
适用场景: 在构建或调优流式管道时使用，例如从 Kafka 接入到 Delta、实现实时模式、配置触发器、使用水位线处理有状态操作，或优化流式成本与性能。
运行要求: 需要 Databricks CLI（>= v1.0.0）以及具备 Spark Structured Streaming 的 Databricks 环境。不附带脚本，仅包含说明文档、参考 Markdown 文件与图片资源。

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

python

from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streamingdf = (spark    .readStream    .format("kafka")    .option("kafka.bootstrap.servers", "broker:9092")    .option("subscribe", "topic")    .load()    .select(from_json(col("value").cast("string"), schema).alias("data"))    .select("data.*"))
df.writeStream \    .format("delta") \    .outputMode("append") \    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \    .trigger(processingTime="30 seconds") \    .start("/delta/target_table")

from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streamingdf = (spark    .readStream    .format("kafka")    .option("kafka.bootstrap.servers", "broker:9092")    .option("subscribe", "topic")    .load()    .select(from_json(col("value").cast("string"), schema).alias("data"))    .select("data.*"))
df.writeStream \    .format("delta") \    .outputMode("append") \    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \    .trigger(processingTime="30 seconds") \    .start("/delta/target_table")

Core Patterns

Pattern	Description	Reference
Kafka Streaming	Kafka to Delta, Kafka to Kafka, Real-Time Mode	See references/kafka-streaming.md [blocked]
Real-Time Mode (RTM)	Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics	See references/real-time-mode.md [blocked]
Lakebase Sink	Write streaming records into Lakebase Postgres with transactional upserts. Native `format("postgresql")` sink (DBR 18.3+) and manual `foreach` sink as a fallback	See references/lakebase-sink-python.md [blocked]
Stream Joins	Stream-stream joins, stream-static joins	See references/stream-stream-joins.md [blocked], references/stream-static-joins.md [blocked]
Multi-Sink Writes	Write to multiple tables, parallel merges	See references/multi-sink-writes.md [blocked]
Merge Operations	MERGE performance, parallel merges, optimizations	See references/merge-operations.md [blocked]

Configuration

Topic	Description	Reference
Checkpoints	Checkpoint management and best practices	See references/checkpoint-best-practices.md [blocked]
Stateful Operations	Watermarks, state stores, RocksDB configuration	See references/stateful-operations.md [blocked]
Trigger & Cost	Trigger selection, cost optimization, RTM	See references/trigger-and-cost-optimization.md [blocked]

Best Practices

Production Checklist

Checkpoint location is persistent (UC volumes, not DBFS)
Unique checkpoint per stream
Fixed-size cluster (no autoscaling for streaming)
Monitoring configured (input rate, lag, batch duration)
Exactly-once verified (txnVersion/txnAppId)
Watermark configured for stateful operations
Left joins for stream-static (not inner)

来源与署名

来源：databricks/databricks-agent-skills位于plugins/databricks/claude/skills/databricks-spark-structured-streaming提交e77e37e

许可证：无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

Databricks Spark Structured Streaming

Spark Structured Streaming

Quick Start

Core Patterns

Configuration

Best Practices

Production Checklist

来源与署名

更多来自 databricks/databricks-agent-skills 的技能

Databricks Unstructured Pdf Generation

Databricks Setup Local

Databricks Dbsql

Databricks Data Discovery

Databricks Dabs

Databricks App Design

更多Data & Analytics技能

Spanner Basics

Gke Cost Analysis

Datalineage Bigquery Asset Impact Analysis

Bigquery Troubleshooting

Bigquery Optimization

Bigquery Bigframes