Spark Optimization
by wshobson46891e7e60daNo licenseListed Oct 8, 2026Updated Oct 8, 2026
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Add to a SourceWeft workspace
- Open the skill in your dashboard and add it to a workspace.
- Enable it for the chats that should use it.
This skill is instructions only: it ships no scripts to execute.
Add to SourceWeftYou will be asked to sign in first, then taken straight to this skill.
Ask your agent to install it
Paste this prompt into Claude Code, Codex, Cursor or another agent that can run commands — or into SourceWeft chat. The agent reads this skill's install guide, shows you its source, license and scripts, and installs it with the SourceWeft CLI once you agree.
Read https://sourceweft.com/skills/gh-wshobson-agents-spark-optimization/install.md and install the skill it describes. Before installing, show me its source, license and whether it ships scripts, and wait for my OK. Ask me before changing anything else on my machine.Install it yourself from a terminal
For Claude Code, Codex, Cursor and other local agents. The SourceWeft CLI fetches the skill from its source repository at the commit scanned here, and verifies every file against the hashes recorded when the skill was scanned. If anything differs, nothing is written.
npx @sourceweft/cli skills install gh-wshobson-agents-spark-optimizationAdd --agent claude-code, codex, cursor or universal to choose which agent gets it (Claude Code by default).
Upstream installer — not verified by SourceWeft
The open-source skills installer fetches the same pinned commit, but does not check the files against the hashes SourceWeft recorded.
npx skills add https://github.com/wshobson/agents/tree/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/data-engineering/skills/spark-optimizationSource and attribution
Source:wshobson/agentsinplugins/data-engineering/skills/spark-optimizationat commit46891e7
License: No license
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from wshobson/agents
Uv Package Manager
wshobson
Reference guide for using the uv Python package manager for dependencies, virtual environments and project workflows.
Web Component Design
wshobson
Guides building reusable UI components in React, Vue and Svelte, covering composition patterns, CSS-in-JS choices and component API design.
Python Type Safety
wshobson
Guides Python type annotations, generics, protocols, and strict mypy/pyright checking.
Visual Design Foundations
wshobson
Guides typography, color, spacing and iconography decisions for cohesive, accessible visual design systems.
Responsive Design
wshobson
Guides implementation of responsive web layouts using container queries, fluid typography, CSS Grid and breakpoints.
Python Resource Management
wshobson
Guides Python developers in managing resources deterministically with context managers, cleanup patterns, and streaming.
More in Data & Analytics
Spanner Basics
Guides Google Cloud Spanner administration, schema design, querying and performance diagnosis.
Gke Cost Analysis
Answers natural-language questions about GKE cluster and workload costs using BigQuery billing exports and live cluster metrics.
Datalineage Bigquery Asset Impact Analysis
Guides an agent through downstream impact (blast radius) analysis for a BigQuery table or view using Data Lineage.
Bigquery Troubleshooting
Diagnoses failing, slow, or unexpectedly expensive BigQuery jobs through structured root-cause workflows.
Bigquery Optimization
Guides BigQuery cost and performance optimization across capacity editions, storage layout, and SQL queries.
Bigquery Bigframes
Guides writing Python code with BigQuery DataFrames (BigFrames) for data processing, analysis, and machine learning.