Aidp Spark Optimization
by oracle-samples90b42d6c24d4No licenseListed Oct 8, 2026Updated Oct 8, 2026
Use when a Spark job/notebook is slow, missing an SLA, spilling, OOMing, generating too many small files, or shuffling/skewing heavily; when reviewing Spark/PySpark code or a Spark UI for performance; or before running a large Spark workload. Covers open-source Apache Spark 3.5.0 (+ Delta Lake) tuning -- partitions, shuffle, joins, skew, file layout, memory, codegen, caching, AQE, compression, and configuration.
Only the file list is public. File contents are available once the skill is installed in a workspace.
| Path | Size | Type |
|---|---|---|
| references/01-partitioning.md | 21.8 KB | text/markdown |
| references/02-joins.md | 24.8 KB | text/markdown |
| references/03-file-layout-io.md | 22 KB | text/markdown |
| references/04-memory-and-spill.md | 16 KB | text/markdown |
| references/05-codegen.md | 15.4 KB | text/markdown |
| references/06-caching-materialization.md | 6.8 KB | text/markdown |
| references/07-aqe.md | 21.1 KB | text/markdown |
| references/08-delta-lake.md | 18 KB | text/markdown |
| references/09-oracle-database.md | 23.1 KB | text/markdown |
| references/aidp-notes.md | 3.9 KB | text/markdown |
| references/case-studies.md | 6.9 KB | text/markdown |
| references/cluster-sizing.md | 6.6 KB | text/markdown |
| references/config-matrix.md | 13.2 KB | text/markdown |
| references/diagnosis.md | 5.3 KB | text/markdown |
| references/quick-reference.md | 6.1 KB | text/markdown |
| SKILL.md | 10.2 KB | text/markdown |
Source and attribution
Source:oracle-samples/oracle-aidp-samplesinai/claude-code-plugins/oracle-ai-data-platform-workbench-engineer-agent/skills/aidp-spark-optimizationat commit90b42d6
License: No license
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from oracle-samples/oracle-aidp-samples
Aidp Workspace Admin
oracle-samples
Provision and inspect AIDP DataLake instances and workspaces, including private-network workspaces attached to a customer VCN/subnet. Use when the user wants to create/list/get a workspace or DataLake instance, set up a new (e.g. private) AIDP environment, or replicate a customer setup. Create/delete are guarded — confirm before any provisioning.
Aidp Volumes
oracle-samples
Work with AIDP volumes — list volumes, browse files inside a volume, upload/download via the PAR flow, and create directories. Use when the user mentions volumes, needs to stage large/binary files, or move data in/out of a volume (distinct from the workspace filesystem). Control-plane via the official `aidp` CLI.
Aidp Verified Queries
oracle-samples
Maintains a repository of validated question-to-Spark-SQL pairs so an agent reuses trusted SQL before writing new queries.
Aidp User Settings
oracle-samples
Manage AIDP DataLake user settings and preferences via the aidp CLI or oci raw-request fallback.
Aidp Semantic Model
oracle-samples
Maintains a .aidp/semantic.md business-meaning layer defining metrics, joins, synonyms and value dictionaries for NL-to-SQL grounding.
Aidp Roles Access
oracle-samples
Manages AIDP roles and access: list roles, add or remove members, and grant or revoke per-resource permissions.
More in Data & Analytics
Spanner Basics
Guides Google Cloud Spanner administration, schema design, querying and performance diagnosis.
Gke Cost Analysis
Answers natural-language questions about GKE cluster and workload costs using BigQuery billing exports and live cluster metrics.
Datalineage Bigquery Asset Impact Analysis
Guides an agent through downstream impact (blast radius) analysis for a BigQuery table or view using Data Lineage.
Bigquery Troubleshooting
Diagnoses failing, slow, or unexpectedly expensive BigQuery jobs through structured root-cause workflows.
Bigquery Optimization
Guides BigQuery cost and performance optimization across capacity editions, storage layout, and SQL queries.
Bigquery Bigframes
Guides writing Python code with BigQuery DataFrames (BigFrames) for data processing, analysis, and machine learning.