What does this skill help with?

It helps optimize Apache Spark jobs: partitioning strategies, broadcast joins, caching, shuffle reduction, and executor memory tuning. It also explains Spark's execution model and performance factors like data skew and serialization.

When should I use it?

Use it when improving Spark performance, debugging slow jobs, or scaling data processing pipelines. It also fits tuning memory and executor configuration and reducing shuffle and data skew.

Apache Spark Optimization

SkillDocs & knowledge

This is a skill for optimizing Apache Spark jobs. It guides an AI agent through partitioning strategies, broadcast joins, caching, shuffle reduction, and executor memory tuning, and it explains Spark's execution model and the factors that slow jobs down, such as data skew and serialization. Use it when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Connect your AI Use it in a pipeline

Then ask your AI: use the Apache Spark Optimization skill

Details

Instructions available. Your AI can read the instructions. Execution depends on the setup they require.

Account requirements not reviewed. Check the skill instructions before use; Ahel provides instructions and does not run this skill.

No other account needed.

Have an Apache Spark job or pipeline that runs slowly or needs tuning.

What your AI can do with it

Advise on partitioning strategies and right-sizing partitions
Recommend broadcast joins to avoid shuffles on small tables
Explain caching and when to use it
Suggest shuffle reduction and data skew handling such as salting
Guide executor memory tuning and serialization choices like Kryo
Provide a quick-start example of an optimized Spark session

Getting started

Have an Apache Spark job or pipeline that runs slowly or needs tuning.
Add the skill to your agent so it can read the skill instructions.
Describe the job, its transformations, and the symptoms you see, such as long stages or spills.
Ask the agent to review partitioning, caching, shuffles, and memory settings for that job.
Apply the suggested configuration changes and re-run the job to compare.

What this skill tells your AI

The instructions your AI receives, as published by wshobson/agents in plugins/data-engineering/skills/spark-optimization/SKILL.md and read by Ahel’s review.

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

When to Use This Skill

Optimizing slow Spark jobs
Tuning memory and executor configuration
Implementing efficient partitioning strategies
Debugging Spark performance issues
Scaling Spark pipelines for large datasets
Reducing shuffle and data skew

Core Concepts

1. Spark Execution Model

Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)

2. Key Performance Factors

Factor	Impact	Solution
Shuffle	Network I/O, disk I/O	Minimize wide transformations
Data Skew	Uneven task duration	Salting, broadcast joins
Serialization	CPU overhead	Use Kryo, columnar formats
Memory	GC pressure, spills	Tune executor memory
Partitions	Parallelism	Right-size partitions

Quick Start

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

# Create optimized Spark session
spark = (SparkSession.builder
    .appName("OptimizedJob")
    .config("spark.sql.adaptive.enabled", "true")
    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
    .config("spark.sql.adaptive.skewJoin.enabled", "true")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.sql.shuffle.partitions", "200")
    .getOrCreate())

# Read with optimized settings
df = (spark.read
    .format("parquet")
    .option("mergeSchema", "false")
    .load("s3://bucket/data/"))

# Efficient transformations
result = (df
    .filter(F.col("date") >= "2024-01-01")
    .select("id", "amount", "category")
    .groupBy("category")
    .agg(F.sum("amount").alias("total")))

result.write.mode("overwrite").parquet("s3://bucket/output/")

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

Enable AQE - Adaptive query execution handles many issues
Use Parquet/Delta - Columnar formats with compression
Broadcast small tables - Avoid shuffle for small joins
Monitor Spark UI - Check for skew, spills, GC
Right-size partitions - 128MB - 256MB per partition

Don'ts

Don't collect large data - Keep data distributed
Don't use UDFs unnecessarily - Use built-in functions
Don't over-cache - Memory is limited
Don't ignore data skew - It dominates job time
Don't use .count() for existence - Use .take(1) or .isEmpty()

Signals

GitHub stars: 40k
Forks: 4k
Last commit: Oct 2026

Questions

What does this skill help with?: It helps optimize Apache Spark jobs: partitioning strategies, broadcast joins, caching, shuffle reduction, and executor memory tuning. It also explains Spark's execution model and performance factors like data skew and serialization.
When should I use it?: Use it when improving Spark performance, debugging slow jobs, or scaling data processing pipelines. It also fits tuning memory and executor configuration and reducing shuffle and data skew.

Advanced

Item type: skill
Key: spark-optimization-wshobson
Source: github.com/wshobson/agents

github.com/wshobson/agents