spark-version-upgrade

Files (5)

• .claude-plugin
• .codex-plugin
• plugin.json · .plugin/
• README.md
• SKILL.md

SKILL.md (first lines)

Upgrade Apache Spark applications between major versions with a structured, phase-by-phase workflow.

## When to Use

- Migrating from Spark 2.x → 3.x or Spark 3.x → 4.x
- Updating PySpark, Spark SQL, or Structured Streaming applications
- Resolving deprecation warnings before a Spark version bump

## Workflow Overview

1. **Inventory & Impact Analysis** — Scan the codebase and assess scope
2. **Build File Updates** — Bump Spark/Scala/Java dependencies
3. **API Migration** — Replace deprecated and removed APIs
4. **Configuration Migration** — Update Spark config properties
5. **SQL & DataFrame Migration** — Fix query-level breaking changes
6. **Test Validation** — Compile, run tests, verify results

---

## Phase 1: Inventory & Impact Analysis

Before changing any code, assess what needs to change. Read the official Apache Spark migration guide for the target version — it documents every API removal, config rename, and behavioral change per release:
https://spark.apache.org/docs/latest/migration-guide.html

### Checklist

- [ ] Read the migration guide section for the target Spark version
- [ ] Identify current Spark version (check `pom.xml`, `build.sbt`, `build.gradle`, or `requirements.txt`)
- [ ] Identify target Spark version
- [ ] Search for deprecated APIs: `grep -rn 'import org.apache.spark' --include='*.scala' --include='*.java' --include='*.py'`
- [ ] List all Spark config properties: `grep -rn 'spark\.' --include='*.conf' --include='*.properties' --include='*.scala' --include='*.java' --include='*.py' | grep -v 'test'`
- [ ] On Windows PowerShell, use `Get-ChildItem -Recurse -Include *.scala,*.java,*.py | Select-String 'import org.apache.spark'` and adjust the extensions/pattern for config searches.
- [ ] Check for custom `SparkSession` or `SparkContext` extensions
- [ ] Identify connector dependencies (Hive, Kafka, Cassandra, Delta, Iceberg)
- [ ] Document findings in `spark_upgrade_impact.md`

### Output

```
spark_upgrade_impact.md   # Summary of affected files, APIs, and configs
```

---

## Phase 2: Build File Updates

Update dependency versions and resolve compilation.

### Maven (`pom.xml`)

```xml
<!-- Update Spark version property -->
<spark.version>3.5.1</spark.version>    <!-- or 4.0.0 -->
<scala.version>2.13.12</scala.version>  <!-- Spark 3.x: 2.12/2.13; Spark 4.x: 2.13 -->

<!-- Update artifact IDs if Scala cross-version changed -->
<artifactId>spark-core_2.13</artifactId>
<artifactId>spark-sql_2.13</artifactId>
```

### SBT (`build.sbt`)

```scala
val sparkVersion = "3.5.1" // or "4.0.0"
scalaVersion := "2.13.12"

libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion
```

### Gradle (`build.gradle`)

```groovy
ext {
    sparkVersion = '3.5.1' // or '4.0.0'
}
dependencies {
    implementation "org.apache.spark:spark-core_2.13:${sparkVersion}"
    implementation "org.apache.spark:spark-sql_2.13:${sparkVersion}"
}
```

### PySpark (`requirements.txt` / `pyproject.toml`)

```
pyspark==3.5.1   # or 4.0.0
```

### Checklist

- [ ] Update Spark version in build file
- [ ] Update Scala version if crossing 2.12→2.13 boundary
- [ ] Update Java source/target level if required (Spark 4.x requires Java 17+)
- [ ] Update connector library versions to match new Spark version
- [ ] Resolve dependency conflicts (`mvn dependency:tree` / `sbt dependencyTree`)
- [ ] Confirm project compiles (errors at this stage are expected — they guide Phase 3)

---

## Phase 3: API Migration

Replace removed and deprecated APIs. Work through compiler errors systematically.

### Common Patterns

Consult the official Apache Spark migration guide for the complete list of changes for each version:
https://spark.apache.org/docs/latest/migration-guide.html

#### SparkSession Creation (2.x → 3.x)

```scala
// BEFORE (Spark 1.x/2.x)
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)

// AFTER (Spark 2.x+/3.x)
val spark = SparkSession.builder()
  .config(conf)
  .enableHiveSupport() // if needed

From OpenHands/extensions, MIT.

More from OpenHands/extensions