Maintained with ☕️ by
IcePanel logo

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions

Share

Services

## Announcement Announcement New [**Managed Service for Apache Spark** (formerly Dataproc on Compute Engine) subminor cluster image versions](https://docs.cloud.google.com/managed-spark/docs/concepts/versioning/image-version-lists#supported-dataproc-image-versions): * 2.1.114-debian11, 2.1.114-rocky8, 2.1.114-ubuntu20, 2.1.114-ubuntu20-arm * 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm * 2.3.30-debian12, 2.3.30-ml-ubuntu22, 2.3.30-rocky9, 2.3.30-ubuntu22, 2.3.30-ubuntu22-arm ## Breaking Breaking **Managed Service for Apache Spark** (formerly Dataproc on Compute Engine): The configuration for Spark shuffle partitions (`spark.sql.shuffle.partitions`) has changed from an integer to a string type. This change impacts image versions `2.3.30` and later in version `2.3`, and `2.2.82` and later in version `2.2`. * **Impact:** This change only affects users who are programmatically setting the configuration in code using `spark.conf.set()` with an integer literal. * **Impacted example:** `spark.conf.set("spark.sql.shuffle.partitions", 100)` * **User action:** Update your code to pass a string literal instead of an integer. * **Example fix:** `spark.conf.set("spark.sql.shuffle.partitions", "100")` * **Not impacted:** Setting the configuration via command-line arguments (e.g., `spark-submit --conf spark.sql.shuffle.partitions=100`), properties files, or Spark SQL commands (`spark.sql("SET spark.sql.shuffle.partitions=100")`) remains unaffected, as these methods naturally parse the input as strings.