Maintained with ☕️ by
IcePanel logo

Using Dataproc version 2.2 in your Cloud Data Fusion pipeline can fail in some cases with the following error

Share

Services

## Issue Using Dataproc version 2.2 in your Cloud Data Fusion pipeline can fail in some cases with the following error: `ERROR [Driver:o.a.s.d.y.ApplicationMaster@97] - User class threw exception: java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.encoders.ExpressionEncoder org.apache.spark.sql.catalyst.encoders.RowEncoder.apply(org.apache.spark.sql.types.StructType)' at io.cdap.cdap.etl.spark.batch.OpaqueDatasetCollection.toDataframeCollection(OpaqueDatasetCollection.java:111)`. To avoid this issue, [change the Dataproc image to 2.1](https://cloud.google.com/data-fusion/docs/how-to/change-dataproc-image#change-to-dataproc-21) ([CDAP-21075](https://cdap.atlassian.net/browse/CDAP-21075)).