我有一個 Spring Web 應用程序（內置于 Maven），用它連接到我的 Spark 集群（4 個工作節點和 1 個主節點）和我的 cassandra 集群（4 個節點）。應用程序啟動，工作線程與主節點通信，并且 cassandra 集群也在運行。但是，當我通過網絡應用程序的界面進行 PCA（spark mllib）或任何其他計算（聚類、pearson、spearman）時，出現以下錯誤：java.lang.ClassCastException：無法將 scala.collection.immutable.List$SerializationProxy 的實例分配給 scala.collection.Seq 類型的字段 org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_在 org.apache.spark.rdd.MapPartitionsRDD 的實例中這個錯誤肯定是因為 spring-boot-maven-plugin 和 apache Spark 之間的 Jars 沖突。
1 回答
慕娘9325324
TA貢獻1783條經驗 獲得超4個贊
解決了！我只需要更改插件即可。可能是jar包之間有沖突。
這就是我所擁有的：
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.0.9.RELEASE</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
<configuration>
<requiresUnpack>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
</dependency>
</requiresUnpack>
</configuration>
</plugin>
我將其替換為：
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
和
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.0.9.RELEASE</version>
</dependency>
</dependencies>
<configuration>
<keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
<createDependencyReducedPom>true</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.example.app.Mtbapp1</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
添加回答
舉報
0/150
提交
取消