在這段注釋中的代碼中，正確顯示了列表緩沖區項的長度，但是在第二注釋中，代碼從未執行。為什么會發生？val conf = new SparkConf().setAppName("app").setMaster("local")val sc = new SparkContext(conf)var wktReader: WKTReader = new WKTReader(); val dataSet = sc.textFile("dataSet.txt")val items = new ListBuffer[String]() dataSet.foreach { e => items += e println("len = " + items.length) //1. here length is ok}println("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")items.foreach { x => print(x)} //2. this code doesn't be executed日志在這里：16/11/20 01:16:52 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/11/20 01:16:52 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.56.1:4040 16/11/20 01:16:53 INFO Executor: Starting executor ID driver on host localhost 16/11/20 01:16:53 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58608. 16/11/20 01:16:53 INFO NettyBlockTransferService: Server created on 192.168.56.1:58608 16/11/20 01:16:53 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.56.1, 58608) 16/11/20 01:16:53 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.56.1:58608 with 347.1 MB RAM, BlockManagerId(driver, 192.168.56.1, 58608) 16/11/20 01:16:53 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.56.1, 58608) Starting app
2 回答
萬千封印
TA貢獻1891條經驗 獲得超3個贊
因此，Apache Spark不提供共享內存：
dataSet.foreach { e =>
items += e
println("len = " + items.length) //1. here length is ok
}
您可以在各自的執行者上修改的本地副本items。items驅動程序上定義的原始列表未修改。結果是：
items.foreach { x => print(x) }
執行，但沒有要打印的內容。
請檢查了解關閉
雖然這里建議這樣做，但是您可以用累加器代替物品
val acc = sc.collectionAccumulator[String]("Items")
dataSet.foreach(e => acc.add(e))
動漫人物
TA貢獻1815條經驗 獲得超10個贊
Spark在執行程序中運行并返回結果。上面的代碼無法正常工作。如果您需要從中添加元素，foreach則需要在驅動程序中收集數據并將其添加到中current_set。但是，當您擁有大數據時，收集數據不是一個好主意。
val items = new ListBuffer[String]()
val rdd = spark.sparkContext.parallelize(1 to 10, 4)
rdd.collect().foreach(data => items += data.toString())
println(items)
輸出：
ListBuffer(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
添加回答
舉報
0/150
提交
取消