我試圖將數據從PostgreSQL表中的表移動到HDFS上的Hive表。為此，我想出了以下代碼： val conf = new SparkConf().setAppName("Spark-JDBC").set("spark.executor.heartbeatInterval","120s").set("spark.network.timeout","12000s").set("spark.sql.inMemoryColumnarStorage.compressed", "true").set("spark.sql.orc.filterPushdown","true").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.max","512m").set("spark.serializer", classOf[org.apache.spark.serializer.KryoSerializer].getName).set("spark.streaming.stopGracefullyOnShutdown","true").set("spark.yarn.driver.memoryOverhead","7168").set("spark.yarn.executor.memoryOverhead","7168").set("spark.sql.shuffle.partitions", "61").set("spark.default.parallelism", "60").set("spark.memory.storageFraction","0.5").set("spark.memory.fraction","0.6").set("spark.memory.offHeap.enabled","true").set("spark.memory.offHeap.size","16g").set("spark.dynamicAllocation.enabled", "false").set("spark.dynamicAllocation.enabled","true").set("spark.shuffle.service.enabled","true") val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate() def prepareFinalDF(splitColumns:List[String], textList: ListBuffer[String], allColumns:String, dataMapper:Map[String, String], partition_columns:Array[String], spark:SparkSession): DataFrame = { val colList = allColumns.split(",").toList val (partCols, npartCols) = colList.partition(p => partition_columns.contains(p.takeWhile(x => x != ' '))) val queryCols = npartCols.mkString(",") + ", 0 as " + flagCol + "," + partCols.reverse.mkString(",") val execQuery = s"select ${allColumns}, 0 as ${flagCol} from schema.tablename where period_year='2017' and period_num='12'" 數據將插入到基于以下內容動態分區的配置單元表中 prtn_String_columns: source_system_name, period_year, period_num
3 回答
哈士奇WWW
TA貢獻1799條經驗 獲得超6個贊
以我的經驗，有4種不同的內存設置：
A）[1]用于處理數據的存儲器，用于VS [2]用于保存程序堆棧的堆空間
B）[1]驅動程序VS [2]執行程序內存
到目前為止，通過增加適當的內存種類，我始終能夠使我的Spark作業成功運行：
因此，A2-B1將在驅動程序上有可用的內存來保存程序堆棧。等等。
屬性名稱如下：
A1-B1） executor-memory
A1-B2） driver-memory
A2-B1） spark.yarn.executor.memoryOverhead
A2-B2） spark.yarn.driver.memoryOverhead
請記住，所有* -B1的總和必須小于工作線程上的可用內存，而所有* -B2的總和必須小于驅動程序節點上的內存。
我敢打賭，罪魁禍首是加粗標記的堆設置之一。
添加回答
舉報
0/150
提交
取消