用火花-csv編寫單個csv文件我在用https://github.com/databricks/spark-csv，我試圖寫一個CSV，但不能，它是一個文件夾。需要一個Scala函數，它將接受像路徑和文件名這樣的參數，并編寫那個CSV文件。
3 回答
慕森王
TA貢獻1777條經驗 獲得超3個贊
repartition
df .repartition(1)
.write.format("com.databricks.spark.csv")
.option("header", "true")
.save("mydata.csv")coalesce:
df .coalesce(1)
.write.format("com.databricks.spark.csv")
.option("header", "true")
.save("mydata.csv")mydata.csv/part-00000
catgetmerge
HUWWW
TA貢獻1874條經驗 獲得超12個贊
import org.apache.hadoop.conf.Configurationimport org.apache.hadoop.fs._def merge(srcPath: String, dstPath: String): Unit = {
val hadoopConfig = new Configuration()
val hdfs = FileSystem.get(hadoopConfig)
FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), true, hadoopConfig, null)
// the "true" setting deletes the source files once they are merged into the new output}val newData =
<< create your dataframe >>val outputfile = "/user/feeds/project/outputs/subject"
var filename = "myinsights"var outputFileName = outputfile + "/temp_" + filename
var mergedFileName = outputfile + "/merged_" + filenamevar mergeFindGlob = outputFileName
newData.write .format("com.databricks.spark.csv")
.option("header", "false")
.mode("overwrite")
.save(outputFileName)
merge(mergeFindGlob, mergedFileName )
newData.unpersist()
慕妹3242003
TA貢獻1824條經驗 獲得超6個贊
coalesce(1)repartition(1)
FileUtil.copyMerge()
Coalesce()
copyMerge()
添加回答
舉報
0/150
提交
取消