我有一個復雜tuple類型的 RDD，比如(20190701, [11,21,31], [('A',10), ('B', 20)])schema 可以自己定義。那么如何把它變成一個DataFrame，像這樣：date | 0 | 1 | 2 | A | B
20190701 | 11 | 21 | 31 | 10 | 20
2 回答
MMTTMM
TA貢獻1869條經驗 獲得超4個贊
單程：
from pyspark.sql import Row
rdd = sc.parallelize([(20190701, [11,21,31], [('A',10), ('B', 20)])])
# customize a Row class based on schema
MRow = Row("date", "0", "1", "2", "A", "B")
rdd.map(lambda x: MRow(x[0], *x[1], *map(lambda e:e[1],x[2]))).toDF().show()
+--------+---+---+---+---+---+
| date| 0| 1| 2| A| B|
+--------+---+---+---+---+---+
|20190701| 11| 21| 31| 10| 20|
+--------+---+---+---+---+---+
或者另一種方式：
rdd.map(lambda x: Row(date=x[0], **dict((str(i), e) for i,e in list(enumerate(x[1])) + x[2]))).toDF().show()
+---+---+---+---+---+--------+
| 0| 1| 2| A| B| date|
+---+---+---+---+---+--------+
| 11| 21| 31| 10| 20|20190701|
+---+---+---+---+---+--------+
一只甜甜圈
TA貢獻1836條經驗 獲得超5個贊
rdd = sc.parallelize((20190701, [11,21,31], [('A',10), ('B', 20)]))
elements = rdd.take(3)
a = [elements[0]] + (elements[1]) + [elements[2][0][1], elements[2][1][1]]
import pandas as pd
sdf = spark.createDataFrame(pd.DataFrame([20190701, 11, 21, 31, 10, 20]).T)
添加回答
舉報
0/150
提交
取消