我是 spark 的新手，我想知道如何進行字符串操作，以便 Column1 - Column2 獲得 column3。注意：我的數據在數據框中所以基本上我有兩個不同的列字符串，我只想獲取 column2 中存在但不在 column 1 中的字符串，以便我可以將其生成為 column3Column1SAMPLE_OUT_3_APPLE|BANANA|GUAVA|ORANGEColumn2SAMPLE_OUT_3_APPLE|BANANA|GUAVA|GRAPES|ORANGE|BERRY那么 Column3 應該是...Column3GRAPES,BERRY但是對于 column1 和 column2 我也想展示APPLE,BANANA,ORANGE 只需刪除SAMPLE_OUT_3并用逗號分隔
3 回答
Cats萌萌
TA貢獻1805條經驗 獲得超9個贊
你可以用'|'分割你的專欄 像下面導入 spark.implicits._
val df = mainDf.select("Column1","Column2").map(x => {
val s1 = x.getAsString(0).replaceAll("^.*3_","").split("|");
val s2 = x.getAsString(1).replaceAll("^.*3_","").split("|");
(x.getAsString(0),x.getAsString(1),s2.diff(s1).union(s1.diff(s2)))
}
).toDF("Column1","Column2","Column3")
精慕HU
TA貢獻1845條經驗 獲得超8個贊
你也可以通過regexp_replace和udf來達到你的目的。
regexp_replace 替換“|” 用“，”和“。* 3_”用“”
udf從column2和column1獲取column3的值
val df1 = Seq(("SAMPLE_OUT_3_APPLE|BANANA|GUAVA|ORANGE" ,"SAMPLE_OUT_3_APPLE|BANANA|GUAVA|GRAPES|ORANGE|BERRY")).toDF("column1","column2")
val df2 =df1.columns.foldLeft(df) { (memoDF, colName) =>
memoDF.withColumn(
colName,
regexp_replace(regexp_replace(col(colName), "\\|", ","),".*3_",""))}
val diff_udf = udf { ( a: String, b: String) => (a.split(",") diff b.split(",")).mkString(",") }
df2.withColumn("column3", diff_udf(col("column2"), col("column1"))).show(false)
輸出：
+-------------------------+--------------------------------------+------------+
|column1 |column2 |column3 |
+-------------------------+--------------------------------------+------------+
|APPLE,BANANA,GUAVA,ORANGE|APPLE,BANANA,GUAVA,GRAPES,ORANGE,BERRY|GRAPES,BERRY|
+-------------------------+--------------------------------------+------------+
慕村9548890
TA貢獻1884條經驗 獲得超4個贊
對于Spark >= 2.4
您可以使用array_except
import spark.implicits._
val df = Seq(
("SAMPLE_OUT_3_APPLE|BANANA|GUAVA|ORANGE" ,"SAMPLE_OUT_3_APPLE|BANANA|GUAVA|GRAPES|ORANGE|BERRY")
).toDF("column1", "column2")
val remove = df.columns.map(column => split(col(column), "3_").getItem(1).as(column))
val resultDF = df.select(remove: _*)
.withColumn("column1", split($"column1", "\\|"))
.withColumn("column2", split($"column2", "\\|"))
.withColumn("column3", array_except($"column2", $"column1"))
.withColumn("column1", array_except($"column1", $"column3"))
.withColumn("column2", array_except($"column2", $"column3"))
val convertToString = resultDF.columns.map(column => concat_ws("|", col(column)).as(column))
resultDF.select(convertToString: _*).show(false)
輸出：
+-------------------------+-------------------------+------------+
|column1 |column2 |column3 |
+-------------------------+-------------------------+------------+
|APPLE|BANANA|GUAVA|ORANGE|APPLE|BANANA|GUAVA|ORANGE|GRAPES|BERRY|
+-------------------------+-------------------------+------------+
添加回答
舉報
0/150
提交
取消