亚洲在线久爱草,狠狠天天香蕉网,天天搞日日干久草,伊人亚洲日本欧美

為了賬號安全,請及時綁定郵箱和手機立即綁定
已解決430363個問題,去搜搜看,總會有你想問的

pyspark 使用正則表達式搜索關鍵字,然后加入其他數據框

pyspark 使用正則表達式搜索關鍵字,然后加入其他數據框

一只名叫tom的貓 2023-02-22 13:53:43
我有兩個數據框數據幀Aname       groceries Mike       apple, orange, banana, noodle, red wineKate       white wine, green beans, extra pineapple hawaiian pizzaLeah       red wine, juice, rice, grapes, green beansBen        water, spaghetti數據幀Bid       item0001     red wine0002     green beans我逐行瀏覽 B,并使用正則表達式搜索數據框 A 的雜貨店中是否存在項目df = Nonefor keyword in B.select('item').rdd.flatMap(lambda x : x).collect():    if keyword == None:        continue    pattern = '(?i)^'    start = '(?=.*\\b'    end = '\\b)'    for word in re.split('\\s+', keyword):        pattern = pattern + start + word + end    pattern = pattern + '.*$'        if df == None:        df = A.filter(A['groceries'].rlike(pattern)).withColumn('item', F.lit(keyword))    else:        df = df.unionAll(A.filter(A['groceries'].rlike(pattern)).withColumn('item', F.lit(keyword)))我想要的輸出是 A 中的行,其中包含 B 中的項目,但也將 item 關鍵字作為新列插入name       groceries                                                     itemMike       apple, orange, banana, noodle, red wine                       red wineLeah       red wine, juice, rice, grapes, green beans                    red wineKate       white wine, green beans, extra pineapple hawaiian pizza       green beansLeah       red wine, juice, rice, grapes, green beans                    green beans實際輸出不是我想要的,我不明白這種方法有什么不對。我還想知道是否有一種方法可以使用 rlike 直接連接 A 和 B,這樣只有當 A 中的項目存在于 B 的雜貨店中時,行才會連接。謝謝!
查看完整描述

1 回答

?
慕尼黑的夜晚無繁華

TA貢獻1864條經驗 獲得超6個贊

使用 F.expr() 可以進行類連接。在您的情況下,您需要將它與內部聯接一起使用。嘗試這個,


    #%%

import pyspark.sql.functions as F

test1 =sqlContext.createDataFrame([("Mike","apple,greenbeans,redwine,the little prince 70th anniversary gift set (book/cd/downloadable audio)" ),("kate","Whitewine,greenbeans,pineapple"),("Ben","Water,Spaghetti")],schema=["name","groceries"])

test2 = sqlContext.createDataFrame([("001","redwine"),("002","greenbeans"),("003","cd")],schema=["id","item"])

#%%

test_join =test1.join(test2,F.expr("""groceries rlike item"""),how='inner')

結果:


 test_join.show(truncate=False)

   +----+-------------------------------------------------------------------------------------------------+---+----------+

|name|groceries                                                                                        |id |item      |

+----+-------------------------------------------------------------------------------------------------+---+----------+

|Mike|apple,greenbeans,redwine,the little prince 70th anniversary gift set (book/cd/downloadable audio)|001|redwine   |

|Mike|apple,greenbeans,redwine,the little prince 70th anniversary gift set (book/cd/downloadable audio)|002|greenbeans|

|Mike|apple,greenbeans,redwine,the little prince 70th anniversary gift set (book/cd/downloadable audio)|003|cd        |

|kate|Whitewine,greenbeans,pineapple                                                                   |002|greenbeans|

+----+-------------------------------------------------------------------------------------------------+---+----------+

對于您的復雜數據集,contains() 函數必須有效


import pyspark.sql.functions as F

test1 = spark.createDataFrame([("Mike","apple, oranges, red wine,green beans"),("Kate","Whitewine, green beans waterrr, pineapple, red wine"), ("Leah", "red wine, juice, rice, grapes, green beans"),("Ben","Water,Spaghetti, the little prince 70th anniversary gift set (book/cd/downloadable audio)")],schema=["name","groceries"])

test2 = spark.createDataFrame([("001","red wine"),("002","green beans waterrr"), ("003", "the little prince 70th anniversary gift set (book/cd/downloadable audio)")],schema=["id","item"])

#%%

test_join =test1.join(test2,F.col('groceries').contains(F.col('item')),how='inner')

結果:


+----+-----------------------------------------------------------------------------------------+---+------------------------------------------------------------------------+

|name|groceries                                                                                |id |item                                                                    |

+----+-----------------------------------------------------------------------------------------+---+------------------------------------------------------------------------+

|Mike|apple, oranges, red wine,green beans                                                     |001|red wine                                                                |

|Kate|Whitewine, green beans waterrr, pineapple, red wine                                      |001|red wine                                                                |

|Kate|Whitewine, green beans waterrr, pineapple, red wine                                      |002|green beans waterrr                                                     |

|Leah|red wine, juice, rice, grapes, green beans                                               |001|red wine                                                                |

|Ben |Water,Spaghetti, the little prince 70th anniversary gift set (book/cd/downloadable audio)|003|the little prince 70th anniversary gift set (book/cd/downloadable audio)|

+----+-----------------------------------------------------------------------------------------+---+------------------------------------------------------------------------+



查看完整回答
反對 回復 2023-02-22
  • 1 回答
  • 0 關注
  • 104 瀏覽
慕課專欄
更多

添加回答

舉報

0/150
提交
取消
微信客服

購課補貼
聯系客服咨詢優惠詳情

幫助反饋 APP下載

慕課網APP
您的移動學習伙伴

公眾號

掃描二維碼
關注慕課網微信公眾號