我正在嘗試在 python 中進行并行處理。我有一個包含多4M行的巨大數據框。因此，作為下面給出的示例，我想將 dataframe( ) 劃分為在不同的結果數據幀上df will be divided into df1,df2應用相同的集合。transpose operations感謝 Jezrael 幫助我達到這個水平。請在我的輸入數據框下方找到df = pd.DataFrame({'subject_id':[1,1,1,1,2,2,2,2,3,3,4,4,4,4,4],'readings' : ['READ_1','READ_2','READ_1','READ_3','READ_1','READ_5','READ_6','READ_8','READ_10','READ_12','READ_11','READ_14','READ_09','READ_08','READ_07'],'val' :[5,6,7,11,5,7,16,12,13,56,32,13,45,43,46],})劃分數據框的代碼N=2 # dividing into two dataframes.dfs = [x for _,x in df.groupby(pd.factorize(df['subject_id'])[0] // N)] # dfs is an iterable which will have two dataframes并行處理代碼import multiprocessing as mppool = mp.Pool(mp.cpu_count())results = []def transpose_ope(df): #this function does the transformation like I want df_op = (df.groupby(['subject_id','readings'])['val'] .describe() .unstack() .swaplevel(0,1,axis=1) .reindex(df['readings'].unique(), axis=1, level=0)) df_op.columns = df_op.columns.map('_'.join) df_op = df_op.reset_index()results.append(pool.map(transpose_ope, [df for df in dfs])) # am I storing the output correctly here?實際上，我想將每個階段的輸出附加到主數據框。你能幫我做這件事嗎？即使只有 10-15 條記錄，我的代碼也會繼續運行
1 回答
波斯汪
TA貢獻1811條經驗 獲得超4個贊
你在map中使用的函數需要返回你想要的對象。
我還將使用可用于池的更慣用的上下文管理器。
編輯：固定導入
import multiprocessing as mp
def transpose_ope(df): #this function does the transformation like I want
df_op = (df.groupby(['subject_id','readings'])['val']
.describe()
.unstack()
.swaplevel(0,1,axis=1)
.reindex(df['readings'].unique(), axis=1, level=0))
df_op.columns = df_op.columns.map('_'.join)
df_op = df_op.reset_index()
return df_op
def main():
with mp.Pool(mp.cpu_count()) as pool:
res = pool.map(transpose_ope, [df for df in dfs])
if __name__=='__main__':
main()
不知道為什么要將單個列表附加到另一個列表...但是如果您只想要 [transformed(df) for df in dfs] 的最終列表，map 只會返回該列表。
添加回答
舉報
0/150
提交
取消