我需要評估 DEM 模擬中的 1800 個數據文件。每個數據文件在某個時間點有效,并包含粒子及其溫度的列表。我想繪制一段時間內粒子子集的平均溫度。不幸的是,在評估過程中一段時間后我的內存不足了。每個數據文件大約有 15 MB。這就是我所做的:import pandas as pdimport numpy as npimport linecacheimport os as osimport gcpath = "E:/Simulationen/35_1100_700/DEM/post/dump/"timesList = [] # create empty list for timeTcentralList = [] # create empty list for temperatures for files in os.walk(os.path.normpath(path)): for file in files[2]: # files is a tuple with a list of filenames in the third element (index 2) of the tuple time = (int(file[0:6])-300000)*0.1+3 # read the timestamps from filenames (first six characters) and convert to time timesList.append(time) # write time to times list for later creation of dataframe # Read the headerline (line 9), write items to column title list coltitles = [sub.replace('[0]','') for sub in linecache.getline(path+file,9).split()[2:]] columns=list(range(0,len(coltitles),1)) # list of columns to read df = pd.read_csv(path+file, sep=' ', skiprows=8, index_col=0, usecols=columns) df.columns = coltitles[1:] df.index.names = [coltitles[0]] T_central = df[df.r.le(0.01) & df.z.ge(0.045) & df.z.lt(0.055)]['f_Temp'].mean(axis=0) # Filter all rows (particles) where radius r is lower/equal than 0.01 m and z is between 0.045 m (greater/equal) and 0.055 m (lower) and average their temperatures # List of average temperatures of central particles for later creation of dataframe TcentralList.append(T_central)我正在讀取路徑中的所有文件。時間是從文件名中獲取的,進行轉換并存儲在列表中 - 我稍后想創建一個帶有“時間”和“溫度”列的數據框。然后,我將數據文件讀取到數據幀并僅過濾中心區域的粒子,然后平均它們的溫度。數據文件有 17 列。我嘗試的第一件事是通過縮短列表“列”來僅讀取必要的列,但這并沒有減少內存使用量。然后我嘗試手動啟動垃圾收集(gc):gc.collect()del dfdel T_central這也沒有幫助。我還嘗試重新初始化 df 和 T_central 以刪除對它們的引用T_central=[]df=pd.DataFrame()但沒有任何效果。我沒主意了。有人給我提示嗎?
在 pandas 中處理大量大數據文件
慕碼人8056858
2023-09-12 17:46:37