我們正在嘗試刪除異常值，但出現了無限循環對于一個學校項目，我們（我和一個朋友）認為創建一個基于數據科學的工具是個好主意。為此，我們開始清理數據庫（我不會在這里導入它，因為它太大（xlsx 文件、csv 文件））。我們現在嘗試使用“duration_分鐘”列的“標準差*3 + 平均值”規則刪除異常值。這是我們用來計算標準差和平均值的代碼：def calculateSD(database, column): column = database[[column]] SD = column.std(axis=None, skipna=None, level=None, ddof=1, numeric_only=None) return SDdef calculateMean(database, column): column = database[[column]] mean = column.mean() return mean我們認為要做到以下幾點：#Now we have to remove the outliers using the code from the SD.py and SDfunction.py filesminutes = trainsData['duration_minutes'].tolist() #takes the column duration_minutes and puts it in a listSD = int(calculateSD(trainsData, 'duration_minutes')) #calculates the SD of the columnmean = int(calculateMean(trainsData, 'duration_minutes'))SDhigh = mean+3*SD上面的代碼計算起始值。然后我們啟動一個 while 循環來刪除異常值。刪除異常值后，我們重新計算標準差、均值和 SDhigh。這是 while 循環：while np.any(i >= SDhigh for i in minutes): #used to be >=, it doesnt matter for the outcome trainsData = trainsData[trainsData['duration_minutes'] < SDhigh] #used to be >=, this caused an infinite loop so I changed it to <=. Then to < minutes = trainsData['duration_minutes'].tolist() SD = int(calculateSD(trainsData, 'duration_minutes')) #calculates the SD of the column mean = int(calculateMean(trainsData, 'duration_minutes')) SDhigh = mean+3*SD print(SDhigh) #to see how the values changed and to confirm it is an infinite loop輸出如下：611652428354322308300296296296296它繼續打印 296，經過幾個小時的嘗試解決這個問題，我們得出的結論是我們沒有我們希望的那么聰明。
1 回答
呼啦一陣風
TA貢獻1802條經驗 獲得超6個贊
你讓事情變得比原本應該的更加困難。計算標準差以消除異常值，然后重新計算等等過于復雜（并且統計上不合理）。使用百分位數而不是標準差會更好
import numpy as np
import pandas as pd
# create data
nums = np.random.normal(50, 8, 200)
df = pd.DataFrame(nums, columns=['duration'])
# set threshold based on percentiles
threshold = df['duration'].quantile(.95) * 2
# now only keep rows that are below the threshold
df = df[df['duration']<threshold]
添加回答
舉報
0/150
提交
取消