作為可重現的示例，我有以下數據集：import pandas as pdimport numpy as npfrom sklearn.model_selection import train_test_splitdata = np.random.randint(0,20,size=(300, 5))df = pd.DataFrame(data, columns=['ID', 'A', 'B', 'C', 'D'])df = df.set_index(['ID'])df.head()Out: A B C DID 12 3 14 4 79 5 9 8 412 18 17 3 141 0 10 1 09 10 5 11 9我需要執行 70%-30% 的分層分割（在 y 上），我知道它看起來像這樣：# Train/Test SplitX = df.iloc[:,0:-1] # Columns A, B, and Cy = df.iloc[:,-1] # Column DX_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.70, test_size = 0.30, stratify = y)然而，雖然我希望訓練和測試集具有相同（或足夠相似）的“D”分布，但我不希望測試和訓練中都存在相同的“ID”。我怎么能這樣做呢？
1 回答
飲歌長嘯
TA貢獻1951條經驗 獲得超3個贊
編輯：執行（類似）您要求的操作的一種方法可能是按類別存儲 ID，然后對于每個類別，獲取 70% 的 ID，并將具有這些 ID 的樣本插入到訓練中，其余的插入到測試集中。
請注意，如果每個 ID 出現的次數不同，這仍然不能保證分布相同。此外，鑒于每個 ID 可以屬于 D 中的多個類，并且不應在訓練集和測試集之間共享，因此尋求相同的分布成為一個復雜的優化問題。這是因為每個 ID 只能包含在train或test中，同時向分配的集合添加可變數量的類，這取決于給定 ID 在其出現的所有行中所具有的類。
在近似平衡分布的同時分割數據的一種相當簡單的方法是按隨機順序迭代類，并僅考慮每個 ID 出現的其中一個類，因此將其分配給其所有行進行訓練/測試，因此為以后的課程刪除它。
我發現將 ID 視為一列有助于完成此任務，因此我更改了您提供的代碼，如下所示：
# Given snippet (modified)
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
data = np.random.randint(0,20,size=(300, 5))
df = pd.DataFrame(data, columns=['ID', 'A', 'B', 'C', 'D'])
建議的解決方案：
import random
from collections import defaultdict
classes = df.D.unique().tolist() # get unique classes,
random.shuffle(classes) # shuffle to eliminate positional biases
ids_by_class = defaultdict(list)
# iterate over classes
temp_df = df.copy()
for c in classes:
c_rows = temp_df.loc[temp_df['D'] == c] # rows with given class
ids = temp_df.ID.unique().tolist() # IDs in these rows
ids_by_class[c].extend(ids)
# remove ids so they cannot be taken into account for other classes
temp_df = temp_df[~temp_df.ID.isin(ids)]
# now construct ids split, class by class
train_ids, test_ids = [], []
for c, ids in ids_by_class.items():
random.shuffle(ids) # shuffling can eliminate positional biases
# split the IDs
split = int(len(ids)*0.7) # split at 70%
train_ids.extend(ids[:split])
test_ids.extend(ids[split:])
# finally use the ids in train and test to get the
# data split from the original df
train = df.loc[df['ID'].isin(train_ids)]
test = df.loc[df['ID'].isin(test_ids)]
讓我們測試一下分割比大致符合 70/30，數據被保留并且訓練和測試數據幀之間沒有共享 ID：
# 1) check that elements in Train are roughly 70% and Test 30% of original df
print(f'Numbers of elements in train: {len(train)}, test: {len(test)}| Perfect split would be train: {int(len(df)*0.7)}, test: {int(len(df)*0.3)}')
# 2) check that concatenating Train and Test gives back the original df
train_test = pd.concat([train, test]).sort_values(by=['ID', 'A', 'B', 'C', 'D']) # concatenate dataframes into one, and sort
sorted_df = df.sort_values(by=['ID', 'A', 'B', 'C', 'D']) # sort original df
assert train_test.equals(sorted_df) # check equality
# 3) check that the IDs are not shared between train/test sets
train_id_set = set(train.ID.unique().tolist())
test_id_set = set(test.ID.unique().tolist())
assert len(train_id_set.intersection(test_id_set)) == 0
輸出示例：
Numbers of elements in train: 209, test: 91| Perfect split would be train: 210, test: 90
Numbers of elements in train: 210, test: 90| Perfect split would be train: 210, test: 90
Numbers of elements in train: 227, test: 73| Perfect split would be train: 210, test: 90
添加回答
舉報
0/150
提交
取消