我正在嘗試像下面這樣劃分數據框：from io import StringIOimport pandas as pddata = """A,B,C87jg,28,3012h372,28,3011kj87,27,30112yh8,54,3010802h,53,30105d8b,52,3010"""df = pd.read_csv(StringIO(data), sep=',')for key, group in df.groupby(['C','B']): group.to_csv(f'df_{key}.csv', index=False)這會將按數據幀分組的結果導出到本地機器。有沒有辦法執行此操作并將這些多個拆分 csv 上傳到 s3（類似于 boto3 的 put_object）
2 回答
人到中年有點甜
TA貢獻1895條經驗 獲得超7個贊
您也可以使用必須安裝的s3fs ?？梢允褂?完成安裝pip，例如：
pip install s3fs
根據您的代碼驗證的示例：
import os
from io import StringIO
import pandas as pd
import s3fs
# I did not use my default aws profile
# so had to provide key and secret. If you use
# the default aws profile, providing `key`
# and `secret` should not be required
fs = s3fs.S3FileSystem(
? ? ? ? anon=False,
? ? ? ? key='<access_key>',
? ? ? ? secret='<secret_key>')
data = """?
A,B,C
87jg,28,3012
h372,28,3011
kj87,27,3011
2yh8,54,3010
802h,53,3010
5d8b,52,3010
"""
df = pd.read_csv(StringIO(data), sep=',')
for key, group in df.groupby(['C','B']):
? ? group.to_csv(fs.open(f's3://<bucket-name>/df_{key[0]}-M{key[1]}.csv', 'w'), index=False)
代碼正確上傳文件：
一只萌萌小番薯
TA貢獻1795條經驗 獲得超7個贊
from io import StringIO
import pandas as pd
import boto3
data = """
A,B,C
87jg,28,3012
h372,28,3011
kj87,27,3011
2yh8,54,3010
802h,53,3010
5d8b,52,3010
"""
df = pd.read_csv(StringIO(data), sep=',')
client = boto3.client('s3')
for key, group in df.groupby(['C', 'B']):
group.to_csv(f'df_{key}.csv', index=False)
client.upload_file(f'df_{key}.csv', 'my-another-test-bucket-2',
f'df_{key[0]}-M{key[1]}.csv')
S3 桶
添加回答
舉報
0/150
提交
取消