我正在嘗試從https://www.msamb.com/ApmcDetail/ArrivalPriceInfo網站抓取數據。這是我要抓取的數據。所以，在高亮的下拉選擇框中有148個商品。截至目前，我正在通過選擇每個單獨的商品來手動復制數據。這需要大量的手動工作來提取數據。所以，為了讓它自動化，我開始使用 Python。以下是我在 Python (3.7.8) 代碼中使用的庫。硒美湯熊貓這是我的 Python 代碼。from selenium import webdriverfrom bs4 import BeautifulSoupimport pandas as pdfrom selenium.webdriver.support.ui import Select#from selenium.webdriver.common.by import Bydriver = webdriver.Chrome(executable_path='G:/data/depend/chromedriver.exe')driver.get('https://www.msamb.com/ApmcDetail/ArrivalPriceInfo/')commodity = Select(driver.find_element_by_id("CommoditiesId"))#able to select commodities by valuecommodity.select_by_value('08005')# Iterating over the all the commodity an fetching <td> elementfor option in commodity.options: #print(option.text) soup = BeautifulSoup(option.text) print(soup) rows = soup.select('tr') print(rows) for row in rows[1:]: td = row.find_all('td') print(td) APMC = td[0].text.strip() print(APMC)在這里，我可以從下拉選擇框中通過等于CommoditiesId的id獲取商品。獲取商品列表 (148) 后，我將嘗試解析為該特定商品獲取的 HTML 表格內容。在這里，我能夠為每次迭代打印商品名稱，但無法打印APMC 、Variety、Unit、Quantity、Lrate、Hrate、Modal列數據。如果以上解決了，我想要分隔格式的輸出~|~并想要添加兩列，即Date, Commodity。因此，示例輸出將如下所示（截至目前，手動準備以下數據文件）。Date~|~Commodity~|~APMC~|~Variety~|~Unit~|~Quantity~|~Lrate~|~Hrate~|~Modal 2020-07-11~|~APPLE~|~KOLHAPUR~|~QUINTAL~|~17~|~8500~|~14500~|~11500 2020-07-11~|~APPLE~|~CHANDRAPUR-GANJWAD~|~QUINTAL~|~9~|~15000~|~17000~|~16000 2020-07-11~|~APPLE~|~NASHIK~|~DILICIOUS- No.1~|~QUINTAL~|~60~|~9500~|~16000~|~13000 2020-07-11~|~AMBAT CHUKA~|~PANDHARPUR~|~~|~NAG~|~7~|~10~|~10~|~10 2020-07-10~|~AMBAT CHUKA~|~PUNE-MANJRI~|~~|~NAG~|~400~|~3~|~6~|~4 2020-07-10~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~1300~|~4~|~5~|~4
2 回答
交互式愛情
TA貢獻1712條經驗 獲得超3個贊
此腳本將遍歷所有頁面并將它們保存到標準 csv 和~|~分隔文本文件中：
import requests
import numpy as np
import pandas as pd
from bs4 import BeautifulSoup
url = 'https://www.msamb.com/ApmcDetail/ArrivalPriceInfo'
detail_url = 'https://www.msamb.com/ApmcDetail/DataGridBind?commodityCode={code}&apmcCode=null'
headers = {'Referer': 'https://www.msamb.com/ApmcDetail/ArrivalPriceInfo'}
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
values = [(o['value'], o.text) for o in soup.select('#CommoditiesId option') if o['value']]
all_data = []
for code, code_name in values:
print('Getting info for code {} {}'.format(code, code_name))
soup = BeautifulSoup(requests.get(detail_url.format(code=code), headers=headers).content, 'html.parser')
current_date = ''
for row in soup.select('tr'):
if row.select_one('td[colspan]'):
current_date = row.get_text(strip=True)
else:
row = [td.get_text(strip=True) for td in row.select('td')]
all_data.append({
'Date': current_date,
'Commodity': code_name,
'APMC': row[0],
'Variety': row[1],
'Unit': row[2],
'Quantity': row[3],
'Lrate': row[4],
'Hrate': row[5],
'Modal': row[6],
})
df = pd.DataFrame(all_data)
print(df)
df.to_csv('data.csv') # <-- saves standard csv
np.savetxt('data.txt', df, delimiter='~|~', fmt='%s') # <-- saves .txt file with '~|~' delimiter
印刷：
...
Getting info for code 08071 TOMATO
Getting info for code 10006 TURMERIC
Getting info for code 08075 WAL BHAJI
Getting info for code 08076 WAL PAPDI
Getting info for code 08077 WALVAD
Getting info for code 07011 WATER MELON
Getting info for code 02009 WHEAT(HUSKED)
Getting info for code 02012 WHEAT(UNHUSKED)
Date Commodity APMC Variety Unit Quantity Lrate Hrate Modal
0 18/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 50 5 5 5
1 16/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 50 5 5 5
2 15/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 100 9 9 9
3 13/07/2020 AMBAT CHUKA PANDHARPUR ---- NAG 16 7 7 7
4 13/07/2020 AMBAT CHUKA PUNE LOCAL NAG 2400 4 7 5
... ... ... ... ... ... ... ... ... ...
4893 12/07/2020 WHEAT(HUSKED) SHIRUR No. 2 QUINTAL 2 1400 1400 1400
4894 17/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 863 4000 4600 4300
4895 16/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 475 4000 4500 4250
4896 15/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 680 3900 4400 4150
4897 13/07/2020 WHEAT(UNHUSKED) SANGLI-MIRAJ ---- QUINTAL 1589 3900 4450 4175
[4898 rows x 9 columns]
節省data.txt：
0~|~18/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~50~|~5~|~5~|~5
1~|~16/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~50~|~5~|~5~|~5
2~|~15/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~100~|~9~|~9~|~9
3~|~13/07/2020~|~AMBAT CHUKA~|~PANDHARPUR~|~----~|~NAG~|~16~|~7~|~7~|~7
4~|~13/07/2020~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~2400~|~4~|~7~|~5
5~|~12/07/2020~|~AMBAT CHUKA~|~PUNE~|~LOCAL~|~NAG~|~1700~|~3~|~8~|~5
6~|~19/07/2020~|~APPLE~|~KOLHAPUR~|~----~|~QUINTAL~|~3~|~9000~|~14000~|~11500
7~|~18/07/2020~|~APPLE~|~KOLHAPUR~|~----~|~QUINTAL~|~12~|~8500~|~15000~|~11750
8~|~18/07/2020~|~APPLE~|~NASHIK~|~DILICIOUS- No.1~|~QUINTAL~|~110~|~9000~|~16000~|~13000
9~|~18/07/2020~|~APPLE~|~SANGLI-PHALE BHAJIPALAM~|~LOCAL~|~QUINTAL~|~8~|~12000~|~16000~|~14000
10~|~17/07/2020~|~APPLE~|~MUMBAI-FRUIT MARKET~|~----~|~QUINTAL~|~264~|~9000~|~12000~|~10500
...
來自 LibreOffice 的 csv 文件的屏幕截圖：
慕后森
TA貢獻1802條經驗 獲得超5個贊
您可以將它們保存到 txt 文件中，您可以這樣做df = pd.read_csv("out.txt",delimiter='~|~')，或者
date = df['Date'] commodity = df['Commodity']
您可以將 apmc 附加到列表中，并在最后附加 read_as 數據框。
添加回答
舉報
0/150
提交
取消