我正在嘗試從一些網站上抓取數據以進行概念驗證項目。目前使用 Python3 和 BS4 來收集所需的數據。我有一本來自三個站點的 URLS 字典。每個站點都需要不同的方法來收集數據，因為它們的 HTML 不同。我一直在使用“Try, If, Else, stack 但我一直遇到問題，如果你能看看我的代碼并幫助我修復它那就太好了！當我添加更多要抓取的站點時，我將無法使用“Try、If、Else”循環通過各種方法來找到抓取數據的正確方法，我如何才能讓這段代碼面向未來添加未來有多少網站并從其中包含的各種元素中抓取數據？# Scraping Script Here:def job():prices = { # LIVEPRICES "LIVEAUOZ": {"url": "https://www.gold.co.uk/", "trader": "Gold.co.uk", "metal": "Gold", "type": "LiveAUOz"}, # GOLD "GLDAU_BRITANNIA": {"url": "https://www.gold.co.uk/gold-coins/gold-britannia-coins/britannia-one-ounce-gold-coin-2020/", "trader": "Gold.co.uk", "metal": "Gold", "type": "Britannia"}, "GLDAU_PHILHARMONIC": {"url": "https://www.gold.co.uk/gold-coins/austrian-gold-philharmoinc-coins/austrian-gold-philharmonic-coin/", "trader": "Gold.co.uk", "metal": "Gold", "type": "Philharmonic"}, "GLDAU_MAPLE": {"url": "https://www.gold.co.uk/gold-coins/canadian-gold-maple-coins/canadian-gold-maple-coin/", "trader": "Gold.co.uk", "metal": "Gold", "type": "Maple"}, # SILVER "GLDAG_BRITANNIA": {"url": "https://www.gold.co.uk/silver-coins/silver-britannia-coins/britannia-one-ounce-silver-coin-2020/", "trader": "Gold.co.uk", "metal": "Silver", "type": "Britannia"}, "GLDAG_PHILHARMONIC": {"url": "https://www.gold.co.uk/silver-coins/austrian-silver-philharmonic-coins/silver-philharmonic-2020/", "trader": "Gold.co.uk", "metal": "Silver", "type": "Philharmonic"}}
1 回答
三國紛爭
TA貢獻1804條經驗 獲得超7個贊
為抓取添加一個配置，其中每個配置都是這樣的：
prices = {
"LIVEAUOZ": {
"url": "https://www.gold.co.uk/",
"trader": "Gold.co.uk",
"metal": "Gold",
"type": "LiveAUOz",
"price": {
"selector": '#id > div > table > tr',
"parser": lambda x: float(re.sub(r"[^0-9\.]", "", x))
}
}
}
使用 price 的選擇器部分獲取 HTML 的相關部分，然后使用解析器函數對其進行解析。
例如
for key, config in prices.items():
response = requests.get(config['url'])
soup = BeautifulSoup(response.text, 'html.parser')
price_element = soup.find(config['price']['selector'])
if price_element:
AG_GRAM_SPOT = price_element.get_text()
# convert to float
AG_GRAM_SPOT = config['price']['parser'](AG_GRAM_SPOT)
# etc
您可以根據需要修改配置對象，但對于大多數站點來說它可能非常相似。例如，文本解析很可能總是相同的，所以不用 lambda 函數，而是用 def 創建一個函數。
def textParser(text):
return float(re.sub(r"[^0-9\.]", "", text))
然后在配置中添加對 textParser 的引用。
prices = {
"LIVEAUOZ": {
"url": "https://www.gold.co.uk/",
"trader": "Gold.co.uk",
"metal": "Gold",
"type": "LiveAUOz",
"price": {
"selector": '#id > div > table > tr',
"parser": textParser
}
}
}
這些步驟將允許您編寫通用代碼，保存所有那些嘗試異常。
添加回答
舉報
0/150
提交
取消