我試圖根據其類別刮擦黃頁。因此，我從文本文件加載類別并將其提供給start_urls。我在這里面臨的問題是為每個類別單獨保存輸出。以下是我試圖實現的代碼：CATEGORIES = []with open('Catergories.txt', 'r') as f: data = f.readlines() for category in data: CATEGORIES.append(category.strip())在 settings.py 中打開文件，并在蜘蛛中列出要訪問的列表。蜘蛛：# -*- coding: utf-8 -*-from scrapy.linkextractors import LinkExtractorfrom scrapy.spiders import CrawlSpider, Rulefrom ..items import YellowItemfrom scrapy.utils.project import get_project_settingssettings = get_project_settings()class YpSpider(CrawlSpider): categories = settings.get('CATEGORIES') name = 'yp' allowed_domains = ['yellowpages.com'] start_urls = ['https://www.yellowpages.com/search?search_terms={0}&geo_location_terms=New%20York' '%2C ' '%20NY'.format(*categories)] rules = ( Rule(LinkExtractor(restrict_xpaths='//a[@class="business-name"]', allow=''), callback='parse_item', follow=True), Rule(LinkExtractor(restrict_xpaths='//a[@class="next ajax-page"]', allow=''), follow=True), ) def parse_item(self, response): categories = settings.get('CATEGORIES') print(categories) item = YellowItem() # for data in response.xpath('//section[@class="info"]'): item['title'] = response.xpath('//h1/text()').extract_first() item['phone'] = response.xpath('//p[@class="phone"]/text()').extract_first() item['street_address'] = response.xpath('//h2[@class="address"]/text()').extract_first() email = response.xpath('//a[@class="email-business"]/@href').extract_first() try: item['email'] = email.replace("mailto:", '') except AttributeError: pass
2 回答
繁華開滿天機
TA貢獻1816條經驗 獲得超4個贊
我會在后期處理中這樣做。將所有項目導出到一個帶有類別字段.csv文件。我認為你沒有以正確的方式思考這個問題，并使其過于復雜。不確定這是否有效，但它值得一試:)
with open('parent.csv', 'r') as file:
reader = csv.reader(file)
for row in reader:
with open('{}.csv'.format(row[category]), 'a') as f:
writer = csv.writer(f)
writer.writerow(row)
您也可以使用蜘蛛閉合信號應用此代碼。
https://docs.scrapy.org/en/latest/topics/signals.html#scrapy.signals.spider_closed
青春有我
TA貢獻1784條經驗 獲得超8個贊
dict.items()返回可迭代，其中每個項目看起來像 要擺脫此錯誤，您需要刪除并解壓縮該項目，例如tuple (key, value)iterfor category, exporter in self.exporter.items():
添加回答
舉報
0/150
提交
取消