由于scrapy,我正在嘗試抓取lacentrale.fr 的網站,但即使我輪換了我的用戶代理和IP 地址(感謝TOR),該網站也會檢測到我的機器人并向我發送錯誤值。請您檢查我在中間件和設置中使用的代碼,并告訴我是否出現問題。中間件中的代碼:from tutorial.settings import * #USER_AGENT_LISTimport randomfrom stem.control import Controllerfrom toripchanger import TorIpChangerfrom stem import Signalclass RandomUserAgentMiddleware(object):def process_request(self, request, spider): ua = random.choice(USER_AGENT_LIST) if ua: request.headers.setdefault('User-Agent', ua)def _set_new_ip(): with Controller.from_port(port=9051) as controller: controller.authenticate(password='') controller.signal(Signal.NEWNYM)ip_changer = TorIpChanger(reuse_threshold=10)class ProxyMiddleware(object): _requests_count = 0def process_request(self, request, spider): self._requests_count += 1 if self._requests_count > 10: self._requests_count = 0 ip_changer.get_new_ip() print("New Tor connection processed") request.meta['proxy'] = 'http://127.0.0.1:8118' spider.log('Proxy : %s' % request.meta['proxy'])設置中使用的代碼:BOT_NAME = 'tutorial'SPIDER_MODULES = ['tutorial.spiders']NEWSPIDER_MODULE = 'tutorial.spiders'ROBOTSTXT_OBEY = TrueDOWNLOAD_DELAY = 1RANDOMIZE_DOWNLOAD_DELAY = TrueDOWNLOADER_MIDDLEWARES = { 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, 'tutorial.middlewares.RandomUserAgentMiddleware': 400, 'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware':None, # to avoid the raise IOError, 'Not a gzipped file' exceptions.IOError: Not a gzipped file 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110, 'tutorial.middlewares.ProxyMiddleware': 100 }
1 回答

搖曳的薔薇
TA貢獻1793條經驗 獲得超6個贊
您可能會因多種因素被檢測到,包括您的抓取工具是否下載/運行 javascript 文件。如果是這種情況,您可能需要將 selenium 之類的工具與 Python/Scrapy 結合使用,以進一步偽裝成普通人類用戶。
我認為我無法為您的 TOR 設置可能出現的問題提供太多指導
添加回答
舉報
0/150
提交
取消