需要身份驗證的網站提供搜索服務。搜索包括兩個步驟。首先，從產品序列號檢索基本信息（庫存、尺寸等）的請求。其次，鑒于之前的搜索和幾個附加字段，第二個請求將顯示產品價格。問題是必須以嚴格的順序調用步驟。例如，給定兩個產品A和B，以下序列將產生錯誤 -> basic_info(A), basic_info(B), get_price(A)=> 顯示錯誤，因為服務器期望get_price(B). 鑒于必須進行身份驗證，我不能丟棄 cookie。在下面的場景中，有沒有辦法保證順序請求調用順序？def after_auth_success(self, response): for product in prod_list: yield FormRequest("basic_info_url", ..., calback = self.on_basic_info)def on_basic_info(self, response): yield FormRequest("get_price_url", ..., calback = self.on_price_info)def on_price_info(self, response): #Scrape result... #<price would be scraped correctly only if the requests are done in order> yield result預期結果：Only one thread running the sequence basic_info_url | get_price_url | basic_info_url | get_price_url ...實際結果：If CONCURRENT_REQUEST=1 => Invoke all basic_info_url and after invoke all get_price_url
1 回答
Smart貓小萌
TA貢獻1911條經驗 獲得超7個贊
最后，我找到了一種獲得所需行為的方法。這個想法是進行一種遞歸，最后一步將返回整個結果。為了迭代遞歸，我們使用元屬性共享列表。
result = list()
def after_auth_success(self, response):
first_prod = prod_list.pop(0)
basic_url = build_url("basic_info_url", first_prod)
yield FormRequest(basic_url, meta = {'prod_list': prod_list}, calback = self.on_basic_info)
def on_basic_info(self, response):
yield FormRequest("get_price_url", meta = {'prod_list':response.meta['prod_list']}, calback = self.on_price_info)
def on_price_info(self, response):
#Scrape result and add the result into a dict called node
result.append(result_node)
prod_list = response.meta['prod_list']
if prod_list:
first_prod = prod_list.pop(0)
basic_url = build_url("basic_info_url", first_prod)
yield FormRequest(basic_url, meta = {'prod_list': prod_list}, calback = self.on_basic_info)
else:
yield {'data': result}
添加回答
舉報
0/150
提交
取消