首頁猿問使用 Python 的...

使用 Python 的 requests.post 函數抓取動態 HTML 時如何指定“頁碼”？

Python

慕田峪7331174 2023-12-08 16:46:55

我正在嘗試學習如何使用requestsPython 中的庫使用動態 HTML 抓取網頁。我已經能夠獲取 HTML，但我不知道如何實際獲取不同頁面的數據。如果您注意到圖像底部有頁碼：如何指定頁碼requests.post我如何知道有多少頁可用？它會像頁面范圍然后和一樣簡單try嗎except？我如何知道哪些參數可用于我的data字典requests.post？以下是我訪問請求的方式：import requestsr = requests.post('https://search.aap.org/nicu/', data = {'SearchCriteria.Level':'1', 'X-Requested-With':'XMLHttpRequest'}) #key:valueprint(r.text)這是輸出：https://pastebin.com/raw/HMBZWR36這是顯示頁碼的部分：# <div id="myPager"># <div class="pagination-container"><ul class="pagination blue-pager"><li class="blue-pager active"><span>1</span></li></ul></div># </div>

查看完整描述

2 回答

慕俠2389804

TA貢獻1719條經驗獲得超6個贊

該頁面上的數據是使用 JavaScript 動態加載的，因此您不能僅通過該頁面的請求來獲取它。使用瀏覽器中的 Developer 標簽，您可以攔截生成數據的 xhr 并將其用于請求。這就是本例中的情況。

import requests

headers = {

'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0',

'Accept': '*/*',

'Accept-Language': 'en-US,en;q=0.5',

'X-Requested-With': 'XMLHttpRequest',

'Connection': 'keep-alive',

'Referer': 'https://search.aap.org/nicu/',

}

params = (

('Model', 'X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]'),

('jsonSearchVmString', '{"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":2,"LevelAssigner":null,"BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}'),

('page', '2'),

('class', 'blue-pager'),

('X-Requested-With', 'XMLHttpRequest'),

('_', '1603318486561'),

)

response = requests.get('https://search.aap.org/nicu/', headers=headers, params=params)

參數之一是('page', '2')；如果您將請求包裝在for適當范圍內的循環中，您應該能夠requests.get()在所有頁面上執行此操作。

反對回復 2023-12-08

蠱毒傳說

TA貢獻1895條經驗獲得超3個贊

在這種情況下，您不必使用 POST 請求。如果單擊下一頁時查看 URL，您將看到可以更改的所有參數。此外，要獲取所有頁面，您可以解析 HTML 并查找<a rel="next">標簽。如果標簽不存在，則您位于最后一頁（嘗試將參數更改"LevelAssigner"為3其他值）：

import requests

from bs4 import BeautifulSoup

url = 'https://search.aap.org/nicu/?Model=X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]&jsonSearchVmString={"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":0,"LevelAssigner":"3","BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}&class=blue-pager&X-Requested-With=XMLHttpRequest&page='

page = 1

while True:

print('Page {}..'.format(page))

soup = BeautifulSoup(requests.get(url + str(page)).content, 'html.parser')

for row in soup.select('.row.font-700'):

print(row.get_text(strip=True, separator=' '))

next_page = soup.select_one('a[rel="next"]')

if not next_page:

break

page += 1

印刷：

Page 1..

ANDERSON REGIONAL MEDICAL CENTER Meridian, MS 39301

Ann & Robert H. Lurie Children's Hospital of Chicago Chicago, IL 60611

BAYSTATE MEDICAL CENTER Springfield, MA 01199

Beacon Children's Hospital South Bend, IN 46601-1033

BEN TAUB GENERAL HOSPITAL Houston, TX 77030

Page 2..

BETH ISRAEL DEACONESS MEDICAL CENTER Boston, MA 02215

BRIGHAM AND WOMEN'S HOSPITAL Boston, MA 02115

CHILDREN'S HOSPITAL OF NEW ORLEANS New Orleans, LA 70118

CLEVELAND CLINIC formally MERIDIA HILLCREST HOSPITAL Mayfield Heights, OH 44124

EAST JEFFERSON GENERAL HOSPITAL Metairie, LA 70006

Page 3..

GEORGETOWN UNIVERSITY HOSPITAL Washington, DC 20007

GREENWICH HOSPITAL Greenwich, CT 06830

HELEN DEVOS CHILDREN'S HOSPITAL Grand Rapids, MI 49503

INOVA LOUDOUN HOSPITAL Leesburg, VA 20176

JACOBI MEDICAL CENTER Bronx, NY 10461

Page 4..

LAKEVIEW REGIONAL MEDICAL CENTER HOSPITAL Covington, LA 70433

LOYOLA UNIVERSITY MEDICAL CENTER Maywood, IL 60153

MELROSE WAKEFIELD HOSPITAL Melrose, MA 02176

MERCYHEALTH HOSPITAL ROCKTON AVENUE CAMPUS formally ROCKFORD HEALTH SYSTEM Rockford, IL 61103

RUSH FOUNDATION HOSPITAL Meridian, MS 39301

Page 5..

Shetal Shah Valhalla, NY 10595

SLIDELL MEMORIAL HOSPITAL Slidell, LA 70458

SOUTHERN NEW HAMPSHIRE MEDICAL CENTER Nashua, NH 03061

St Mary Medical Center Apple Valley, CA 92307

ST. AGNES HOSPITAL Baltimore, MD 21229

Page 6..

THE CHILDREN'S HOSPITAL AT NAVICENT HEALTH Macon, GA 31208-6000

TOURO INFIRMARY New Orleans, LA 70115

TULANE LAKESIDE HOSPITAL Metairie, LA 70001

UNIVERSITY MEDICAL CENTER - SUNY STONY BROOK Stony Brook, NY 11794-8111

UNIVERSITY OF IOWA CHILDREN'S HOSPITAL Iowa City, IA 52242

Page 7..

UNIVERSITY OF LOUISVILLE SCHOOL OF MEDICINE Louisville, KY 40232-5070

UNIVERSITY OF MISSISSIPPI MEDICAL CENTER Jackson, MS 39216-4505

UNIVERSITY OF ROCHESTER MEDICAL CENTER Rochester, NY 14642

UNIVERSITY OF WASHINGTON MEDICAL CENTER Seattle, WA 98195

WakeMed Cary Hospital Cary, NC 27518

Page 8..

WakeMed North Family Health & Women’s Hospital Raleigh, NC 27614

WakeMed Raleigh Hospital Raleigh, NC 27610

Wellstar Kennestone Hospital NICU Marietta, GA 30066

WILLIS KNIGHTON CENTER FOR WOMEN'S HEALTH Shreveport, LA 71118

WOMAN'S HOSPITAL Baton Rouge, LA 70817

反對回復 2023-12-08

2 回答
0 關注
198 瀏覽

關注

添加回答

舉報

0/150

提交

取消

亚洲在线久爱草,狠狠天天香蕉网,天天搞日日干久草,伊人亚洲日本欧美

熱搜

最近搜索清空

使用 Python 的 requests.post 函數抓取動態 HTML 時如何指定“頁碼”？

使用 Python 的 requests.post 函數抓取動態 HTML 時如何指定“頁碼”？

2 回答

添加回答

使用 Python 的 requests.post 函數抓取動態 HTML 時如何指定“頁碼”？

使用 Python 的 requests.post 函數抓取動態 HTML 時如何指定“頁碼”？