代码之家 › 专栏 › 技术社区 › CodeNoob

发布查询以访问网站上的表数据(Python)

response request web-scraping post python

CodeNoob · 技术社区 · 7 年前

this website 然而,该网站没有API来方便数据提取。因此,网站上有一个包含数据的“表格”,但该表格分布在多个页面上,下载选项不支持“批量”下载。所以我想找到一种通过Python下载的方法。
我想我可以通过post请求来实现这一点。我所做的是从第1页开始,然后单击“下一页”按钮并观看网络选项卡,这给了我:

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 200-399/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:36:40 GMT
ETag:W/"8ac0b-cFdEBUwfdiyTQm/gpJHYzQ"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311; _gat=1
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=200-399
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=200-399

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)
Name
genome/

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 400-599/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:44:24 GMT
ETag:W/"5dc73-ODopjbbyl5M2vUJvUGN0Gw"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=400-599
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=400-599

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)

因此,从第1页开始,在post请求过程中(在我看来)发生了变化的重要事情-->2和第2页-->3个是:

响应头 内容范围: --&燃气轮机;内容范围:
范围: --&燃气轮机;范围: 项目=400-599
项目=200-399

这是网站上说的:

第1页:45999个结果中的1-200个(但针对0-199的帖子,上面未显示)
第3页:45999个结果中的401-600个(但针对400-599的帖子,见上文)

我有零

requests.post(some url) and extract the total page numbers (or number of genomes and devide these by the max number per page, which is 200 see post requests).

然后是这样的:

for page_numb in range(page_numbers):
    r = requests.post(some_url...... + page_numb)
    #extract the information from the table

最后一个问题:) 我不知道如何设置POST请求,例如

p1 = requests.post(https://www.patricbrc.org/api/genome/... 0-199)
p2 = requests.post(https://www.patricbrc.org/api/genome/... 200-399)
etc...

(我希望响应将包含由一些分隔符分隔的表数据)

import requests
import json

url = 'https://www.patricbrc.org/api/genome/'
payload = {"eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)"}
headers = {
    "X-Range": "items=0-199",
    "Range": "items=0-199",
    "content-type": "application/json"}


r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r)

这段代码不起作用,但我想应该是这样的。
我知道我应该将负载作为JSON发布,但我不知道如何翻译 eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)

1 回复 | 直到 7 年前

sKwa 7 年前

这是一个源代码(它的代码很难看,但它只是一个示例,我建议您使用 request.Session ):

import sys
import json
import requests

URL = 'https://www.patricbrc.org/view/Taxonomy/2'
API = 'https://www.patricbrc.org/api/genome/'

# we need to get a Cookie
response = requests.get(URL)
if response.status_code != 200:
    sys.exit('Cookie request failed')
cookie = response.headers['Set-Cookie'].split(';')[0]
etag   = response.headers['ETag']

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
    'Cookie'          : '{};'.format(cookie),
    'ETag'            : etag
}

data = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=data)
if response.status_code != 200:
    sys.exit('API request failed')
data = json.loads(response.text)

# test data
row200 = data[0]
print 'FAMILY     :', row200['family']
print 'GC_CONTENT :', row200['gc_content'] 
print 'GENUS      :', row200['genus']
print 'ORDER      :', row200['order']

输出 test data 块:

FAMILY     : Eubacteriaceae
GC_CONTENT : 38.78
GENUS      : Eubacterium
ORDER      : Clostridiales

你不需要饼干,这也很好:

import json
import requests

API = 'https://www.patricbrc.org/api/genome/'

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
}

query = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=query)
data = json.loads(response.text)