Scrapy的JSONDecodeError:应为值:第1行第1列(字符0)

我使用请求来获取和解析一些使用Scrapy刮取的数据 具有 Scrapyrt(实时刮削)。

我就是这样做的:

#pass spider to requests parameters # 
params = {
        'spider_name': spider,
        'start_requests':True
        }
# scrape items
response = requests.get('http://scrapyrt:9080/crawl.json', params)
print ('RESPONSE JSON',response.json())
data = response.json()

根据碎屑 documentation ,与 'start_requests' 参数设置为 True ,spider自动请求url并将响应传递给parse方法,parse方法是用于解析请求的默认方法。

启动请求

类型:布尔型

可选择的

spider是否应该执行Scrapy.spider.start_requests方法。在没有ScrapyRT的情况下正常运行Scrapy Spider时,默认情况下会执行start_请求,但此方法默认情况下不会在API中执行。默认情况下,我们假设spider只对参数中提供的url进行爬网,而不请求启动spider类中定义的url。start_requests参数重写此行为。 如果存在此参数,则API将执行start_requests Spider方法 .

但设置不起作用。日志:

[2019-05-19 06:11:14,835: DEBUG/ForkPoolWorker-4] Starting new HTTP connection (1): scrapyrt:9080
[2019-05-19 06:11:15,414: DEBUG/ForkPoolWorker-4] http://scrapyrt:9080 "GET /crawl.json?spider_name=precious_tracks&start_requests=True HTTP/1.1" 500 7784
[2019-05-19 06:11:15,472: ERROR/ForkPoolWorker-4] Task project.api.routes.background.scrape_allmusic[87dbd825-dc1c-4789-8ee0-4151e5821798] raised unexpected: JSONDecodeError('Expecting value: line 1 column 1 (char 0)',)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/app/project/api/routes/background.py", line 908, in scrape_allmusic
    print ('RESPONSE JSON',response.json())
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)