代码之家 › 专栏 › 技术社区 › olegario

Mongo-多次请求后数据库脱机

scrapy amazon-web-services mongodb

olegario · 技术社区 · 6 年前

我使用MongoDB来存储从web上获取的数据,使用Scrapy作为scraper。问题是,当我开始使用多个爬行器运行一个长的刮取过程时,Mongo崩溃,爬行器开始接收以下消息:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/search/decapod/updater/updater/pipelines.py", line 90, in process_item
    self.db[self.collection_name].insert_one(dict(item))
  File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 693, in insert_one
    session=session),
  File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 607, in _insert
    bypass_doc_val, session)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/collection.py", line 595, in _insert_one
    acknowledged, _insert_command, session)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1242, in _retryable_write
    with self._tmp_session(session) as s:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1571, in _tmp_session
    s = self._ensure_session(session)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1558, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1511, in __start_session
    server_session = self._get_server_session()
  File "/usr/local/lib/python3.5/dist-packages/pymongo/mongo_client.py", line 1544, in _get_server_session
    return self._topology.get_server_session()
  File "/usr/local/lib/python3.5/dist-packages/pymongo/topology.py", line 427, in get_server_session
    None)
  File "/usr/local/lib/python3.5/dist-packages/pymongo/topology.py", line 199, in _select_servers_loop
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: mongodb.getmore.com.br:27017: timed out

如何在Mongo崩溃时自动重启或防止这种情况发生?

我目前正在EC2实例上运行mongo t2.small .

1 回复 | 直到 6 年前

Akrion 6 年前

我怀疑你没有 keepAlive 在您的连接设置中。

你能试着把你的头发调好吗 keepAlive: 1 在您的连接选项中:

options: {
  server: {
    socketOptions: {
      keepAlive: 1,
      connectTimeoutMS: 30000
    }
  }
}

这里有更多关于 optional parameters

推荐文章

Anna Berezko · AWS匹配不支持的TLD域名和S3 bucket静态网站

1 年前

R0bert2 · Ansible-使用with_项创建列表

2 年前

renzCNFT · 与s3相比,workdocs有什么优势

2 年前

Eva · Github与AWS codecommit镜像和同步的操作

2 年前

Heritage Squad · 如何使节点组模块中的每个实例都使用特定的安全组?

2 年前

Christian Townsend · 如何从Terraform中的ElastiCache获取cache_节点地址?

2 年前

Hasham · 如何将多个本地文件上载到s3中的一个文件

2 年前

shesupplypi · 如何将一个应用程序或服务的无服务器输出引用到另一个应用程序或服务?

2 年前

sebas flores · S3 URL-使用python下载

2 年前

Ibrahim Patel · AWS CloudWatch Error“尝试获取图形数据时出错。”

2 年前