使用requests/urllib3在每次重试时添加回调函数

你可以将 Retry 类来添加该功能。

这是与 重试 给定连接尝试的实例:

• Retry.increment() 无论何时引发异常或返回30x重定向响应,或 Retry.is_retry() 方法返回true。
  • .increment() 将重新引发错误(如果有),并且对象被配置为不重试该特定类型的错误。
  • .增量() 电话 Retry.new() 若要创建更新的实例,并更新所有相关计数器, history 属性修改为新的 RequestHistory() instance (命名元组)。
  • .增量() 会引起 MaxRetryError 例外情况 Retry.is_exhausted() 调用的返回值为 重试。新建() 是真的。 is_exhausted() 当它跟踪的任何计数器低于0时返回true(计数器设置为 None 被忽略)。
  • .增量() 返回新的 重试 实例。
• 的返回值 重试。增量() 取代旧的 重试 实例已跟踪。如果有重定向,那么 Retry.sleep_for_retry() 被称为(如果有 Retry-After 页眉),否则 Retry.sleep() 被称为 self.sleep_for_retry() 尊敬 稍后重试 头,否则,如果有退避策略,则只会休眠)。然后使用新的 重试 实例。

这给了你3个好的回调点;在 .增量() ,当创建新的 重试 实例,并在周围的上下文管理器中 super().increment() 允许回调否决异常,或在退出时更新返回的重试策略。

这就是为什么要把钩子放在 .增量() 看起来像:

import logging

logger = getLogger(__name__)

class CallbackRetry(Retry):
    def __init__(self, *args, **kwargs):
        self._callback = kwargs.pop('callback', None)
        super(CallbackRetry, self).__init__(*args, **kwargs)
    def new(self, **kw):
        # pass along the subclass additional information when creating
        # a new instance.
        kw['callback'] = self._callback
        return super(CallbackRetry, self).new(**kw)
    def increment(self, method, url, *args, **kwargs):
        if self._callback:
            try:
                self._callback(url)
            except Exception:
                logger.exception('Callback raised an exception, ignoring')
        return super(CallbackRetry, self).increment(method, url, *args, **kwargs)

注意,那个 url 争论真的只是 URL路径 ,将省略请求的净位置部分(您必须从 _pool 争论,它有 .scheme 我是说, .host 和 .port 属性)。

演示:

>>> def retry_callback(url):
...     print('Callback invoked with', url)
...
>>> s = requests.Session()
>>> retries = CallbackRetry(total=5, status_forcelist=[500, 502, 503, 504], callback=retry_callback)
>>> s.mount('http://', HTTPAdapter(max_retries=retries))
>>> s.get('http://httpstat.us/500')
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Callback invoked with /500
Traceback (most recent call last):
  File "/.../lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
    body_pos=body_pos, **response_kw)
  [Previous line repeated 1 more times]
  File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 712, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "<stdin>", line 8, in increment
  File "/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/.../lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/.../lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/.../lib/python3.6/site-packages/requests/adapters.py", line 499, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))

用钩子钩住 .new() 方法可以让您为下一次尝试调整策略,也可以让您反思 .history 属性,但不会让您避免重新引发异常。