Python:检查对象是否可以原子地pickle

鉴于您愿意打破封装,我认为这是您所能做的最好的:

from pickle import Pickler
import os

class AtomicPickler(Pickler):
  def __init__(self, protocol):
    # You may want to replace this with a fake file object that just
    # discards writes.
    blackhole = open(os.devnull, 'w')

    Pickler.__init__(self, blackhole, protocol)
    self.depth = 0

  def save(self, o):
    self.depth += 1
    if self.depth == 1:
      return Pickler.save(self, o)
    self.depth -= 1
    return

def is_atomically_pickleable(o, protocol=None):
  pickler = AtomicPickler(protocol)
  try:
    pickler.dump(o)
    return True
  except:
    # Hopefully this exception was actually caused by dump(), and not
    # something like a KeyboardInterrupt
    return False

在Python中,判断某个东西是否有效的唯一方法是尝试它。这就是像Python一样动态的语言的本质。你的问题的困难在于你想区分“顶层”的失败和深层次的失败。

Pickler.save 本质上是Python的pickling逻辑的控制中心,因此上面创建了一个 Pickler 忽略对其的递归调用 save 方法。在顶级保存中引发的任何异常都将被视为酸洗失败。您可能要将限定符添加到 except 陈述。不合格 excepts 在Python中,异常通常是一个坏主意,因为异常不仅用于程序错误,还用于诸如 KeyboardInterrupt 和 SystemExit .

对于具有奇怪的自定义pickling逻辑的类型,这可能会给出可以证明是错误的否定。例如,如果您创建一个类似于自定义列表的类,而不是 Pickler.保存 为了递归地调用它,它实际上试图以某种方式对它自己的元素进行pickle,然后创建了这个类的一个实例,其中包含一个自定义逻辑无法pickle的元素, is_atomically_pickleable 会回来的 False 对于此实例,即使删除有问题的元素也会导致可pickle的对象。

另外,请注意 原子是可以腌制的吗 . 理论上,当一个对象被不同的协议pickle时,它的行为可能会有所不同(尽管那会很奇怪),所以你应该使这个与你给出的协议参数相匹配 dump .

考虑到Python的动态特性,我认为除了启发式或白名单之外,没有一种真正定义良好的方法来完成您所要求的任务。

如果我说:

x = object()

x“原子可腌制”吗?如果我说:

x.foo = threading.Lock()

? x现在是“原子泡菜”了吗?

如果我创建了一个始终具有lock属性的单独类呢?如果我从实例中删除了该属性怎么办?

我认为persistent_id接口与您尝试的不匹配。它设计用于当对象引用新程序上的等效对象,而不是旧程序的副本时。您正在尝试筛选出每个无法pickle的对象,这是不同的,为什么您要尝试这样做。

我认为这是你的代码有问题的迹象。事实上,您想要pickle引用gui小部件、文件和锁的对象,这表明您正在做一些奇怪的事情。通常持久化的对象类型不应该与此类对象相关或包含对此类对象的引用。

话虽如此,我认为你最好的选择是:

class MyPickler(Pickler):
    def save(self, obj):
        try:
             Pickler.save(self, obj)
        except PicklingEror:
             Pickle.save( self, FilteredObject(obj) )

这应该对python实现有效,我不能保证C实现中会发生什么。保存的每个对象都将传递给save方法。当无法pickle对象时,此方法将引发PicklingError。此时,您可以介入并调用函数,要求它对您自己的对象进行pickle,该对象应该可以进行pickle。

编辑

据我所知,你基本上有一个用户创建的对象字典。有些对象是可拾取的,有些则不是。我将执行以下操作:

class saveable_dict(dict):
    def __getstate__(self):
        data = {}
        for key, value in self.items():
             try:
                  encoded = cPickle.dumps(value)
             except PicklingError:
                  encoded = cPickle.dumps( Unpickable() )
        return data

    def __setstate__(self, state):
       for key, value in state:
           self[key] = cPickle.loads(value)

然后当你想保存对象集合时,使用字典。用户应该能够获取任何可拾取的对象,但其他所有对象都将作为Unpicklable()对象返回。这种方法与以前的方法的区别在于对象本身是可拾取的,但引用了不可拾取的对象。但不管怎样,这些东西很可能会破碎。

这种方法还有一个好处,那就是它完全保留在定义的API中,因此应该在cPickle或pickle中工作。

我最终编写了自己的解决方案。

Here's the code . Here are the tests . 它是 GarlicSim ,所以你可以通过 installing garlicsim 和做 from garlicsim.general_misc import pickle_tools .

如果要在Python3代码中使用它,请使用 Python 3 fork of garlicsim .

以下是模块摘录(可能已过时):

import re
import cPickle as pickle_module
import pickle # Importing just to get dispatch table, not pickling with it.
import copy_reg
import types

from garlicsim.general_misc import address_tools
from garlicsim.general_misc import misc_tools


def is_atomically_pickleable(thing):
    '''
    Return whether `thing` is an atomically pickleable object.

    "Atomically-pickleable" means that it's pickleable without considering any
    other object that it contains or refers to. For example, a `list` is
    atomically pickleable, even if it contains an unpickleable object, like a
    `threading.Lock()`.

    However, the `threading.Lock()` itself is not atomically pickleable.
    '''
    my_type = misc_tools.get_actual_type(thing)
    return _is_type_atomically_pickleable(my_type, thing)


def _is_type_atomically_pickleable(type_, thing=None):
    '''Return whether `type_` is an atomically pickleable type.'''
    try:
        return _is_type_atomically_pickleable.cache[type_]
    except KeyError:
        pass

    if thing is not None:
        assert isinstance(thing, type_)

    # Sub-function in order to do caching without crowding the main algorithm:
    def get_result():

        # We allow a flag for types to painlessly declare whether they're
        # atomically pickleable:
        if hasattr(type_, '_is_atomically_pickleable'):
            return type_._is_atomically_pickleable

        # Weird special case: `threading.Lock` objects don't have `__class__`.
        # We assume that objects that don't have `__class__` can't be pickled.
        # (With the exception of old-style classes themselves.)
        if not hasattr(thing, '__class__') and \
           (not isinstance(thing, types.ClassType)):
            return False

        if not issubclass(type_, object):
            return True

        def assert_legit_pickling_exception(exception):
            '''Assert that `exception` reports a problem in pickling.'''
            message = exception.args[0]
            segments = [
                "can't pickle",
                'should only be shared between processes through inheritance',
                'cannot be passed between processes or pickled'
            ]
            assert any((segment in message) for segment in segments)
            # todo: turn to warning

        if type_ in pickle.Pickler.dispatch:
            return True

        reduce_function = copy_reg.dispatch_table.get(type_)
        if reduce_function:
            try:
                reduce_result = reduce_function(thing)
            except Exception, exception:
                assert_legit_pickling_exception(exception)
                return False
            else:
                return True

        reduce_function = getattr(type_, '__reduce_ex__', None)
        if reduce_function:
            try:
                reduce_result = reduce_function(thing, 0)
                # (The `0` is the protocol argument.)
            except Exception, exception:
                assert_legit_pickling_exception(exception)
                return False
            else:
                return True

        reduce_function = getattr(type_, '__reduce__', None)
        if reduce_function:
            try:
                reduce_result = reduce_function(thing)
            except Exception, exception:
                assert_legit_pickling_exception(exception)
                return False
            else:
                return True

        return False

    result = get_result()
    _is_type_atomically_pickleable.cache[type_] = result
    return result

_is_type_atomically_pickleable.cache = {}

dill 拥有 pickles 这种支票的方法。

>>> import threading
>>> l = [threading.Lock()]
>>> 
>>> import dill
>>> dill.pickles(l)
True
>>> 
>>> dill.pickles(threading.Lock())
True
>>> f = open('whatever', 'w') 
>>> f.close()
>>> dill.pickles(open('whatever', 'r'))
True

好, 小茴香 原子泡菜你所有的例子,所以让我们尝试其他东西:

>>> l = [iter([1,2,3]), xrange(5)]
>>> dill.pickles(l)
False

好吧,这失败了。现在,让我们调查一下:

>>> dill.detect.trace(True)
>>> dill.pickles(l)
T4: <type 'listiterator'>
False
>>> map(dill.pickles, l)
T4: <type 'listiterator'>
Si: xrange(5)
F2: <function _eval_repr at 0x106991cf8>
[False, True]

好 啊。我们可以看到 iter 失败了,但是 xrange 泡菜。所以,让我们替换 iter .

>>> l[0] = xrange(1,4)
>>> dill.pickles(l)
Si: xrange(1, 4)
F2: <function _eval_repr at 0x106991cf8>
Si: xrange(5)
True

现在我们的目标原子泡菜。