Self-deadlock due to garbage collector in single-threaded code
I have design problem: there is a global resource that cannot be accessed
from multiple threads at once, and so I need a lock around it to serialize
the access to it. However, Python's garbage collector can run a __del__
method while I am doing some processing while holding the lock. If the
destructor tries to access the resource, this ends up with a deadlock.
As an example, consider the following innocent-looking single-threaded
code, which deadlocks if you run it:
import threading
class Handle(object):
def __init__(self):
self.handle = do_stuff("get")
def close(self):
h = self.handle
self.handle = None
if h is not None:
do_stuff("close %d" % h)
def __del__(self):
self.close()
_resource_lock = threading.Lock()
def do_stuff(what):
_resource_lock.acquire()
try:
# GC can be invoked here -> deadlock!
for j in range(20):
list()
return 1234
finally:
_resource_lock.release()
for j in range(1000):
xs = []
b = Handle()
xs.append(b)
xs.append(xs)
The resource can deal with several "handles" being open at the same time,
and I'd need to deal with their life cycle. Abstracting this into a Handle
class and putting the cleanup in __del__ seemed like a smart move, but the
above issue breaks this.
One way to deal with the cleanup is to keep a "pending cleanup" list of
handles, and if the lock is held when __del__ is run, insert the handle
there, and clean up the list later on.
The question is:
Is there a threadsafe version of gc.disable() / gc.enable() that would
solve this in a cleaner way?
Other ideas how to deal with this?