[Python-Dev] Usage of the multiprocessing API and object lifetime
Le mar. 11 d?c. 2018 ? 16:14, Antoine Pitrou <solipsis at pitrou.net> a ?crit :
> What you are proposing here starts to smell like an anti-pattern to
> me. Python _is_ a garbage-collected language, so by definition, there
> _are_ going to be resources that are automatically collected when an
> object disappears. If I'm allocating a 2GB bytes object, then PyPy may
> delay the deallocation much longer than CPython. Do you propose we add
> a release() method to bytes objects to avoid this issue (and emit a
> warning for people who don't call release() on bytes objects)?
We are not talking about simple strings, but processes and threads.
> You can't change the language's philosophy. We warn about open files
> because those have user-visible consequences (such as unflushed
> buffers, or not being able to delete the file on Windows). If there is
> no user-visible consequence to not calling join() on a Pool, then we
> shouldn't warn about it.
"user-visible consequences" are that resources are kept alive longer
than I would expect. When I use a context manager, I expect that
Python will magically releases everything for me.
For example, "with subprocess.Popen() as popen: ..." ensures that all
pipes are closed and the process completes, before we exit the block.
Another example, "with open() as fp: ..." ensures that the file
descriptor is closed before we exit the block.
I modified subprocess.Popen.__del__() in Python 3.6 to emit a
ResourceWarning if the subprocess is still running, to suggest the
developer to explicitly manage the resource (ex: call .wait()).
I prefer to explicitly manager resources like processes and threads
since they can exit with error: killed by a signal, waitpid() failure
(exit status already read by a different function), etc. I prefer to
control where the error occurs. I hate when Python logs strange error
during shutdown. Logging errors during shutdown is too late: for
example, the log triggers a new error because a stdlib module has been
cleared. That's why we need hacks like "_warn=warnings.warn" below:
def __del__(self, _maxsize=sys.maxsize, _warn=warnings.warn):
if self.returncode is None:
_warn("subprocess %s is still running" % self.pid,