[Python-Dev] Usage of the multiprocessing API and object lifetime
On Tue, 11 Dec 2018 17:22:49 +0100
Victor Stinner <vstinner at redhat.com> wrote:
> Le mar. 11 d?c. 2018 ? 17:06, Antoine Pitrou <solipsis at pitrou.net> a ?crit :
> > > We are not talking about simple strings, but processes and threads.
> > Right, but do those have an impact on the program's correctness, or
> > simply on its performance (or memory consumption)?
Well, at least we shouldn't emit ResourceWarning for performance issues.
So if someone used "with pool:", they shouldn't get a ResourceWarning
afterwards, even if some threads are still not finished running.
> I don't know if these changes are similar to my questions about
> multiprocessing API, since socketserver didn't expose the list of
> processes/threads and doesn't provide a method to wait until they
> complete. Well... ForkingMixIn has an *undocumented*
> collect_children() which is non-blocking by default (I added a
> keyword-only 'blocking' parameter).
socketserver is not the same as a multiprocessing Pool *at all*. The
usage will be vastly different between the two. Just because we did
this change for socketserver is not enough of a reason to do it for
> Another example: the close() method of an asyncio event loop doesn't
> wait until threads/processes complete:
> "asyncio: BaseEventLoop.close() shutdowns the executor without waiting
> causing leak of dangling threads"
It's not a leak though, it's a resource that's collected a bit later.
It may annoy the test machinery (and this could be a reason to add
private or public APIs to help finish the threads), but it shouldn't be
an actual issue for user code.
Here as well, I think you should be careful not to introduce annoyances
(unwanted latencies) in user code.