osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Python-Dev] Sub-interpreters: importing numpy causes hang


Hi!

Well, the plugins would be created by third-parties and I'd like them
to enable bunding of modules with their plugins.
I am afraid of modules with the same name, but being different, or
different versions of modules being used by different plugins. If
plugins share an interpreter, the module with a given name that is
imported first sticks around forever and for all plugins.

I am thinking about this design:
- Plugins don't maintain state in their Python world. They expose
functions, my application calls them.
- Everytime I call into them, they are presented with a clean global
namespace. After the call, the namespace (dict) is thrown away. That
releases any objects the plugin code has created.
- So, then I could also actively unload modules they loaded. But I do
know that this is problematic in particular for modules that use
native code.

I am interested in both a short-term and a long-term solution.
Actually, making subinterpreters work better is pretty sexy ...
because it's hard. :-)

Stephan

Am Mi., 23. Jan. 2019 um 11:30 Uhr schrieb Petr Viktorin <encukou at gmail.com>:
>
> On 1/23/19 3:33 AM, Stephan Reiter wrote:
> > Thanks for the answers so far. I appreciate them!
> >
> > Nathaniel, I'd like to allow Python plugins in my application. A
> > plugin should be allowed to bring its own modules along (i.e.
> > plugin-specific subdir is in sys.path when the plugin is active) and
> > hence some isolation of them will be needed, so that they can use
> > different versions of a given module. That's my main motivation for
> > using subinterpreters.
> > I thought about running plugins out-of-processes - a separate process
> > for every plugin - and allow them to communicate with my application
> > via RPC. But that makes it more complex to implement the API my
> > application will offer and will slow down things due to the need to
> > copy data.
> > Maybe you have another idea for me? :)
>
> Try to make the plugins work together. Look into using pip/PyPI for your
> plugins. Try to make it so each package ("plugin") would have only one
> module/package, and dependencies would be other packages that can be
> installed individually and shared. And keep in mind you can set up your
> own package index, or distribute/install individual package files.
>
> If that's not possible, and you want things to work now, go with subprocess.
>
> If you want to help make subinterpreters work better, there are several
> people scratching at the problem from different angles. Most/all would
> welcome help, but don't expect any short-term benefits.
> (FWIW, my own effort is currently blocked on PEP 580, and I hope to move
> forward after a Council is elected.)
>
>
> > Henry, Antoine, thanks for your input; I'll check out the tests and
> > see what I can learn from issue 10915.
> >
> > Stephan
> >
> > Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith <njs at pobox.com>:
> >>
> >> There are currently numerous incompatibilities between numpy and subinterpreters, and no concrete plan for fixing them. The numpy team does not consider subinterpreters to be a supported configuration, and can't help you with any issues you run into. I know the concept of subinterpreters is really appealing, but unfortunately the CPython implementation is not really mature or widely supported... are you absolutely certain you need to use subinterpreters for your application?
> >>
> >> On Tue, Jan 22, 2019, 08:27 Stephan Reiter <stephan.reiter at gmail.com wrote:
> >>>
> >>> Hi all!
> >>>
> >>> I am new to the list and arriving with a concrete problem that I'd
> >>> like to fix myself.
> >>>
> >>> I am embedding Python (3.6) into my C++ application and I would like
> >>> to run Python scripts isolated from each other using sub-interpreters.
> >>> I am not using threads; everything is supposed to run in the
> >>> application's main thread.
> >>>
> >>> I noticed that if I create an interpreter, switch to it and execute
> >>> code that imports numpy (1.13), my application will hang.
> >>>
> >>>    ntdll.dll!NtWaitForSingleObject() Unknown
> >>>    KernelBase.dll!WaitForSingleObjectEx() Unknown
> >>>> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
> >>>    [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
> >>>    python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
> >>>    python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
> >>>    python36.dll!PyGILState_Ensure() Line 855 C
> >>>    umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
> >>>    umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
> >>>    [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
> >>>    [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
> >>>    python36.dll!call_function(_object * * *
> >>> pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
> >>> 4822 C
> >>>
> >>> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
> >>> PyEval_RestoreThread on the (auto) threadstate of the main
> >>> interpreter. And that's wrong.
> >>> We are already holding the GIL with the threadstate of our current
> >>> sub-interpreter, so there's no need to switch.
> >>>
> >>> I know that the GIL API is not fully compatible with sub-interpreters,
> >>> as issues #10915 and #15751 illustrate.
> >>>
> >>> But since I need to support calls to PyGILState_Ensure - numpy is the
> >>> best example -, I am trying to improve the situation here:
> >>> https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597
> >>>
> >>> That change may be naive, but it does the trick for my use case. If
> >>> totally wrong, I don't mind pursuing another alley.
> >>>
> >>> Essentially, I'd like to ask for some guidance in how to tackle this
> >>> problem while keeping the current GIL API unchanged (to avoid breaking
> >>> modules).
> >>>
> >>> I am also wondering how I can test any changes I am proposing. Is
> >>> there a test suite for interpreters, for example?
> >>>
> >>> Thank you very much,
> >>> Stephan
> >>> _______________________________________________
> >>> Python-Dev mailing list
> >>> Python-Dev at python.org
> >>> https://mail.python.org/mailman/listinfo/python-dev
> >>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com
> >