[Python-Dev] Sub-interpreters: importing numpy causes hang
I am new to the list and arriving with a concrete problem that I'd
like to fix myself.
I am embedding Python (3.6) into my C++ application and I would like
to run Python scripts isolated from each other using sub-interpreters.
I am not using threads; everything is supposed to run in the
application's main thread.
I noticed that if I create an interpreter, switch to it and execute
code that imports numpy (1.13), my application will hang.
> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
[Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
python36.dll!PyGILState_Ensure() Line 855 C
[Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
[Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
python36.dll!call_function(_object * * *
pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
PyEval_RestoreThread on the (auto) threadstate of the main
interpreter. And that's wrong.
We are already holding the GIL with the threadstate of our current
sub-interpreter, so there's no need to switch.
I know that the GIL API is not fully compatible with sub-interpreters,
as issues #10915 and #15751 illustrate.
But since I need to support calls to PyGILState_Ensure - numpy is the
best example -, I am trying to improve the situation here:
That change may be naive, but it does the trick for my use case. If
totally wrong, I don't mind pursuing another alley.
Essentially, I'd like to ask for some guidance in how to tackle this
problem while keeping the current GIL API unchanged (to avoid breaking
I am also wondering how I can test any changes I am proposing. Is
there a test suite for interpreters, for example?
Thank you very much,