osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Counting Python threads vs C/C++ threads


I'm looking at a performance problem in a large CPython 2.x/3.x codebase
with quite a few dependencies.

I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
hard, and I/O on the system appears to be low - but throughput is poor.
I'm wondering if it could be CPU-bound Python threads causing the problem
(because of the threading+GIL thing).

The non-dependency Python portions don't Appear to have much in the way of
threading going on based on a quick grep, but csysdig says a process
running the code has around 32 threads running - the actual thread count
varies, but that's the ballpark.

I'm wondering if there's a good way to find two counts of those threads -
how many are from CPython code that could run afoul of the GIL, and how
many of them are from C/C++ extension modules that wouldn't be responsible
for a GIL issue.

Does anyone know how to find out these different thread counts, without
mounting a huge effort to scrutinize all the code and its dependencies?
I'd prefer a tool that can scrutinize the process(es) from the outside, but
modifying the code is not out of the question.

Please see
https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside
for more information.  Or perhaps
https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation

Thanks!