Subject: Re: compiling more .pyx and .pxd files into one .so - msg#00080

Previous Message by Date: click to view message preview

Re: [Pyrex] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

Stefan Behnel wrote: > I was proposing the exact opposite: disallow "with GIL" in .pxd files and in > function pre-declarations, and only allow them on the function implementation. Yes, that would work, but it would mean that the declaration could only be taken advantage of within the one module. Not a severe limitation, but a limitation nonetheless. BTW, we seem to have a terminology clash here -- what you're calling "with GIL" is what I'm calling "without GIL", since the signature is seen from the point of view of the calling function. I'm going to call it "nogil" from here on, because it's shorter and avoids conflicting with the potential control structures for acquiring and releasing the GIL. > There is no way Pyrex/Cython can > always make sure it knows if the GIL is held by the calling code. It's C after > all, things can get called from all over the place. The programmer would be responsible for making the correct declarations when dealing with external C code, as always. But as long as those declarations correctly reflect the behaviour of the external code, Pyrex should be able to check that what the programmer does with the GIL makes sense. Perhaps I can give an example of how I see it working. Suppose you have a C library with a call that does some heavy work and takes a callback. You want to release the GIL around calls to this function. As a first attempt, the programmer writes # Version 1 cdef extern from "heavy.h": ctypedef void (*callback)(void *) void do_something_heavy(callback) cdef void my_callback(void *data): print "Called back!" def heavy(): without GIL: do_something_heavy(my_callback) Now Pyrex notices that do_something_heavy() is being called with the GIL released, but hasn't been declared as safe to do so, and complains. So the programmer changes the declarations to: # Version 2 cdef extern from "heavy.h": ctypedef void (*callback)(void *) nogil void do_something_heavy(callback) nogil (At this point, the programmer needs to be smart enough to realise that if he calls do_something_heavy() with the GIL released, then the callback is going to get called with the GIL released too, and add a nogil declaration to it as well.) Now he tries to compile it again, but this time Pyrex complains that my_callback is being passed to something that expects a different function signature. So he amends it: # Version 3 cdef void my_callback(void *data) nogil: print "Called back!" The nogil declaration does two different things here: it gives the function the appropriate signature, and it also triggers the generation of code to acquire the GIL if necessary around the body. Now everything okay, and the code compiles. We can take this further. Suppose instead of calling the external function directly, the programmer wants to do some of the calculation in Pyrex, using pure C code, still with the GIL released: # Version 4 cdef void very_heavy(): cdef int i for i in 0 <= i < 10: do_something_heavy(my_callback) def heavy(): without GIL: very_heavy() Whoops - Pyrex complains again, because we haven't declared very_heavy() safe to call without the GIL: # Version 5 cdef void very_heavy() nogil: cdef int i for i in 0 <= i < 10: do_something_heavy(my_callback) Now at this point, Pyrex sees that nothing in the function body requires the GIL -- only C variables and operations are used, and the only function called is declared safe to call without the GIL. So it doesn't generate any code to acquire the GIL in this function. Looking back, the *only* thing the programmer had to be trusted to get right was the two external declarations. All the rest was checked automatically by the compiler, and it was able to optimise away GIL acquiring code when it wasn't needed. I hope this gives a better idea of what I have in mind. -- Greg

Next Message by Date: click to view message preview

Re: [Pyrex] Callbacks from threads and PyGILState_Ensure/PyGILState_Release

Greg Ewing wrote: > Stefan Behnel wrote: >> There is no way Pyrex/Cython can >> always make sure it knows if the GIL is held by the calling code. It's C >> after >> all, things can get called from all over the place. > > The programmer would be responsible for making the > correct declarations when dealing with external C > code, as always. But as long as those declarations > correctly reflect the behaviour of the external > code, Pyrex should be able to check that what the > programmer does with the GIL makes sense. > > Perhaps I can give an example of how I see it working. > Suppose you have a C library with a call that does > some heavy work and takes a callback. You want to > release the GIL around calls to this function. > > As a first attempt, the programmer writes > > # Version 1 > > cdef extern from "heavy.h": > ctypedef void (*callback)(void *) > void do_something_heavy(callback) > > cdef void my_callback(void *data): > print "Called back!" > > def heavy(): > without GIL: > do_something_heavy(my_callback) > > Now Pyrex notices that do_something_heavy() is being > called with the GIL released, but hasn't been declared > as safe to do so, and complains. Ok, that's a simple example as you are passing the callback pointer *outside* of the GIL context *directly* into the function. So you might succeed in making Pyrex smart enough to complain about *some* problems. But how would Pyrex deal with this: cdef class _Context: cdef someStruct* _struct def __init__(self): self._struct = extlib.newSomeStruct() self._struct.callback = callback self._struct.callbackData = <void*>self cdef int calcSomething(self): return True cdef int callback(void* context): return <_Context>context.calcSomething() def heavy(): cdef _Context context context = _Context() without GIL: do_something_heavy(context._struct) Back to my point that Pyrex can't know what's going on. > We can take this further. Suppose instead of calling > the external function directly, the programmer wants to > do some of the calculation in Pyrex, using pure C > code, still with the GIL released: > > # Version 4 > > cdef void very_heavy(): > cdef int i > for i in 0 <= i < 10: > do_something_heavy(my_callback) > > def heavy(): > without GIL: > very_heavy() > > Whoops - Pyrex complains again, because we haven't declared > very_heavy() safe to call without the GIL: > > # Version 5 > > cdef void very_heavy() nogil: > cdef int i > for i in 0 <= i < 10: > do_something_heavy(my_callback) > > Now at this point, Pyrex sees that nothing in the function > body requires the GIL -- only C variables and operations > are used, and the only function called is declared safe > to call without the GIL. So it doesn't generate any code > to acquire the GIL in this function. > > Looking back, the *only* thing the programmer had to > be trusted to get right was the two external declarations. > All the rest was checked automatically by the compiler, > and it was able to optimise away GIL acquiring code > when it wasn't needed. > > I hope this gives a better idea of what I have in mind. I like the idea, but you should not try to make it perfect. There will always be code where Pyrex just can't know what's going on, so if you make it too smart, it will start generating code that crashes and the programmer will have a hard time to make it work. So we need at least both, "nogil" and "with GIL", to override the default behaviour of Pyrex explicitly. Stefan

Previous Message by Thread: click to view message preview

Re: compiling more .pyx and .pxd files into one .so

Stefano Esposito wrote: > Is it possible to write several pyx and pxd files and have 'em > compiled into an unique shared object? It ought to be possible, but would need a bit of hacking. One possibility that comes to mind is to choose one of the .pyx files to be the "main" one that you import from Python, and put explicit calls to the module initialisation functions of the other modules in it. Then link all the .o files together into a single shared object named after the main module. I'm not sure whether you could do this with distutils -- you might need to use a makefile or some such to do the compiling and linking. I may add support for this kind of thing to Pyrex at some point.

Next Message by Thread: click to view message preview

Re: compiling more .pyx and .pxd files into one .so

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gustavo Sverzut Barbieri ÐÐÑÐÑ: > On 9/14/07, Greg Ewing <greg.ewing@xxxxxxxxxxxxxxxx> wrote: >> Stefano Esposito wrote: >>> Is it possible to write several pyx and pxd files and have 'em >> > compiled into an unique shared object? >> >> It ought to be possible, but would need a bit of hacking. >> One possibility that comes to mind is to choose one of the >> .pyx files to be the "main" one that you import from Python, >> and put explicit calls to the module initialisation functions >> of the other modules in it. Then link all the .o files together >> into a single shared object named after the main module. >> >> I'm not sure whether you could do this with distutils -- >> you might need to use a makefile or some such to do the >> compiling and linking. >> >> I may add support for this kind of thing to Pyrex at some >> point. > > It would be great. > > Right now I do "include 'file.pyx'", but it sucks due lack of > dependency tracking and I have to "touch main.pyx" to get it rebuilt. Much worse is namespace pollution that you have in this case. [Â] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG68V8zYr338mxwCURAgg/AKCTsF/RKDZ6wIfEnCZW4i4ChCzr5ACfdMxM WtXkp4B/iSXXUJfrXFfk1iI= =+a/6 -----END PGP SIGNATURE----- _______________________________________________ Pyrex mailing list Pyrex@xxxxxxxxxxxxxxxxx http://lists.copyleft.no/mailman/listinfo/pyrex