[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

array.array()'s memory shared with multiprocessing.Process()

On 12/09/17 10:21, gerlando.falauto at gmail.com wrote:
> Il giorno luned? 11 settembre 2017 12:19:27 UTC+2, Thomas Jollans ha scritto:
>> On 2017-09-10 23:05, iurly wrote:
>>> As far as I'm concerned, I'm probably better off using double buffers to avoid this kind of issues.
>>> Thanks a lot for your help!
>> That should work. Some other things to consider, if both processes are
>> on the same machine, are a series of memory-mapped files to pass the
>> data without pickling, or, if your code is only going to run on Unix,
>> something involving shared memory through multiprocessing.Array.
> Oh, I see, multiprocessing.Array() sounds like a pretty good idea, thanks!
> It's simple enough and actually already compatible with what I'm doing.
> That would involve double buffers which I would have to use as the first optimization step anyway.
> Notice however how I'd have to create those Arrays dynamically in the producer thread. Would I then be able to pass them to the consumer by putting a reference in a queue? I wouldn't want them to be pickled at all in that case, of course.
> Also, why do you say it only works on Unix? I couldn't find any reference to such limitation in the documentation.

I'm not sure actually. Maybe someone else here can help. I have a hunch
that on Windows the memory might not be shared in the same way that it
would on Linux/Unix, since Windows simply doesn't have the same process
forking capabilities as Unix. This stack overflow question appears to be
in agree with that hunch, but I can't judge whether something else isn't
going wrong there:

If the consumer process is always working from the same shared buffer,
then you don't have to pass any data through the queue at all! (Or, if
you have a small number of shared buffers, just tell the consumer which
one to look at) You do have to somehow guarantee that the producer
thread doesn't overwrite the buffer while it is being processed, of
course. I'll let you be the judge of how well this kind of scheme would
work for your data processing and acquisition rates.

> Thank you so much again!

Thomas Jollans