[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: IPC Example


hi Pearu,

On Sat, Sep 1, 2018 at 3:15 PM Pearu Peterson
<pearu.peterson@xxxxxxxxxxxxx> wrote:
>
> Hi,
>
> I'd also like to enforce the raised question, in particular, it would be very useful to have basic examples of IPC between same or different languages, including C/C++, Python, Java, etc.
>
> Whatever combination of languages is used, the principles of IPC should be the same. For instance, in Python-Java or Python-Python IPC cases, the Python code should not depend on in what language is written the code running another process. Is this understanding correct?

Right, the IPC protocol is not language dependent; this is one of the
raison d'êtres of this project.

>
> Btw, while reading the arrow tests, I noticed a comment
>
>   IPC only supported on Linux
>
> in https://github.com/apache/arrow/blob/master/cpp/src/arrow/gpu/cuda-test.cc#L126
>
> Does this restriction apply only for CUDA IPC or is the comment more general?
> What would it take to add IPC support for Windows or OSX?

This only applies to CUDA IPC. I tried to find a definitive reference
but straight from NVIDIA:

https://github.com/NVIDIA/cuda-samples#cuda-interprocess-communication

We test shared memory IPC on all three platforms in

https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-read-write-test.cc

- Wes

>
> Best regards,
> Pearu
>
> On Sat, Sep 1, 2018 at 9:46 PM, Wes McKinney <wesmckinn@xxxxxxxxx> wrote:
>>
>> + dev@
>>
>> There's several examples of sending record batches from Java to Python
>> (and vice versa) over a network socket -- e.g. Jacques and I are
>> working on a prototype of a general purpose Arrow-native RPC framework
>> in Java and C++ respectively. Where there's some R&D needed is in Java
>> interactions with shared memory. So if you want to do a zero copy read
>> from a memory mapped file, then some development in the Arrow Java
>> libraries is required.
>>
>> I'm not an expert but it seems like Netty has a mechanism to interact
>> with ByteBuffer, which should include MappedByteBuffer
>>
>> https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ReadOnlyUnsafeDirectByteBuf.java#L25
>>
>> Correspondingly, an interface could be developed to enable the Java
>> IPC code path to write to a shared memory region.
>>
>> To the Java developers, could we create some JIRA issues (if there are
>> not already) around Java shared memory IPC?
>>
>> - Wes
>> On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso
>> <alberto_boc_94@xxxxxxxxxx> wrote:
>> >
>> > I want to reinforce this request. I am interested in the same topic.
>> >
>> > I’d like an example specially focused on the creation of a RecordBatch passed from a Java program to a python one and vice-versa.
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Alberto
>> >
>> >
>> >
>> > ________________________________
>> > Da: Clive Cox <cc@xxxxxxxxx>
>> > Inviato: Saturday, September 1, 2018 6:12:29 PM
>> > A: user@xxxxxxxxxxxxxxxx
>> > Oggetto: IPC Example
>> >
>> > Hi,
>> >
>> >  Is there any example of how to do say Java - Python IPC? I'm not sure how to get started.
>> >
>> >  I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communication when everything can be run on a single computer node and low-latency is the goal - hoping to remove the cost of serialization/deserialization and network costs. Would this make sense.
>> >
>> >  Thanks,
>> >
>> >  Clive
>> >
>
>