osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Progress on Arrow RPC a.k.a. Arrow Flight


 Hey All,

I used my Strata talk today as a forcing function to make additional
progress on a GRPC-based Arrow RPC protocol [1]. I’m calling it “Apache
Arrow Flight”. You can take a look at the work here [2]. I’ll work to clean
up my work and explain my thoughts about the protocol in the coming days.
High-level: use protobuf as a encapsulation format so that any client that
is supported in GRPC will work. However, we can optimize the read/write
path for targeted languages and hand control the
serialization/deserialization and memory handling. (I did that in this Java
patch [3][4][5].) I also looked at starting to use GRPC generated bindings
within Python but it looks like some glue code may be needed in the C++
layer since Python delegates down frequently. I also am still trying to
understand GRPC back-pressure patterns and whether the protocol
realistically needs to change to cover real-world high performance use
cases.

I’ll send out some slides about the ideas and update README, etc. soon.

Thanks,
Jacques

[1]
https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/protobuf/flight.proto
[2] http://github.com/jacques-n/arrow/
[3]
https://github.com/jacques-n/arrow/tree/flight/java/flight/src/main/java/org/apache/arrow/flight/grpc
[4]
https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java#L253
<https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java#L253>
[5]
https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java#L185