[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Progress on Arrow RPC a.k.a. Arrow Flight

 Hey All,

I used my Strata talk today as a forcing function to make additional
progress on a GRPC-based Arrow RPC protocol [1]. I’m calling it “Apache
Arrow Flight”. You can take a look at the work here [2]. I’ll work to clean
up my work and explain my thoughts about the protocol in the coming days.
High-level: use protobuf as a encapsulation format so that any client that
is supported in GRPC will work. However, we can optimize the read/write
path for targeted languages and hand control the
serialization/deserialization and memory handling. (I did that in this Java
patch [3][4][5].) I also looked at starting to use GRPC generated bindings
within Python but it looks like some glue code may be needed in the C++
layer since Python delegates down frequently. I also am still trying to
understand GRPC back-pressure patterns and whether the protocol
realistically needs to change to cover real-world high performance use

I’ll send out some slides about the ideas and update README, etc. soon.


[2] http://github.com/jacques-n/arrow/