[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

plan for Go implementation of Plasma

Hi all!

I am a developer on qri <https://qri.io/>, a data-science tool built on
IPFS written in go. We're interested in integrating Arrow and especially
Plasma, in order to be able to share datasets with other apps like Jupyter
Notebook. Having this functionality is going to be key for how we plan to
integrate with existing frameworks.

I've been investigating possible approaches for how to use Plasma in our
codecase. I realize that Plasma is still a work in progress, and doesn't
have stable API yet, but we're also a ways off from being ready to fully
integrate it on our side. Just figured it would be good to start this
conversation early in order to plan ahead for how development should

So, the prototypes I've been hacking on have revealed a few choices of how
to make our golang codebase call Plasma's C++, and I wanted to see what the
Plasma devs think about these approaches or if they have any preference for
how the go bindings should be behave.

Here are the options in order of what seems to be least to most usable:

1. cgo
  Use go's builtin cgo facility to call the Plasma C++ implementation. cgo
is relatively easy to use, however it only can call C functions. So this
would require writing and maintaining a pure C language wrapper around the
C++ functionality we want to expose. A lot would be lost in translation and
the resulting go code would look nothing like the original library.

2. dlopen
  Install Plasma as a library on the user's system, then load the library
at run-time, looking up function calls and data structures as needed.
Removes the need for a static dependency, but still requires a lot of shim
code to be written to load the shared library calls. C++'s name mangling
gets in the way a lot.

3. Swig
  Wrap a swig interface file that exposes whatever functionality we want to
golang. The standard go tool has builtin swig support, which knows when to
invoke the swig generator, in order to create go bindings that resemble the
C++ original. The build process is relatively uninterrupted.

I noticed there doesn't seem to be any swig in use currently in the arrow
codebase, which made me think there might have been a reason that it has
been avoided for other languages. I'm interested to hear any thoughts, or
see if there are other suggestions on how to proceed?