osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: questions about arrow/go


hi Matt,

Arrow in Go is still very much WIP territory AFAICT. I've been hoping
to see more of a developer community grow here but we still have a
ways to go it seems. You can help get one going =)

On Wed, Oct 10, 2018 at 2:53 PM Matthew Perpick <mattp@xxxxxxxxxxxxx> wrote:
>
> I'm experimenting with arrow in an online streaming analytics project. It
> looks like it could potentially be a great fit but i'm hitting a few
> roadblocks.
>
> A goal for my project is to stream data into pre-allocated arrays in memory
> and include them in analytics with arrays stored elsewhere (db, disk, etc).
> I'm currently hitting a barrier with Array Builders. They can't produce an
> array without resetting their memory (of course, correct me if I'm wrong).
> This forces me to couple the sizes of my buffers in memory/disk with the
> query rate rather than something more logical like time elapsed, bytes,
> number of elements.
>
> Ultimately, I'd like to have array builders be able to produce an unsafe
> array which shares the same memory buffers. I'd have to protect
> resizes/copies/resets with mutexes, but I believe it should work. Is this a
> case you'd like to support with Arrow? I have an incomplete PR
> <https://github.com/clutchski/arrow/pull/1/files>which illustrates the
> idea, which I could complete if you all think it's a reasonable direction.

If you could open a JIRA issue and describe the change you are
proposing to make, it would be then good to go ahead and open a PR to
stir the pot. Some of the existing contributors to the Go library will
be able to comment

>
> One other question: The Go implementation seems pretty incomplete. If i
> wanted to better understand Arrow's full capabilities, which implementation
> is most complete? C++?

The deepest library is indeed the C++ library, so that would be a good
place to start.

- Wes

>
> Thanks for all the work so far.
> Matt