osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Thoughts about 2019 Arrow development focus areas


Wes,

This is very exciting. Thanks for writing up the detailed document.

I think it is time for me to start brushing up on modern C++.

Andy.

On Thu, Dec 6, 2018 at 8:37 PM Wes McKinney <wesmckinn@xxxxxxxxx> wrote:

> hi folks,
>
> I jotted down some high level ideas about directions I'd like to push
> the various parts of the project on the C++ side along with the
> language bindings in Python, R, Ruby, and others. Many people may know
> that I am building a not-for-profit open source development team to
> focus on Apache Arrow (https://ursalabs.org/), so this document is
> partly for my colleagues to organize some lower-level technical
> discussions and planning in the Arrow JIRA. I'm interested from
> feedback from the whole Arrow community, and we obviously would love
> to have as many people as possible involved who have an interest in
> the C++ libraries and their bindings.
>
> The simplified summary is that I would like to work toward an
> embeddable in-memory query engine in C++ that can be used in all the
> bindings. This can be used in numerous contexts, from data frame
> libraries to streaming data transformation. As a simple example, we
> could compile filter expressions with Gandiva and apply these to a
> stream of record batches being materialized from a directory of
> Parquet files.
>
> There's a lot of pieces that still have to fall into place to do this
> in a sustainable and non-hacky way.
>
>
> https://docs.google.com/document/d/12dWBniKW2JQ-5djE3SPjyQXVquCAEmLXVlb1dnhLhQ0/edit#heading=h.62rx18p423rw
>
> Looking forward to the feedback of others!
>
> Thanks
> Wes
>