[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Adding CudaBuffer to pyarrow


Hi Wes,

Thanks for the hints!

I have created a very minimal GPU support to pyarrow, see


https://github.com/apache/arrow/compare/master...Quansight:pearu-cuda-pyarrow?expand=1

This includes:
1. introducing libarrow_gpu that is built into pyarrow.lib_gpu extension
module, currently only CudaBuffer is exposed
2. updating cmake configuration files to detect libarrow_gpu.{so,a}
libraries
3. updating setup.py with new options --with{,out}-arrow-gpu and
PYARROW_BUILD_ARROW_GPU env variable support.

Let me know if something in my approach needs to be changed.

Regarding packaging:
1. FYI, there is a cudatoolkit-dev package in conda-forge:
  https://github.com/conda-forge/cudatoolkit-dev-feedstock

2. One approach is to introduce new conda packages arrow-cpp-gpu and
pyarrow-gpu that are build using -DARROW_GPU and --with-arrow-gpu,
respectively.

Best regards,
Pearu

On Wed, Aug 29, 2018 at 12:32 AM, Wes McKinney <wesmckinn@xxxxxxxxx> wrote:

> hi Pearu,
>
> Sounds good to me; I had wanted to do it already but got caught up
> working on other things. You'll probably want to create
> libarrow_gpu.pxd since this is an optional library. Otherwise it
> should be reasonably straightforward.
>
> some relevant issues
>
> - https://issues.apache.org/jira/browse/ARROW-1424
> - https://issues.apache.org/jira/browse/ARROW-2512
>
> If we wanted to ship Python wheels and conda packages with CUDA
> support built in, what would we have to do (currently these binaries
> are produced in Circle CI, Travis CI, and Appveyor)?
>
> - Wes
> On Tue, Aug 28, 2018 at 4:34 PM Pearu Peterson
> <pearu.peterson@xxxxxxxxxxxxx> wrote:
> >
> > Hi,
> >
> > Is there any interest in implementing CudaBuffer support in pyarrow? Are
> > there any obstacles that would prevent to do this?
> >
> > I would be willing to give it a try.
> >
> > As I understand, the entry point for this task would be to patch
> > pyarrow/includes/libarrow.pxd . Any hints for what to take into account
> are
> > appreciated.
> >
> > Best regards,
> > Pearu
>