Re: Some initial GPU questions
Le 26/06/2018 à 17:55, Anthony Scopatz a écrit :
> 2. Should arrow compute assume that everything fits in memory? Arrow
> seem to handle data that is larger than memory via the Buffer API. Are
> there restrictions that using Buffers imply that we should be aware of?
A Buffer certainly assumes that your data can be accessed through its
linear address range; it doesn't assume that everything is present at a
given point in physical memory. So the solution to larger than memory
data would be to use virtual memory facilities, which is how Plasma works.
As for CudaBuffer, I *think* it assumes that everything is in the
GPU-addressable range of memory it describes. Whether or not the GPU
supports some notion of virtual memory depends on the GPU model, I guess?
On the topic of Buffers and GPUs, I'd like to point out the following
issue with the current APIs: