osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Proposed Arrow Graph representations


hi Paul,

This is a great discussion to get started. I will review the patch in
some more detail and send feedback

> I've pushed Joe's initial GraphSchema.fbs to this branch on my Arrow fork

I'm concerned the way this patch is set up right now is a little bit
problematic from an IP lineage standpoint (since this is his/Nvidia's
code and not yours). Would it be possible for Joe to create a pull
request directly for this instead? We can create a branch somewhere
where we can collaborate, too, if that helps.

Thanks,
Wes

On Sat, May 19, 2018 at 11:35 PM, Paul Taylor <ptaylor@xxxxxxxxxx> wrote:
> At GTC San Jose last month, NVidia's Joe Eaton (cc'd) presented on the
> nvGraph <https://developer.nvidia.com/nvgraph> team's goals for
> accelerating in-memory graph processing and analytics. A major component of
> that is advancing and standardizing a common, efficient representation for
> graphs that can support a broad ranges of use-cases, from small to large.
>
> To that end, I'd like to kick off the discussion about native graph
> representations in Arrow.
>
> Joe's team has prepared a preliminary FlatBuffers schema for efficient
> columnar representations of the four most common graph formats. It includes
> embedded edge and vertex property tables, and is designed to be compatible
> with the existing Arrow column types. My initial thoughts are that we could
> add an optional 5th Graph Message type, similar to how Tensor Messages are
> presently implemented.
>
> I've pushed Joe's initial GraphSchema.fbs to this branch on my Arrow fork
> <https://github.com/trxcllnt/arrow/blob/78f6b6c6a5b9e4e7bf96f5bbc4dfed7528b1cca7/format/GraphSchema_Triples_Quads.fbs>.
> From what I understand, the tables have been expanded into separate
> definitions for the sake of comprehension, and the final forms will be
> collapsed into each distinct Graph type, parameterized by sizes defined at
> the top.
>
> I also understand the nvGraph team supports these layouts natively,
> enabling the community to take advantage of high-performance GPU kernels
> very early on, and possibly align with libraries like Hornet
> <https://github.com/hornet-gt/hornetsnest> (previously cuStinger).
>
> Cheers,
> Paul