[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[jira] [Created] (ARROW-3587) Efficient serialization for Arrow Objects (array, table, tensor, etc)

Siyuan Zhuang created ARROW-3587:

             Summary: Efficient serialization for Arrow Objects (array, table, tensor, etc)
                 Key: ARROW-3587
                 URL: https://issues.apache.org/jira/browse/ARROW-3587
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++, Plasma (C++), Python
            Reporter: Siyuan Zhuang

Currently, Arrow seems to have poor serialization support for its own objects.

For example,
import pyarrow 
arr = pyarrow.array([1, 2, 3, 4]) 
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "pyarrow/serialization.pxi", line 337, in pyarrow.lib.serialize
 File "pyarrow/serialization.pxi", line 136, in pyarrow.lib.SerializationContext._serialize_callback
 pyarrow.lib.SerializationCallbackError: pyarrow does not know how to serialize objects of type <class 'pyarrow.lib.Int64Array'>.

I am working Ray & modin project, using plasma to store Arrow objects. Lack of direct serialization support harms the performance, so I would like to push a PR to fix this problem.
I wonder if it is welcome or is there someone else doing it?

This message was sent by Atlassian JIRA