OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Arrow sync at 12pm EDT today


Meeting notes from the call:

Attendees/Topics to discuss

   -

   Wes
   -

      Packaging
      -

   Uwe
   -

      Packaging
      -

   Simba
   -

   Li Two Sigma
   -

   Ethan Two Sigma
   -

   Josh Two Sigma
   -

      Exceptions vs status codes
      -

      Class design question
      -

         What data goes in parent/child classes?
         -

      Parquet arrow code location
      -

   Phillip Two Sigma
   -

   Kristizian
   -

      Packaging
      -

   Aneesh


Packaging

   -

   Questions
   -

      Where are the artifacts going?
      -

      BinTray
      -

         Each job posts an artifact to a specific project
         -

         Issues
         -

            Apache bintray cannot be used
            -

               Possibly, but involve infra for encrypting secrets
               -

            Artifacts to upload from the apache org challenging
            -

            Worst case setup arrow pmc packaging project
            -

      Upload to github directly?
      -

         Total size of all binaries?
         -

            Issue there?
            -

         LFS 2GB limit
         -

            Aneesh: very slow
            -

      Other options
      -

         Glacier?
         -

         Buckets?
         -

      Download all artifacts per release
      -

         Usable from github
         -

         Make sure there’s a way to list files
         -

         No-click workflow
         -

      Take crossbow for a spin
      -

      Nightly builds
      -

         Appveyor
         -

         Travis

Parent/Child Class Design Question

   -

   Arrays/Types both have a children list
   -

      Potential errors around assumptions about lists of children for
      arrays that do not have children
      -

   Non-nested types have empty vector, not necessary, perf hit but probably
   not apparent?
   -

   Nested type interface
   -

      Move the child data types to nested class?
      -

   ArrayData idea: POD for data

Parquet Arrow code

   -

   Circular dependency not ideal
   -

   Monorepo?
   -

   Duplicated build system code
   -

      Build scripts in a separate repo
      -

      Potentially more work?
      -

   Need to be careful about how we package scripts
   -

   Shared library of C++ code for apache projects?
   -

      Lots of build scripts, C++ code
      -

      Copypasted things in a few places
      -

   Potentially pull out memory pool
   -

   Mailing list thread about the circular dep issue and reduce build system
   dependency
   -

   Impala has copy pasted code from parquet-cpp apparently

Exceptions

   -

   Status codes potentially not C++
   -

   Uwe
   -

      Exceptions fail in weird ways with different compilers (not caught in
      some cases)
      -

   Josh
   -

      Status codes vs exceptions
      -

   Wes
   -

      Compromise on exceptions internally, and status codes in the public
      API
      -

   Phillip
   -

      Tool for checking for potentially uncaught exceptions

JavaScript PR: https://github.com/apache/arrow/pull/2035

   -

   JavaScript IPC writer
   -

   Streaming/File format
   -

   Needs help with alignment


On Wed, May 16, 2018 at 12:10 PM Wes McKinney <wesmckinn@xxxxxxxxx> wrote:

> Here's a new Hangout:
> https://hangouts.google.com/call/RN8qAVjTdPwXmGZmMx7zAAEE. Let's talk
> there
>
> On Thu, May 17, 2018 at 1:07 AM, Krisztián Szűcs
> <szucs.krisztian@xxxxxxxxx> wrote:
> > Same
> >
> > On May 16 2018, at 6:06 pm, Uwe L. Korn <uwelk@xxxxxxxxxx> wrote:
> >>
> >> On my side I'm waiting to someone to let me in...
> >> On Wed, May 16, 2018, at 6:05 PM, Wes McKinney wrote:
> >> > Google Meet says the meeting is full
> >> >
> >> > On Wed, May 16, 2018, 11:25 AM Alex Hagerman <alex@xxxxxxxxxxxxxxxxx>
> wrote:
> >> > > Aneesh and I had some good conversations during the sprint at
> PyCon. Not
> >> > > sure if he will be on the call today to share, but I won’t be able
> to make
> >> > > it until the next call.
> >> > >
> >> > > Alex
> >> > > From: Wes McKinney
> >> > > Sent: Wednesday, May 16, 2018 11:00 AM
> >> > > To: dev@xxxxxxxxxxxxxxxx
> >> > > Subject: Arrow sync at 12pm EDT today
> >> > >
> >> > > See you at https://meet.google.com/vtm-teks-phx
>