OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Unification of Hadoop related IO modules


In next release it will be still compatible because we keep module “hadoop-input-format” but we make it deprecated and propose to use it through module “hadoop-format” and proxy class HadoopFormatIO (or HadoopMapReduceFormatIO, whatever we name it) which will provide Write/Read functionality by using MapReduce InputFormat or OutputFormat classes. 
Then, in future releases after next one, we can drop “hadoop-input-format”  since it was deprecated and we provided a time to move to new API. I think this is less painful way for user but most complicated for us if the final goal it to merge “hadoop-input-format” and “hadoop-output-format” together.

On 7 Sep 2018, at 13:45, Robert Bradshaw <robertwb@xxxxxxxxxx> wrote:

Agree about not impacting users. Perhaps I misread (3), isn't it fully backwards compatible as well? 

On Fri, Sep 7, 2018 at 1:33 PM Jean-Baptiste Onofré <jb@xxxxxxxxxxxx> wrote:
Hi,

in order to limit the impact for the existing users on Beam 2.x series,
I would go for (1).

Regards
JB

On 06/09/2018 17:24, Alexey Romanenko wrote:
> Hello everyone,
>
> I’d like to discuss the following topic (see below) with community since
> the optimal solution is not clear for me.
>
> There is Java IO module, called “/hadoop-input-format/”, which allows to
> use MapReduce InputFormat implementations to read data from different
> sources (for example, org.apache.hadoop.mapreduce.lib.db.DBInputFormat).
> According to its name, it has only “Read" and it's missing “Write” part,
> so, I'm working on “/hadoop-output-format/” to support MapReduce
> OutputFormat (PR 6306 <https://github.com/apache/beam/pull/6306>). For
> this I created another module with this name. So, in the end, we will
> have two different modules “/hadoop-input-format/” and
> “/hadoop-output-format/” and it looks quite strange for me since, afaik,
> every existed Java IO, that we have, incapsulates Read and Write parts
> into one module. Additionally, we have “/hadoop-common/” and
> /“hadoop-file-system/” as other hadoop-related modules. 
>
> Now I’m thinking how it will be better to organise all these Hadoop
> modules better. There are several options in my mind: 
>
> 1) Add new module “/hadoop-output-format/” and leave all Hadoop modules
> “as it is”. 
> Pros: no breaking changes, no additional work 
> Cons: not logical for users to have the same IO in two different modules
> and with different names.
>
> 2) Merge “/hadoop-input-format/” and “/hadoop-output-format/” into one
> module called, say, “/hadoop-format/” or “/hadoop-mapreduce-format/”,
> keep the other Hadoop modules “as it is”.
> Pros: to have InputFormat/OutputFormat in one IO module which is logical
> for users
> Cons: breaking changes for user code because of module/IO renaming 
>
> 3) Add new module “/hadoop-format/” (or “/hadoop-mapreduce-format/”)
> which will include new “write” functionality and be a proxy for old
> “/hadoop-input-format/”. In its turn, “/hadoop-input-format/” should
> become deprecated and be finally moved to common “/hadoop-format/”
> module in future releases. Keep the other Hadoop modules “as it is”.
> Pros: finally it will be only one module for hadoop MR format; changes
> are less painful for user
> Cons: hidden difficulties of implementation this strategy; a bit
> confusing for user 
>
> 4) Add new module “/hadoop/” and move all already existed modules there
> as submodules (like we have for “/io/google-cloud-platform/”), merge
> “/hadoop-input-format/” and “/hadoop-output-format/” into one module. 
> Pros: unification of all hadoop-related modules
> Cons: breaking changes for user code, additional complexity with deps
> and testing
>
> 5) Your suggestion?..
>
> My personal preferences are lying between 2 and 3 (if 3 is possible). 
>
> I’m wondering if there were similar situations in Beam before and how it
> was finally resolved. If yes then probably we need to do here in similar
> way.
> Any suggestions/advices/comments would be very appreciated.
>
> Thanks,
> Alexey

--
Jean-Baptiste Onofré
jbonofre@xxxxxxxxxx
http://blog.nanthrax.net
Talend - http://www.talend.com