[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Split one dataset into multiple

Hi madan,

I think you need to hash partition your records. 
Flink supports hash partitioning of data. 
The operator is keyBy. 
If the value of your tag field is enumerable, you can also use split/select to achieve your purpose.

Thanks, vino.

madan <madan.yellanki@xxxxxxxxx> 于2018年11月5日周一 下午6:37写道:

I have a custom iterator which gives data of multitple entities. For example iterator gives data of Department, Employee and Address. Record's entity type is identified by a field value. And I need to apply different set of operations on each dataset. Ex., Department data may have aggregations, Employee and Address data are simply joined together after some filteration. 

If I have different datasets for each entity type the job is easy. So I am trying to split incoming data to different datasets. What is the best possible way to achieve this ?

*Iterator can be read only once.

Thank you,