osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

about PCollection process


hello,every
        I'm running a Beam pipeline which uses the TextIO read same text from text file, PTransform  peer line search hbase.  result is   PCollection<PCollection<KV<String, RecordData>>>


@Override
public PCollection<PCollection<KV<String, RecordData>>> expand(PCollection<String> lines) {

PCollection<PCollection<KV<String, RecordData>>> results = lines
.apply(ParDo.of(new DoFn<String, PCollection<KV<String, RecordData>>>() {
@ProcessElement
public void processElement(ProcessContext c) {
String vin = c.element();

Pipeline pipelineHbase = Pipeline.create(c.getPipelineOptions());

HBaseIO.Read read = HBaseIO.read().withConfiguration(conf).withTableId(hbaseTable).withKeyRange(
Bytes.toBytes(String.format("%s-%s", vin, startTime)),
Bytes.toBytes(String.format("%s-%s", vin, endTime)));
PCollection<Result> results = pipelineHbase.apply(read);

PCollection<KV<String, RecordData>> recordResults = results
.apply(ParDo.of(new Result2RecordNoModifyDataFn()));

c.output(recordResults);
}

}));

return results;
}


what process PCollection<PCollection<KV<String, RecordData>>> ????