[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[jira] [Created] (FLINK-10205) Batch Job: InputSplit Fault tolerant for DataSourceTask

JIN SUN created FLINK-10205:

             Summary: Batch Job: InputSplit Fault tolerant for DataSourceTask
                 Key: FLINK-10205
                 URL: https://issues.apache.org/jira/browse/FLINK-10205
             Project: Flink
          Issue Type: Sub-task
          Components: JobManager
            Reporter: JIN SUN

Today DataSource Task pull InputSplits from JobManager to achieve better performance, however, when a DataSourceTask failed and rerun, it will not get the same splits as its previous version. this will introduce inconsistent result or even data corruption.

Furthermore,  if there are two executions run at the same time (in batch scenario), this two executions should process same splits.

we need to fix the issue to make the inputs of a DataSourceTask deterministic. The propose is save all splits into ExecutionVertex and DataSourceTask will pull split from there.


This message was sent by Atlassian JIRA