OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Flink 1.6 job cluster mode, job_id is always 00000000000000000000000000000000


Hey Hao Sun,

this has been changed recently [1] in order to properly support
failover in job cluster mode.

A workaround for you would be to add an application identifier to the
checkpoint path of each application, resulting in S3 paths like
application-XXXX/00...00/chk-64.

Is that a feasible solution?

As a side note: It was considered to keep the job ID fixed, but make
it configurable (e.g. by providing a --job-id argument) which would
also help to avoid this situation, but I'm not aware of any concrete
plans to move forward with that approach.

Best,

Ufuk

[1] https://issues.apache.org/jira/projects/FLINK/issues/FLINK-10291
On Sun, Nov 4, 2018 at 3:39 AM Hao Sun <hasun@xxxxxxxxxxx> wrote:
>
> I am wondering if I can customize job_id for job cluster mode. Currently it is always 00000000000000000000000000000000. I am running multiple job clusters and sharing s3, it means checkpoints will be shared by different jobs as well e.g. 00000000000000000000000000000000/chk-64, how can I avoid this?
>
> Thanks