[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Flink 1.6 job cluster mode, job_id is always 00000000000000000000000000000000

Thanks that also works. To avoid same issue with zookeeper, I assume I have to do the same trick?

On Sun, Nov 4, 2018, 03:34 Ufuk Celebi <uce@xxxxxxxxxx> wrote:
Hey Hao Sun,

this has been changed recently [1] in order to properly support
failover in job cluster mode.

A workaround for you would be to add an application identifier to the
checkpoint path of each application, resulting in S3 paths like

Is that a feasible solution?

As a side note: It was considered to keep the job ID fixed, but make
it configurable (e.g. by providing a --job-id argument) which would
also help to avoid this situation, but I'm not aware of any concrete
plans to move forward with that approach.



[1] https://issues.apache.org/jira/projects/FLINK/issues/FLINK-10291

On Sun, Nov 4, 2018 at 3:39 AM Hao Sun <hasun@xxxxxxxxxxx> wrote:
> I am wondering if I can customize job_id for job cluster mode. Currently it is always 00000000000000000000000000000000. I am running multiple job clusters and sharing s3, it means checkpoints will be shared by different jobs as well e.g. 00000000000000000000000000000000/chk-64, how can I avoid this?
> Thanks