Hey Hao Sun,
this has been changed recently  in order to properly support
failover in job cluster mode.
A workaround for you would be to add an application identifier to the
checkpoint path of each application, resulting in S3 paths like
Is that a feasible solution?
As a side note: It was considered to keep the job ID fixed, but make
it configurable (e.g. by providing a --job-id argument) which would
also help to avoid this situation, but I'm not aware of any concrete
plans to move forward with that approach.
On Sun, Nov 4, 2018 at 3:39 AM Hao Sun <hasun@xxxxxxxxxxx
> I am wondering if I can customize job_id for job cluster mode. Currently it is always 00000000000000000000000000000000. I am running multiple job clusters and sharing s3, it means checkpoints will be shared by different jobs as well e.g. 00000000000000000000000000000000/chk-64, how can I avoid this?