osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CoreOptions.TMP_DIRS bug


Hi Oleksandr,

sorry for my late reply. This sounds indeed like a bug. Thanks for opening
FLINK-9762 and the corresponding PR. I'll give it a review shortly so that
we can include it in Flink 1.6.

Cheers,
Till

On Mon, Jul 2, 2018 at 2:31 PM Oleksandr Nitavskyi <o.nitavskyi@xxxxxxxxxx>
wrote:

> Hello guys,
>
> We have discovered minor issue with Flink 1.5 on YARN particularly which
> was related with the way Flink manages temp paths (io.tmp.dirs
> ) in configuration:
> https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#io-tmp-dirs
>
>
> 1.       From what we can see in the code, default option doesn’t
> correspond to reality on YARN or on Mesos deployments. Looks like it equals
> to env variable ‘_FLINK_TMP_DIR’ on Mesos and to `LOCAL_DIRS` on Yarn.
>
> 2.       The issue on Yarn is that it is impossible to have different
> LOCAL_DIRS on JobManager and TaskManager, despite LOCAL_DIRS value depends
> on the container.
>
> The issue is that CoreOptions.TMP_DIRS is configured to the default value
> during JobManager initialization and added to the configuration object.
> When TaskManager is launched the appropriate configuration object is cloned
> with LOCAL_DIRS which makes sense only for Job Manager container. When YARN
> container with TaskManager from his point of view CoreOptions.TMP_DIRS is
> always equal either to path in flink.yml or to the or to the LOCAL_DIRS of
> Job Manager (default behaviour). Is TaskManager’s container do not have an
> access to another folders, that folders allocated by YARN TaskManager
> cannot be started.
>
> Could you please confirm that it is a bug and I will create a Jira ticket
> to track it?
>
> Thanks
> Kind Regards
> Oleksandr Nitavskyi
>
>
>