osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: User jar is present in the flink job manager's class path


Hi,

Could it be that you are submitting the job in attached mode, i.e., without -d
parameter? In the "job cluster attached mode", we actually start a Flink
session cluster (and stop it again from the CLI) [1]. Therefore, in attached
mode, the config option "yarn.per-job-cluster.include-user-jar" is effectively
ignored. If you submit with -d, a "true job cluster" is started, and the user
jar should be added to the system classpath. Alternatively, if the detached
mode is not an option for you, you could add a jar with your custom logger
implementation to the flink /lib directory.

If the behavior in Flink 1.3 is indeed different, then I would consider this a
regression. Can you open a jira issue for that?

Best,
Gary

[1] https://github.com/apache/flink/blob/d13015e23c805e1aaefdc7d8037f2fa87ea74830/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L261

On Thu, Oct 11, 2018 at 3:54 PM Timo Walther <twalthr@xxxxxxxxxx> wrote:
Yes, you are right. I was not aware that the resolution order depends on
the cluster deployment. I will loop in Gary (in CC) that might know
about such a YARN setup.

Regards,
Timo

Am 11.10.18 um 15:47 schrieb yinhua.dai:
> Hi Timo,
>
> I didn't tried to configure the classloader order, according to the
> document, it should only be needed for yarn-session mode, right?
>
> I can see the ship files(-yt /path/dir/) is present in job manager's class
> path, so maybe I should put my uber jar in the -yt path so that it will be
> shipped and add to class path in flink 1.5?
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/