osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Flink Yarn Deployment Issue - 1.7.0


Can you check the Flink log files? You should get there a better description of the error.

> Am 08.12.2018 um 18:15 schrieb sohimankotia <sohimankotia@xxxxxxxxx>:
> 
> Hi ,
> 
> I have installed flink-1.7.0 Hadoop 2.7 scala 2.11 .  We are using
> hortonworks hadoop distribution.(hdp/2.6.1.0-129/)
> 
> *Flink lib folder looks like :*
> 
> 
> -rw-r--r-- 1 hdfs hadoop 93184216 Nov 29 02:15 flink-dist_2.11-1.7.0.jar
> -rw-r--r-- 1 hdfs hadoop    79219 Nov 29 03:33
> flink-hadoop-compatibility_2.11-1.7.0.jar
> -rw-r--r-- 1 hdfs hadoop   141881 Nov 29 02:13 flink-python_2.11-1.7.0.jar
> -rw-r--r-- 1 hdfs hadoop   489884 Nov 28 23:01 log4j-1.2.17.jar
> -rw-r--r-- 1 hdfs hadoop     9931 Nov 28 23:01 slf4j-log4j12-1.7.15.j
> 
> *My code :*
> 
>       ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
> 
>       String p = args[0];
> 
> 
>       Job job = Job.getInstance();
>       SequenceFileInputFormat<Text, BytesWritable> inputFormat = new
> SequenceFileInputFormat<>();
> 
> job.getConfiguration().setBoolean(FileInputFormat.INPUT_DIR_RECURSIVE,
> true);
>       final HadoopInputFormat<Text, BytesWritable> hInputEvents =
> HadoopInputs.readHadoopFile(inputFormat, Text.class, BytesWritable.class, p,
> job);
>       org.apache.flink.configuration.Configuration fileReadConfig = new
> org.apache.flink.configuration.Configuration();
> 
>       env.createInput(hInputEvents)
>               .output(new PrintingOutputFormat<>());
> 
> 
> *pom.xml*
> 
> flink.version = 1.7.0
> 
>    <dependency>
>      <groupId>org.apache.flink</groupId>
>      <artifactId>flink-java</artifactId>
>      <version>${flink.version}</version>
>      <scope>provided</scope>
>    </dependency>
>    <dependency>
>      <groupId>org.apache.flink</groupId>
>      <artifactId>flink-clients_2.11</artifactId>
>      <version>${flink.version}</version>
>      <scope>provided</scope>
>    </dependency>
>    <dependency>
>      <groupId>org.apache.flink</groupId>
>      <artifactId>flink-streaming-java_2.11</artifactId>
>      <version>${flink.version}</version>
>      <scope>provided</scope>
>    </dependency>
> 
>    <dependency>
>      <groupId>org.apache.flink</groupId>
>      <artifactId>flink-hadoop-compatibility_2.11</artifactId>
>      <version>${flink.version}</version>
>      <scope>provided</scope>
>    </dependency>
> 
>    <dependency>
>      <groupId>org.apache.flink</groupId>
>      <artifactId>flink-shaded-hadoop2</artifactId>
>      <version>${flink.version}</version>
>      <scope>provided</scope>
>    </dependency>
> 
> *
> in script :*
> 
> 
> 
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export HADOOP_CLASSPATH="/usr/hdp/2.6.1.0-129/hadoop/hadoop-*":`hadoop
> classpath`
> 
> echo ${HADOOP_CLASSPATH}
> 
> PARALLELISM=1
> JAR_PATH="jar"
> CLASS_NAME="CLASS_NAME"
> NODES=1
> SLOTS=1
> MEMORY_PER_NODE=2048
> QUEUE="default"
> NAME="sample"
> 
> IN="input-file-path"
> 
> 
> /home/hdfs/flink-1.7.0/bin/flink run -m yarn-cluster  -yn ${NODES} -yqu
> ${QUEUE} -ys ${SLOTS} -ytm ${MEMORY_PER_NODE} --parallelism ${PARALLELISM}
> -ynm ${NAME} -c ${CLASS_NAME} ${JAR_PATH} ${IN} 
> 
> 
> *where classpath is printing:*
> 
> /usr/hdp/2.6.1.0-129/hadoop/hadoop-*:/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*:/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf
> 
> 
> But I am getting class not found error for hadoop related jar . Error is
> attached .
> 
> 
> error.txt
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t894/error.txt>  
> *Another Problem :*
> 
> If i added hadoop shaded jar in lib folder
> 
> 
> -rw-r--r-- 1 hdfs hadoop 93184216 Nov 29 02:15 flink-dist_2.11-1.7.0.jar
> -rw-r--r-- 1 hdfs hadoop    79219 Nov 29 03:33
> flink-hadoop-compatibility_2.11-1.7.0.jar
> -rw-r--r-- 1 hdfs hadoop   141881 Nov 29 02:13 flink-python_2.11-1.7.0.jar
> *-rw-r--r-- 1 hdfs hadoop 41130742 Dec  8 22:38
> flink-shaded-hadoop2-uber-1.7.0.jar*
> -rw-r--r-- 1 hdfs hadoop   489884 Nov 28 23:01 log4j-1.2.17.jar
> -rw-r--r-- 1 hdfs hadoop     9931 Nov 28 23:01 slf4j-log4j12-1.7.15.jar
> 
> I am getting following error. And this is happening for all version greater
> than 1.4.2 .
> 
> java.lang.IllegalAccessError: tried to access method
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object;
> from class
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider
>    at
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75)
>    at
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163)
>    at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
>    at
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
>    at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187)
>    at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>    at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:985)
>    at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:273)
>    at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:451)
>    at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:96)
>    at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:224)
> 
> 
> Thanks in advance .
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/