OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: connection failed when running flink in a cluster


Hi Felipe,

From the exception information, it seems that you did not start the socket server, the socket source needs to connect to the socket server.

Please make sure the socket server has started and is available.

Thanks, vino.

2018-08-06 18:45 GMT+08:00 Felipe Gutierrez <felipe.o.gutierrez@xxxxxxxxx>:
yes.

when I execute the jps command on the master node I see TaskManagerRunner and StandaloneSessionClusterEntrypoint (which I believe it is the  jobManager). On the slave nodes I see TaskManagerRunner when I run jps command


--
-- Felipe Gutierrez
-- skype: felipe.o.gutierrez


On Mon, Aug 6, 2018 at 12:13 PM miki haiat <miko5054@xxxxxxxxx> wrote:
Did you start job manager and task manager on the same resbery pi ?

On Mon, 6 Aug 2018, 12:01 Felipe Gutierrez, <felipe.o.gutierrez@xxxxxxxxx> wrote:
Hello everyone,

I am trying to run Flink on Raspberry Pis. My first test for word count in a single node worked. I just have to decrease the Heap memory of the jobmanager.heap.mb and taskmanager.heap.mb to 512.
My second test is to add 2 slave nodes I got the error: "Java HotSpot(TM) Client VM warning: G1 GC is disabled in this release." at the file log/flink-root-taskexecutor-0-*.out.

This link (https://blog.sflow.com/2016/06/raspberry-pi-real-time-network-analytics.html) says that in order to Raspberry Pi ARM architecture works with JVM it is necessary to configure the JVM as:
-Xms600M
-Xmx600M
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode

then I set this variables on the path inside the file flink-conf.yaml
env.java.opts: "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
env.java.opts.jobmanager: "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
env.java.opts.taskmanager: "-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"

and the error "Java HotSpot(TM) Client VM warning: G1 GC is disabled in this release." is not showing anymore. However, the connection from the master node to the slave node is still not possible. Does anybody know how I must configure flink to deal with that?

This is the error stack trace:

2017-05-25 12:40:26,421 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Socket Stream -> Flat Map (1/1) (b81b6492fc0860367be422d0b0bf4358) switched from DEPLOYING to RUNNING.
2017-05-25 12:40:26,891 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Socket Stream -> Flat Map (1/1) (b81b6492fc0860367be422d0b0bf4358) switched from RUNNING to FAILED.
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.flink.streaming.api.functions.source.SocketTextStreamFunction.run(SocketTextStreamFunction.java:96)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
at java.lang.Thread.run(Thread.java:745)
2017-05-25 12:40:26,898 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job Socket Window WordCount (71c6d7796eccf6587d9d1deda0490e09) switched from state RUNNING to FAILING.
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.flink.streaming.api.functions.source.SocketTextStreamFunction.run(SocketTextStreamFunction.java:96)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
at java.lang.Thread.run(Thread.java:745)
2017-05-25 12:40:26,921 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, ReduceFunction$1, PassThroughWindowFunction) -> Sink: Print to Std. Out (1/1) (aa1a0e7ee3a1d3ad8f99b2608bd64c5b) switched from RUNNING to CANCELING.
2017-05-25 12:40:26,975 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, ReduceFunction$1, PassThroughWindowFunction) -> Sink: Print to Std. Out (1/1) (aa1a0e7ee3a1d3ad8f99b2608bd64c5b) switched from CANCELING to CANCELED.



Thanks, Felipe
--
-- Felipe Gutierrez
-- skype: felipe.o.gutierrez