OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Akka Http used in custom RichSourceFunction


Hi Piotrek,

Thank you for your response!

I am currently just testing the job in a local environment. I think that means all classes are in the Java classpath, which might also be the issue then. 
If I am correct that means I am currently not using dynamic classloading and just overwriting the Akka version, also for Flink.

I will try moving my websocket connector to a seperate package and shade it's Akka dependency.


The full stack trace of the exception (I think this is the shutdown of the Flink minicluster):

Exception in thread "main" java.lang.NoSuchMethodError: akka.actor.ActorSystem.shutdown()V
at org.apache.flink.runtime.minicluster.FlinkMiniCluster$$anonfun$startInternalShutdown$5$$anonfun$apply$7.apply(FlinkMiniCluster.scala:483)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster$$anonfun$startInternalShutdown$5$$anonfun$apply$7.apply(FlinkMiniCluster.scala:483)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster$$anonfun$startInternalShutdown$5.apply(FlinkMiniCluster.scala:483)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster$$anonfun$startInternalShutdown$5.apply(FlinkMiniCluster.scala:483)
at scala.Option.foreach(Option.scala:257)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.startInternalShutdown(FlinkMiniCluster.scala:482)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.stop(FlinkMiniCluster.scala:434)
at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:112)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1501)
at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:629)
at net.vankaam.flink.WebSocketSample$.main(WebSocketSample.scala:42)
at net.vankaam.flink.WebSocketSample.main(WebSocketSample.scala)


Cheers,
Niels


On Thu, May 24, 2018 at 4:08 PM Piotr Nowojski <piotr@xxxxxxxxxxxxxxxxx> wrote:
Hi,

Please take a look on https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/debugging_classloading.html . Especially check if you are using child-first class loading config. If it doesn’t help, probably you should shade your akka dependency. 

What is the full exception? Is it thrown when YOURS code tries to shutdown, or when FLINK’s code tries to shutdown?

Piotrek


On 24 May 2018, at 14:38, Niels van Kaam <niels@xxxxxxxxxxx> wrote:

Hi All,

I wrote a custom source function (RichSourceFunction) which connects to a web socket using the Akka Http Library. The job using this source runs fine on a local environment until upon shutdown I see the following error in the log: "Exception in thread "main" java.lang.NoSuchMethodError: akka.actor.ActorSystem.shutdown()V"

My impression is the issue is caused by a version conflict between flink's akka dependency and my own one (due to akka http). This seems to be related to this issue: https://issues.apache.org/jira/browse/FLINK-9240

Can I somehow avoid this conflict?
If not, does this mean I should avoid using Akka (or at least other versions than Flink's) within my sources/sinks?
Or can I safely catch and ignore the error? 

My dependencies are:
Flink: 1.4.2
akka-actor: 2.5.12
akka-stream: 2.5.12
akka-http: 10.1.1

Thank you for your help!

Cheers,
Niels