Since you use the message-buffer-process as a dependency and the error tells you class not found, have you ever check your application jar package whether containing the wanted MessageBufferProcess.class? If not existed, try to use assembly-maven or shaded-maven plugin to include your classes.
From: Ly, The Anh <the.a.ly@xxxxxxxxxxxxxxxxxxx>
Sent: Friday, November 2, 2018 6:33
Subject: Starting a seperate Java process within a Flink cluster
I am currently working on my masters and I encountered a difficult problem.
Background (for context): I am trying to connect different data stream processors. Therefore i am using Flink's internal mechanisms of creating custom sinks and sources to receive from and send to different data stream processors. I am starting a separate
process (message-buffer-process) in those custom sinks and sources to communicate and buffer data into that message-buffer-process. My implementation is created with Maven and it could potentially be added as an dependency.
Problem: I already tested my implementation by adding it as an dependency to a simple Flink word-count example. The test was within an IDE which works perfectly fine. But when i package that Flink work-count example and try
to run it with "./flink run " or by uploading and submitting it as a job, it tells me that my buffer-process-class could not be found:
In German: "Fehler: Hauptklasse de.tuberlin.mcc.geddsprocon.messagebuffer.MessageBufferProcess konnte nicht gefunden oder geladen werden"
Roughly translated: "Error: Main class de.tuberlin.mcc.geddsprocon.messagebuffer.MessageBufferProcess could not be found or loaded"
Example - Adding my custom sink to send data to another data stream processor:
dataStream.addSink( (SinkFunction)DSPConnectorFactory .getInstance() .createSinkConnector( new DSPConnectorConfig .Builder("localhost", 9656) .withDSP("flink") .withBufferConnectorString("buffer-connection-string") .withHWM(20) .withTimeout(10000) .build()));