osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Flink 1.5 - Job fails to execute in multiple taskmanagers (parallelism > 1)


Two quick questions:

  - do you explicitly configure Flink memory onheap / offheap?
  - can you check whether this also happens when SSL is disabled?


On Thu, May 24, 2018 at 6:21 PM, Edward Rojas <edward.rojascl@xxxxxxxxx>
wrote:

> Hi all,
>
> I was testing Flink 1.5 rc5 and I found this issue. I'm running a cluster
> in
> HA mode with one jobmanager, several taskmanagers, each one with two task
> slots and default parallelism set to 2.
>
> I'm running two jobs, one simple one with a kafka consumer, a filter and a
> sink. The other a little bit more complex with a kafka consumer, filters,
> flatmaps, keyed process functions and sinks.
>
> Both jobs run correctly when they are assigned to run in the 2 slots of the
> same taskmanager.
>
> But when one slot in in one taskmanager and the other in a different one,
> the simpler job runs cor correctly but the complex one fails with the
> following error:
>
> org.apache.flink.runtime.io.network.netty.exception.
> LocalTransportException:
> java.lang.UnsupportedOperationException: Heap buffer
>         at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestCli
> entHandler.exceptionCaught(CreditBasedPartitionRequestCli
> entHandler.java:170)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:275)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireExceptionCaught(
> AbstractChannelHandlerContext.java:253)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.
> java:131)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:275)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireExceptionCaught(
> AbstractChannelHandlerContext.java:253)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.
> java:131)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:275)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireExceptionCaught(
> AbstractChannelHandlerContext.java:253)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerAdapter.
> exceptionCaught(ChannelHandlerAdapter.java:79)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:275)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireExceptionCaught(
> AbstractChannelHandlerContext.java:253)
>         at
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.
> SslHandler.exceptionCaught(SslHandler.java:697)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeExceptionCaught(
> AbstractChannelHandlerContext.java:275)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.notifyHandlerException(
> AbstractChannelHandlerContext.java:809)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:341)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:324)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.
> fireChannelRead(DefaultChannelPipeline.java:847)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.
> AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.
> processSelectedKey(NioEventLoop.java:511)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.
> processSelectedKeysOptimized(NioEventLoop.java:468)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.
> processSelectedKeys(NioEventLoop.java:382)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.
> NioEventLoop.run(NioEventLoop.java:354)
>         at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.
> SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> java.lang.UnsupportedOperationException: Heap buffer
>         at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.
> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:346)
>         at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.
> ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
>         at
> org.apache.flink.shaded.netty4.io.netty.channel.
> AbstractChannelHandlerContext.invokeChannelRead(
> AbstractChannelHandlerContext.java:339)
>         ... 9 more
> Caused by: java.lang.UnsupportedOperationException: Heap buffer
>         at
> org.apache.flink.runtime.io.network.netty.NettyBufferPool.
> heapBuffer(NettyBufferPool.java:236)
>         at
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.
> SslHandler.decode(SslHandler.java:926)
>         at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.
> ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
>         ... 11 more
>
>
> After reading a note on the release notes.... I tested with taskmanagers
> having one single task slot.
> But I got the same result: the simple job works correctly and the complex
> one fails with the Exception above.
>
> Any thoughts ?
>
> Regards,
> Edward
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>