OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A strange exception while consumption using a multi topic Kafka Connector


Yep, Thanks. We have a set up where topics are constantly being added on a kakfa-to-hdfs pipeline. 

On Sun, May 13, 2018 at 11:58 AM, Ted Yu <yuzhihong@xxxxxxxxx> wrote:
FLINK-9349 was logged.

FYI

On Sat, May 12, 2018 at 7:52 AM, Ted Yu <yuzhihong@xxxxxxxxx> wrote:
I took a look at ./flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java

It seems the List subscribedPartitionStates was being modified when runFetchLoop iterated the List.
This can happen if, e.g., FlinkKafkaConsumer runs the following code concurrently:
                kafkaFetcher.addDiscoveredPartitions(discoveredPartitions);

You can log a JIRA.
If you have unit test reproducing this, that would be great.

FYI

On Fri, May 11, 2018 at 5:15 PM, Vishal Santoshi <vishal.santoshi@xxxxxxxxx> wrote:
java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:134)
.
.
.


This flink 1.4.0

Any ideas folks ?