[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Fwd: Decrease initial source read speed


Right now it is a Kafka source, but I had the same issue when reading data from local FS.

It looks like a common problem for many (all?) sources.
When incoming data is very small (paths to large archives) but each entry requires a significant time to process (unpack, parse, etc.) Flink detects the back pressure with delay and too much data becomes part of the first transaction.

Best regards,
Andrei Shumanski

On Fri, May 18, 2018 at 11:44 AM, makeyang <riverbuilding@xxxxxxxxxxx> wrote:
Andrei Shumanski:
    which source are u using?

Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/