osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Fwd: Decrease initial source read speed


Hi,


Right now it is a Kafka source, but I had the same issue when reading data from local FS.

It looks like a common problem for many (all?) sources.
When incoming data is very small (paths to large archives) but each entry requires a significant time to process (unpack, parse, etc.) Flink detects the back pressure with delay and too much data becomes part of the first transaction.



--
Best regards,
Andrei Shumanski



On Fri, May 18, 2018 at 11:44 AM, makeyang <riverbuilding@xxxxxxxxxxx> wrote:
Andrei Shumanski:
    which source are u using?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/