osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Logging Kafka during exceptions


If so , then you can implement your own deserializer[1] with costume logic  and error handling 





On Thu, Nov 22, 2018 at 8:57 AM Scott Sue <scott.sue@xxxxxxxxxxxxxx> wrote:
Json is sent into Kafka


Regards,
Scott

SCOTT SUE
CHIEF TECHNOLOGY OFFICER

Support Line : +44(0) 2031 371 603
Mobile : +852 9611 3969

9/F, 33 Lockhart Road, Wan Chai, Hong Kong







On 22 Nov 2018, at 14:55, miki haiat <miko5054@xxxxxxxxx> wrote:

Which data format   is sent to kafka ? 
Json Avro Other ?



On Thu, Nov 22, 2018 at 7:36 AM Scott Sue <scott.sue@xxxxxxxxxxxxxx> wrote:
Unexpected data meaning business level data that I didn’t expect to receive. So business level data that doesn’t quite conform

On Thu, 22 Nov 2018 at 13:30, miki haiat <miko5054@xxxxxxxxx> wrote:
 Unexpected data you mean parsing error ?
Which format is sent to Kafka ?



On Thu, 22 Nov 2018, 6:59 Scott Sue <scott.sue@xxxxxxxxxxxxxx wrote:
Hi all,

When I'm running my jobs I am consuming data from Kafka to process in my
job.  Unfortunately my job receives unexpected data from time to time which
I'm trying to find the root cause of the issue.

Ideally, I want to be able to have a way to know when the job has failed due
to an exception, to then log to file the last message that it was consuming
at the time to help track down the offending message consumed.  How is this
possible within Flink?

Thinking about this more, it may not be a consumed message that killed the
job, but maybe a transformation within the job itself and it died in a
downstream Operator.  In this case, is there a way to log to file the
message that an Operator was processing at the time that caused the
exception?


Thanks in advance!



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
--


Regards,
Scott

SCOTT SUE
CHIEF TECHNOLOGY OFFICER

Support Line : +44(0) 2031 371 603
Mobile : +852 9611 3969

9/F, 33 Lockhart Road, Wanchai, Hong Kong

This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.


This message, including any attachments, may include private, privileged and confidential information and is intended only for the personal and confidential use of the intended recipient(s). If the reader of this message is not an intended recipient, you are hereby notified that any review, use, dissemination, distribution, printing or copying of this message or its contents is strictly prohibited and may be unlawful. If you are not an intended recipient or have received this communication in error, please immediately notify the sender by telephone and/or a reply email and permanently delete the original message, including any attachments, without making a copy.