[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Updating external service and then processing response

Hi all,

I need to send a request to an external web service and then store the response in a DB table, and I am wondering how people have approached this or similar problems in the past.

The flow is: Kafka source (msgs only every few seconds)  => filter/map operators => result sent to web service (which updates state in that system) => response stored in DB.

Initially I was thinking of just creating a custom sink which basically: Sends request to webservice  => Get response containing external key => Save key into DB
This feels to me like basically smashing together 2 separate sinks into 1, and I am not sure if that is a good design or not.

Another option would be to create a RichMapFunction (possibly async function) which does the web service call. My map function can then just return the response which I can then feed into a standard DB sink. 
However, with this approach it feels strange to update an external system in a map() function, but maybe that's ok? Also, I presume to make my map function idempotent I would need to store some state (I can key the messages and use a ValueState) so I don't do duplicate web service calls if there is a failure?


Thanks in advance.