[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Call batch job in streaming context?


I’m not sure if I understand your problem and your context, but spawning a batch job every 45 seconds doesn’t sound as a that bad idea (as long as the job is short).

Another idea would be to incorporate this batch job inside your streaming job, for example by reading from Cassandra using an AsyncIO operator:

Quick google search revealed for example this:



> On 23 Nov 2018, at 10:33, eric hoffmann <sigfrid.hoffmann@xxxxxxxxx> wrote:
> Hi
> Is it possible to call batch job on a streaming context?
> what i want to do is:
> for a given input event, fetch cassandra elements based on event data, apply transformation on them and apply a ranking when all elements fetched by cassandra are processed.
> If i do this in batch mode i would have to submit a job on each events and i can have an event every 45 seconds.
> Is there any alternative? can i start a batch job that will receive some external request, process it and wait for another request?
> thx
> Eric