OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

pre-initializing global window state


Hi I am looking for some advice on how to solve the following problem

I'd like to keep track of the all time last n events received for a user.
An event on average takes up 500 bytes and here will be ten's of millions of users for which we need to keep this information. The list of events will be stored in cassandra for serving by an api

One way I can think of to implement this , is to use a global window per user with a count evictor. 

The problem I see with this is that the state would forever remain on the worker nodes, in our case, in rocks db.

This means there would be a *lot* of state to include for savepoints and checkpoints. This would make such a job very unwieldy to operate.

Is it possible to evict state from global state after some period of inactivity. and then reinitalize the global state with data loaded from cassandra when a new event arrives ? 

Or is there an obvious better way to tackle this problem that i am missing

Any pointers would be greatly appreciated