osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Time series data model and tombstones


Thanks for the update. Good to know that TWCS give you more stability

On Wed, Feb 8, 2017 at 6:20 PM, John Sanda <john.sanda@xxxxxxxxx> wrote:
I wanted to provide a quick update. I was able to patch one of the environments that is hitting the tombstone problem. It has been running TWCS for five days now, and things are stable so far. I also had a patch to the application code to implement date partitioning ready to go, but I wanted to see how things went with only making the compaction changes.

On Sun, Jan 29, 2017 at 4:05 PM, DuyHai Doan <doanduyhai@xxxxxxxxx> wrote:
In theory, you're right and Cassandra should possibly skip reading cells having time < 50. But it's all theory, in practice Cassandra read chunks of xxx kilobytes worth of data (don't remember the exact value of xxx, maybe 64k or far less) so you may end up reading tombstones.

On Sun, Jan 29, 2017 at 9:24 PM, John Sanda <john.sanda@xxxxxxxxx> wrote:
Thanks for the clarification. Let's say I have a partition in an SSTable where the values of time range from 100 to 10 and everything < 50 is expired. If I do a query with time < 100 and time >= 50, are there scenarios in which Cassandra will have to read cells where time < 50? In particular I am wondering if compression might have any affect.

On Sun, Jan 29, 2017 at 3:01 PM DuyHai Doan <doanduyhai@xxxxxxxxx> wrote:
"Should the data be sorted by my time column regardless of the compaction strategy" --> It does

What I mean is that an old "chunk" of expired data in SSTABLE-12 may be compacted together with a new chunk of SSTABLE-2 containing fresh data so in the new resulting SSTable will contain tombstones AND fresh data inside the same partition, but of course sorted by clustering column "time".

On Sun, Jan 29, 2017 at 8:55 PM, John Sanda <john.sanda@xxxxxxxxx> wrote:
Since STCS does not sort data based on timestamp, your wide partition may span over multiple SSTables and inside each SSTable, old data (+ tombstones) may sit on the same partition as newer data.

Should the data be sorted by my time column regardless of the compaction strategy? I didn't think that the column timestamp came into play with respect to sorting. I have been able to review some SSTables with sstablemetadata and I can see that old/expired data is definitely living with live data.


On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan <doanduyhai@xxxxxxxxx> wrote:
Ok so give it a try with TWCS. Since STCS does not sort data based on timestamp, your wide partition may span over multiple SSTables and inside each SSTable, old data (+ tombstones) may sit on the same partition as newer data.

When reading by slice, even if you request for fresh data, Cassandra has to scan over a lot tombstones to fetch the correct range of data thus your issue

On Sun, Jan 29, 2017 at 8:19 PM, John Sanda <john.sanda@xxxxxxxxx> wrote:
It was with STCS. It was on a 2.x version before TWCS was available. 

On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan <doanduyhai@xxxxxxxxx> wrote:
Did you get this Overwhelming tombstonne behavior with STCS or with TWCS ?

If you're using DTCS, beware of its weird behavior and tricky configuration.

On Sun, Jan 29, 2017 at 3:52 PM, John Sanda <john.sanda@xxxxxxxxx> wrote:
Your partitioning key is text. If you have multiple entries per id you are likely hitting older cells that have expired. Descending only affects how the data is stored on disk, if you have to read the whole partition to find whichever time you are querying for you could potentially hit tombstones in other SSTables that contain the same "id". As mentioned previously, you need to add a time bucket to your partitioning key and definitely use DTCS/TWCS.

As I mentioned previously, the UI only queries recent data, e.g., the past hour, past two hours, past day, past week. The UI does not query for anything older than the TTL which is 7 days. My understanding and expectation was that Cassandra would only scan live cells. The UI is a separate application that I do not maintain, so I am not 100% certain about the queries. I have been told that it does not query for anything older than 7 days. 

On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:

Your partitioning key is text. If you have multiple entries per id you are likely hitting older cells that have expired. Descending only affects how the data is stored on disk, if you have to read the whole partition to find whichever time you are querying for you could potentially hit tombstones in other SSTables that contain the same "id". As mentioned previously, you need to add a time bucket to your partitioning key and definitely use DTCS/TWCS.





--

- John















--

- John











--

- John