[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Timestamp on hints file and system.hints table data


There are no errors in system.log. Just messages about hints sent and played. We will try stop and delete solution and let you know if that fixed the problem. But my curiosity is more about timestamp on hints files, why do we have files with back date; even though I deleted files manually.

On Monday, June 18, 2018, 9:12:16 PM EDT, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:

Send through some examples (and any errors)? Sounds like the file might be corrupt. Not that there's much you can do about that. You can try stopping C*, deleting the file, then starting C* again. You'll have to repair, assuming you haven't repaired already since that hint file was created.

On 18 June 2018 at 13:56, learner dba <cassandradba@xxxxxxxxx.invalid> wrote:
Yes Kurt, system log is flooded with hints sent and replayed messages. 

On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:

Not sure what to make of that. Are there any log messages regarding the file and replaying hints? Sounds like maybe it's corrupt (although not sure why it keeps getting rewritten).

On 14 June 2018 at 13:19, Nitan Kainth <nitankainth@xxxxxxxxx> wrote:

Hint file matches UUID matches with another node in the cluster:

-rw-r--r--. 1 root root  6848246 May 13 23:37 1b694180-210a-4b75-8f2a- 748f4a5b6a3d-1526254645089-1. hints

/opt/cassandra/bin/nodetool status |grep 1b694180

UN  x.x.x.   23.77 GiB  256          ?       1b694180-210a-4b75-8f2a- 748f4a5b6a3d  RAC1

On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:
Does the UUID on the filename correspond with a UUID in nodetool status?

Sounds to me like it could be something weird with an old node that no longer exists, although hints for old nodes are meant to be cleaned up.

On 14 June 2018 at 01:54, Nitan Kainth <nitankainth@xxxxxxxxx> wrote:

No node is down for months. And yes, I am surprised to look at Unix timestamp on files.

On Jun 13, 2018, at 6:41 PM, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:

system.hints is not used in Cassandra 3. Can't explain the files though, are you referring to the files timestamp or the Unix timestamp in the file name? Is there a node that's been down for several months?

On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <nitankainth@xxxxxxxxx> wrote:

I observed a strange behavior about stored hints. 

Time stamp of hints file shows several months old. I deleted them and saw new hints files created with same old date. Why is that?

Also, I see hints files on disk but if I query system.hints table, it shows 0 rows. Why system.hints is not populated?

Version 3.11-1