osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Improving persistence efficiency


Hi all,

I'd like to improve the efficiency of Brooklyn persistence - depending on your entities, a lot of files/objects can be written a lot of times. This results in overly high traffic to the persistence directory (be it an object store like S3, or the file system).

Below are three big improvements I plan to make - any objections/comments?
*

1. When an entity changes and is persisted, its 'adjuncts' (e.g. policies and enrichers) are also re-persisted.*

I've created a PR to persist only the entity when it changes: https://github.com/apache/brooklyn-server/pull/964.

The old behaviour was for legacy reasons. A long long time ago, some policies/locations had been written such that they did not notify any persistence-listeners when they changed. They piggybacked on the entity's persistence.

There is a risk that some users still have such legacy entities/policies/locations in use. I've therefore included a feature-flag (`referencedObjectsPersistence`) to re-enable the old behaviour.

---
*2. When an attribute is re-published with the same value, don't re-persist*

Currently, when `entity.sensors().set(attribute, val)` is called with the same value as it already has, it still calls `onAttributeChange` which causes the entity to be re-persisted.

We could guard this, to only trigger re-persistence when the value has actually changed.

Note you can currently prevent an enricher re-publishing the attribute by using the config `enricher.suppressDuplicates`. You can can do the same for a sensor feed using `suppressDuplicates`.

---
*3. When any of the entity's attributes change, it is re-persisted (capped at max once per second, configurable).*

We should mark some attributes as non-persisted. If their value changes, it will not cause the entity to be re-persisted.

This would be especially useful for attributes like cpu-usage, requests-per-second, etc.

In the Java code, there is already support for marking an attribute as non-persisted. It will require some work to wire this up properly (e.g. currently an attribute-change still triggers re-persistence, but that attribute is then excluded).

Aled