osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Algorithmic explorations of bitmaps vs. sentinel values


hi folks,

I explored a bit the performance implications of using validity
bitmaps (like the Arrow columnar format) vs. sentinel values (like
NaN, INT32_MIN) for nulls:

http://wesmckinney.com/blog/bitmaps-vs-sentinel-values/

The vectorization results may be of interest to those implementing
analytic functions targeting the Arrow memory format. There's probably
some other optimizations that can be employed, too.

Caveat: it's entirely possible I made some mistakes in my code. I
checked the various implementations for correctness only, and did not
dig too deeply beyond that.

- Wes