osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

valid NaNs versus invalid NaNs?


'Morning,



Regarding https://arrow.apache.org/docs/memory_layout.html, how should is_valid be interpreted for primitive types that have their own notions of is_valid?



Concretely, how should folks interpret a "valid NaN" (is_valid 1 with float NaN) versus an "invalid NaN" (is valid 0 with float NaN)?  In RFC-ese, MUST individual NaNs be valid?  Or, MUST floats all be valid by omitting the validity bitset?



I ask because otherwise I can see a bunch of different systems interpreting this detail in many different ways.  That'd be an interop nightmare.  Especially since understanding why NaNs sneak into large datasets is already quite a hassle.



Anyhow, it seems worth addressing this gap at the written specification level.



(Apologies if this has been discussed previously-- I've found no searchable mailing list archives under http://mail-archives.apache.org/mod_mbox/arrow-dev/ or https://cwiki.apache.org/confluence/display/ARROW.)



Thanks,

Rhys