Optimization Safety

I know that for some things, like GCC itself, or like GLibC, it's dangerous 
to screw around with the default optimization settings, but what about for 
random package X? Just how safe are the optimization algorithms? Let's say 
I compile with -O3, and it compiles and installs successfully, and seems to 
give no immediate horrific failures when running, is that good enough for a 
production system? Or can there be some tiny timebombs hidden in there 
because of some strange construct the package writer used?