Quoting inconsistency

There seems to be an inconsistency in the interaction between "!", "\" 
and double quotes. On my SuSE 9.1 system 
(BASH_VERSION='2.05b.0(1)-release'), if I try the command

   echo "hi!"

I get the error

   bash: !": event not found

The reason seems simple enough: it's interpreting the "!" character as a 
history prefix and failing. But if I escape the shriek with a backslash:

   echo "hi\!"

I get the output

   hi\!

That is, instead of just outputting the shriek as a literal character, 
it outputs the backslash as well! Compare

   echo 'hi!'

which outputs

   hi!

and

   echo 'hi\!'

which outputs

   hi\!

In other words, in single quotes,  the "!" is not special, and "\" is 
also (mostly) not special, as you would expect. But within double 
quotes, "!" is special, but prefixing it with a backslash makes it 
literal but makes the backslash literal as well! Whereas you would only 
expect this behaviour with characters that are not special in the first 
place. Compare

   echo "$"

which outputs

   $

versus

   echo "\$"

which also outputs

   $

(because "$" is special, even inside double quotes), and

   echo "'"

which outputs

   '

versus

   echo "\'"

which outputs

   \'

(which makes sense because single quotes are not special inside double 
quotes).