RE: Re: distinguishing between failures and errors

> > From: J. B. Rainsberger [mailto:jbr@xxxxxxxxxxxxxxxxxxx]
> >...
> > > Isn't documentation the thing that should be used to tell the
> > > writer of code that calls yours what your code is supposed to
> > > do, or what the caller is supposed to do (or not do)?  (At
> > > least for code meant to be re-used, such as that implementing
> > > an API?)
> > 
> > Yes, it is. Please explain to me why unit tests are not 
> > documentation. :)
> 
> Because tests aren't specifications. 
> 
> (How can a test show that the caller is responsible for not 
> passing in a null pointer?

I do this by *not* testing that I can handle the null pointer. The absence of 
a test indicates "undefined behavior" and you shouldn't rely on undefined 
behavior. If the tests don't define behavior well enough to make the code 
usable, then likely important tests are missing.

<snip />
> (How can a test show that a caller is responsible for not
> depending on certain current behavior of an implementation
> that is not part of the interface contract?)

Same answer. If you want to rely on my implementation details, you take a 
risk. By not providing a test on that behavior, I implicitly state that that 
behavior may change at any time.

> Because tests are code.  
> 
> (They have syntactic clutter that is necessary for use as tests 
> but which is just clutter relative to the information useful to a 
> client programmer.)

I'm not sure I agree with this. If tests are my primary source of information 
on a class' behavior, then any syntactic clutter is either

    * part of the language, or
    * a sign of a poor test

In the first case, I'm probably used to the syntactic clutter and can read it 
as easily as I read English or French. In the second case, the test needs 
improvement, which will benefit everyone all round. That's why it's good to 
read your own tests from time to time.

> Because tests, being code, can't relate code to external concepts.  
> 
> (A specification can say that a function f returns the mathematical
> sum of two given integers subject to the limitations of 32-bit
> signed binary arithmetic.  How can a test say something like that?)

I would think that the name of the method says it, so the test doesn't need 
to. The test *can* say it, of course. If neither does, then there's trouble.

Tell me why three tests adding two signed integers together, at least one 
showing overflow, wouldn't communicate the intent 'add'? What if the tests 
were named testAdd_WithZero, testAdd and testAdd_Overflow?

> Tests are great, but don't forget the documentation.
> 
> It seems that some people who write tests waste one advantage of it:  
> 
> I agree that testing, especially writing tests even before you 
> implement, is good because it forces you to think about what your 
> code is supposed to do and how it's supposed to be used.  
> 
> Well, if you're already thinking about the contract, then that's a
> good time to document the contract.

Please explain why a list of well-named tests with clear implementations 
doesn't provide ample description of the contract. Assume that the person 
writing the tests is thorough and only leaves out tests when he decides that 
something is *not* part of the contract.

> Yes, I know this has to be modulated by what kind of code we're
> talking about (e.g., intra-program calls versus trans-API calls) 
> and by how stable the interface contract is.  I just wouldn't
> want to have to read test cases instead of Javadoc to figure 
> out how, say, Sun's Servlet API worked and was supposed to be 
> used. 

OK, so now we see the point: you'd rather read Javadoc than test cases to 
understand enough about something to be able to use it. I understand that 
well-written, complete and thorough Javadoc is very useful and even better 
than test cases written in code; however:

    Most Javadoc is crap

because:

    I can deliver a working system without Javadoc;

however:

    My tests must run all the time, so my tests must be up to date.

Code or comments -- which do you trust more, in general? I trust code, since 
working code can't lie.

I, for one, would be very interested to see Sun's servlet API described in a 
test suite.

--
J. B. Rainsberger,
President, Diaspar Software Services
Let's write software that people understand.
http://www.diasparsoftware.com/
telephone: +1 416 791-8603