Re: distinguishing between failures and errors

--- In junit@xxxx, Mike Clark <mike@xxxx> wrote:
> Q: What's the difference between a failure and an error?
> 
> A:
> 
> Assertions are used to check for the possibility of failures,
> therefore failures are anticipated.

good.

> In the following example, the IndexOutOfBoundsException is expected
> and checked with an assertion.  If the expected exception is not
> raised, then a failure is produced.  

simplifying (the ctor that takes an int kind of misdirects):

public void testExceptionExpected() {
  try {
    new ArrayList().get( 23 );
    fail( "Should have thrown IndexOutOfBoundsException" );
  }
  catch ( IndexOutOfBoundsException success ) {
  }
}

> Errors are unanticipated problems resulting in uncaught exceptions
> being propagated from a test method.  In the following example, the
> IndexOutOfBoundsException is not expected. The JUnit framework
> produces an error if the IndexOutOfBoundsException, or any other
> unchecked exception, is raised.  

how about changing the example so that IndexOutOfBoundsException
shouldn't be thrown.  plus, IndexOutOfBoundsException is an unchecked
exception, so don't bother with adding it to the throws clause:

  public void testExceptionUnexpected() {
    List list = new ArrayList();
    list.add( "blah" );
    assertEquals( "0th element", "blah", list.get( 0 ) );
  }

> Both failures and errors will cause the test to fail.  However, it
> is useful to differentiate between failures and errors because the
> debugging process is slightly different.  
> 
> In the first example, the use of fail() will not generate a
> complete stack trace including the method that raised the
> exception.  In this case that's sufficient since we anticipate that
> the exception will be raised.  If it's not raised, then it's a
> problem with the test itself.  
> 
> In the second example,  the JUnit framework catches the exception
> and generates an error with a complete stack trace for the
> exception.  Since we don't expect this exception to be raised, a
> complete stack trace is useful in debugging why it was raised.

golden.

i was struggling with whether checked exceptions advertised in the
"throws" clause of a method being tested represented "anticipated"
faults that merited catching and failing the test if caught.
after reading all the discussion here and at the wiki, i'm inclined
only to catch the exception that should be thrown during the course
of a test (if any), letting the others propagate out as errors.

if that's the majority opinion of the user community, then i'd go so
far as to mention it in the FAQ...that this leads to test methods
that themselves advertise "throws Exception", allowing checked
exceptions which aren't anticipated in a given test to be treated as
errors rather than failures, resulting in cleaner code:

  public void testReflectiveInstantiation() throws Exception {
    MyClass instance = (MyClass) MyClass.class.newInstance();
    assertNotNull( instance );
  }

versus:

  public void testReflectiveInstantiation() {
    try {
      MyClass instance = (MyClass) MyClass.class.newInstance();
      assertNotNull( instance );
    }
    catch ( IllegalAccessException ex ) {
      fail( "threw: " + ex );
    }
    catch ( InvocationTargetException ex ) {
      fail( "ctor threw: " + ex.getTargetException() );
    }
    catch ( ExceptionInInitializerError error ) {
      fail( "threw: " + error );
    }
    catch ( SecurityException ex ) {
      fail( "threw: " + ex );
    }
  }

thanks for the enlightenment!

--p