Subject: Testing the new phishing detection module: Test 1 - Capability - msg#00031

The new clamav phishing module implements a number of different methods
for detecting phishing emails.

The algorithms/heuristics used are documented (although there do seem to
be more things implemented than I expected!) on the wiki.

The first test that I have conducted is to assess the coverage of these
methods over a reasonably large sample of distinct phishing emails.

The test data set consists of just under 1000 distinct phishing emails.
These are distinct as defined by the official clam AV signatures (i.e. a
single example of as many different "viruses" as possible).

The dataset can be accessed at http://phishery.internetdefence.net/malware/

By coverage, I mean that I am not testing _accuracy_ (with respect to
false positives/false negatives). The test is run against _known_
phishing emails to see how many can be detected by the new methods that
the phishing module implements. In particular, the test is run _without_
a set of signatures (other than a dummy signature to enable the various
heuristics).

For example, a phishing email that does not contain URLs (e.g. just a
VoIP phone number to call) will not be detected by the new module.

The full results can be found at

http://phishery.internetdefence.net/clamav-test1.html

The bottom line is:
- The module performed well with no crashes or other failures in
operation. Which is very good!
- Performance caused no concerns (this wasn't a test of performance, but
there appear to be no issues of significance with the new module and the
extra parsing work that it does)
- The new module detected 45% of the sample as phishing email (this was
actually I bit better than I expected - I had guessed around 33%), so
well done!

- More importantly, it does detect phishing emails which the existing
clam signatures do not detect. This is also very good.

For an additional comparison, I've also tested it against the most
recent version of the Sane Security phishing signatures.

Note 1: We've been feeding undetected phishing emails to clam, where
Sven has been processing them, so the current clam signatures are pretty
good at detecting our data set. In addition, Sane Security takes a
direct feed from the phishery.internetdefence.net data, so the sane sigs
are pretty up to date with our data set also (more so than the clam
official sigs).

Note 2: The source is clamav-devel from 17 Sept 2006, 08:30 GMT.

It was built with the following options:

./configure --prefix=/var/pss/clamav --disable-clamav
--with-dbdir=/var/pss/lib/clamav --enable-id-check --with-user=pss
--with-group=pss --enable-experimental

Clam was run with the options:

/var/pss/clamav/bin/clamscan --no-summary
--database=/var/pss/phishery/clamav --phish-scan-alldomains <phishemail>

The 0.88.4 engine was used for the tests with the official and Sane
Security sigs.

The directory /var/pss/phishery/clamav contains the single database

phishery.pdb

which contains the single signature

H paypal.com

this is simply in the file to enable the heurstic tests.

Please let me know if
- This test is useful
- If you want me to change the test in anyway.
- If I've missed anything obvious in the set up

My next tests will require a real .pdb files - but this will take a bit
of work...

Regards,

Ian Castle.
ECSC Ltd.


_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html