On Sat, Apr 30, 2005 at 10:18:26PM -0400, David A. Wheeler wrote:
> Hi -- I'd like to know what people here think about
> tools to find/counter security vulnerabilities
> (source scanning tools, fuzzers, etc.).
An interesting topic.
> Can people here post their thoughts on how effective various tools are?
> What works? What doesn't? Are there tools or types of tools
> that seem to be really effective, & when, or are they all hopeless?
> What are the best ways to use them? Is there any way to tell
> that a tool is more effective than using only eyeballs & grep?
I've been thinking about this for a while and here are some comments.
One thing I think is growing on me is that most of the automated
scanning tools have way too many false positives to be as useful as I'd
like. When you end up having to look at each one by hand it takes
almost as long as it would if you were to use your eyes in the first
place!
As Imran said earlier there are a lot of cases where some simple
heuristics which are easy to describe can make a lot of difference,
but they may be hard to code, or may just be wrong!
Right now the biggest success I've been having has come from using
a set of protocol fuzzers. Things that will generate "random"
output which can be fed to tools. The big problem is that these
are very much domain specific.
(I'm pretty sure that most of the image bugs discovered recently
have been found by creating images with strange header values via
an automated tool.)
As an example I wrote a script which would generate "random" HTML
with malformed tags, random attributes, and bizarre text. After
using this for only a few hours I discovered this:
https://bugzilla.mozilla.org/show_bug.cgi?id=292279
(I got another one, but it had been independently discovered which
was a bit of a disappointment :
https://bugzilla.mozilla.org/show_bug.cgi?id=141818 )
There are a few existing tools for different classes of problems,
but the real challenge is to write new ones for different file formats,
or protocols.
One tool I've been experimenting with is an indexer for source code,
which is nicely searchable:
http://gonzui.sourceforge.net/.
It's not an auditing tool per se, but the searching, indexing,
and cross-referencing abilities are very useful when looking at
a large codebase.
> I _am_ the author of Flawfinder, but I'm not looking for
> comments specifically on that. Instead, I'm trying to learn more
> generally what works & what doesn't. You don't need to
> sugar-coat things; if something just isn't helpful, say so.
I like flawfinder, and it does do a good job of finding things.
The problem with it is the same as the problem with other similar
tools, even at very high levels of severity it can give way too
much output to be immediately useful.
> I'm aware of the info about security auditing tools at:
> http://www.debian.org/security/audit/tools
> but that page just describes how to USE a few tools.
> I want to know what tools/approaches seem to be EFFECTIVE.
I think the automated fuzz testing / "protocol abusers" have
been the single most effective tool I've used. However they are
very specific and can't often be applied to many programs.
When they work they work well, but too often they are unavailable
or inappropriate.
Steve
--
|
