Re: Rule Load Formula

Martin Holste wrote:

> I understand your points, but what I'm arguing is that if a given rule
> can have a theoretical processing cost, then each Snort admin has at
> least a possibility of evaluating it without estimation.  If you know
> that your sensor sees lots of HTTP traffic, then you can take that
> into account when you see that the dport is 80.  But given the large
> number of HTTP sigs, it would be very helpful to know that sig A with
> its 3 pcre's and 2 content matches takes 37 oinks and sig B with one
> content followed by one pcre takes 12 oinks.
The problem is, there's no way to give a finite, concrete number of 
"oinks" used by a given rule, because of the fact that rules will fail 
out at different points during processing. For example, if you have the 
rule:

alert $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"Performance Test 
Rule"; flow:established,to_client; flowbits:isset,evil.flowbit; 
content:"foo"; nocase; content:"bar"; nocase; distance:0; 
pcre:"/^foo\s*[\d\x2E]+bar/smi"; sid:99999; rev:1; )

you could fail out at 10 separate points during processing (i.e. IP 
range, port, flow, etc.). So while completely processing the rule might 
take, say, 20 "oinks", the rule could use 15 "oinks" if it doesn't make 
it to the PCRE, 10 if it fails to find the first content, etc. Depending 
on how statistics were done, you might never see the impact of a rule 
that processes 3/4 of the way on a regular basis but rarely, if ever, 
gets all the way to the point of alerting.

> For a more accurate idea of what that will do to overall system load,
> multiply the oinks by the total packets your netflow/IPAudit stats
> show destined for port 80 at given points in time.  (You are logging
> your flows to SQL, right?).  

I see why you're redlining your boxes. ;-)

> Then subtract those packets that your BPF
> drops before it hits the detection engine (or only multiply by the
> flows seen after the Snort sensor if you aren't running in-line).
> That analysis will give you a chronological report of when a given
> rule was a strain on a system, and when it wasn't.  Then you can start
> down the road of doing things like pulling rules/adding rules
> on-demand based on overall system performance, but you can do it more
> intelligently than simply removing a given ruleset with cron at a
> certain time, i.e. it can be extremely granular.
>
> A truly complete solution someday would allow for automatic
> load-shedding with a sig directive like "shed" to tell Snort that if
> it needs to drop 200 oinks from port 80 sigs, it can shed this rule
> (hey, Snort's already keeping track of how many flows are on what
> ports, right).  Right now it would be tough to implement that since
> Snort would not have the ability to know how much it would help to
> shed a given rule.  But if it were known how much load could be saved
> by dropping a rule, the internal algorithms would be fairly simple.
>
> This may seem like overkill now, but think how many more rules we will
> need to have in a year's time, and think of how few rules we will be
> able to safely drop from our active rule sets by then.  I think having
> theoretical rule performance measurements marks can really help you
> get the most bang for your buck from your hardware, and would really
> open the door to a lot of improvements to the engine and its
> management as a whole.

The better solution is to identify rules that you can shed up-front 
based upon what it is that you're protecting. If you're a MySQL-only 
shop, you can drop all of oracle.rules right off the bat. Got an 
automated patching policy in place on your Windows desktops (one that 
you have a high level of confidence in)? Drop rules that detect 
client-side exploits against vulnerabilities you've patched (clearly on 
helpful in some situations, but probably very helpful in those places 
where it's applicable). Knowing what's on your network is one of the 
most relevant pieces in determining how to shed rules gracefully, and 
the ability to determine what's there is a lot more acheivable than 
actually quantifying the performance impact of Snort rules on a reliable 
basis.

Alex Kirk