Re: Rule Load Formula

Good points. But to measure load, you have to consider the traffic that
the sensor will see, as well as the rules.

If a high load rule never gets hit in the chain because no traffic
appears that's close to it, then it's not a high load rule. It just
makes the snort startup a few milliseconds longer. :)

One way could be to use the performance stats stats in recent snorts.
You could run a ruleset on average traffic in your environment, then add
the rule in question and go for a similar period on similar traffic and
measure the difference. That's the only real way I see to get stats that
mean something to a specific environment, but even that's subjective as
traffic will be slightly different in each period.

Something I've considered doing before and found to be more work that I
had resources to do, was to make an average traffic corpus and some kind
of framework to do the testing. Tools like tomahawk exist, but aren't
complete for this. There needs to be a very detailed statistical
analysis behind the testing.

But the key I think is the traffic corpus. Throwing packet generator
traffic at snort is 100% wasted effort. Snort needs full streams, real
conversations, and real attacks to be taxed. Companies that throw out
claims of gigabit speed with a packet generator are wasting your time.
It's absolutely meaningless.

For an overall testing 'standard', something we could all use to compare
ourselves and new products to, the corpus in use would have to have a
wide sampling of all types of traffic. Plus, it'll need valid attacks
interspersed, plus network anomaly crud (fragments, dupes, mangled
packets, vlan tags, etc).

THEN... you'd need to test on multiple architectures to find the
bottlenecks in each technology so that you're not saying snort has Xmb/s
limit with this ruleset when the bottleneck was actually the PCI bus, or
the FSB. You'd need to measure all aspects of system performance, and at
many different traffic levels, and then chart that all.

Or, you can throw the sig into production and see when packets start to
drop. :)

Matt


Martin Holste wrote:
> Yes, I've seen that site, but I was unsatisfied with having to rely on
> their results and with the lack of an API (for lack of a better term).
> It seems worth it to me to figure out exactly what it takes (perhaps
> down to count/size of memcpy's, etc?) for a rule op.
> 
> Applications of this formula would be mainly in scripting to run
> historical reports on rule matches per system load.  Or, more
> importantly for those of us who are always redlining our boxes,
> answering questions like "how many pps will I drop with the addition
> of this rule?" or "what is the rule capacity of this given setup?"  I
> think the latter has been more of an issue lately, especially when you
> look at the exponential growth of the rule sets.
> 
> Making truly informed decisions about the entirety of the rule set
> seems like a worthwhile endeavor.
> 
> On 1/11/07, Matt Jonkman 
> <jonkman-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx> wrote:
>> Turbosnort.com is probably what you were thinking about. By the vigilant
>> minds guys.
>>
>> Load has to consider traffic patterns, which this doesn't. Although it
>> is *some* measure of performance. I use them here and there when unsure.
>>
>> Matt
>>
>> Gentoo-Wally wrote:
>> > My math is not so good so... j/k
>> >
>> > Isn't there a site somewhere where you can copy a rule into it and it
>> > would test it for performance?
>> >
>> > Wally
>> >
>> > On 1/11/07, Martin Holste 
>> > <martin.holste-Re5JQEeQqe8AvxtiuMwx3w@xxxxxxxxxxxxxxxx> wrote:
>> >> It seems to me that it should be possible to create a somewhat simple
>> >> (or at least straightforward) formula for how much effort it takes to
>> >> process a given rule.  It could be as simple as a unit of snort
>> >> processing so that rule comparisons can be made, though not translated
>> >> directly to actual CPU cycles or other measurable real-world events.
>> >>
>> >> I'm not sure if benchmarking or code-analysis (or some combination)
>> >> would prove the most accurate.  Obviously, different
>> >> compiling/architecture options affect Snort's performance, but it
>> >> seems to me that a theoretical performance formula is tangible based
>> >> on the source code.  I also think that it would be fair to assume that
>> >> the ac-bnfa matching algorithm is used, since it is destined to be the
>> >> default.
>> >>
>> >> So, to that end, do you think this mailling list could put together
>> >> something to start a list of values where an oink = 1 unit of Snort
>> >> processing load.  Even estimated values would be helpful.
>> >>
>> >> Matching    Oinks
>> >> content        x (per char)
>> >> pcre        x (per char)
>> >> rawbytes    x (per char)
>> >> uricontent    x (per char)
>> >> isdataat    x
>> >> byte_test    x
>> >> byte_jump    x
>> >> ftpbounce    x
>> >> regex        x (per char)
>> >>
>> >> Modifiers    Oinks
>> >> nocase        x
>> >> depth        x
>> >> offset        x
>> >> distance    x
>> >> within        x
>> >>
>> >> Non-payload    Oinks
>> >> fragoffset    x
>> >> ttl        x
>> >> tos        x
>> >> id        x
>> >> ipopts        x
>> >> fragbits    x
>> >> dsize        x
>> >> flags        x
>> >> flow        x
>> >> flowbits    x
>> >> seq        x
>> >> ack        x
>> >> window        x
>> >> itype        x
>> >> icode        x
>> >> icmp_id        x
>> >> icmp_seq    x
>> >> rpc        x
>> >> ip_proto    x
>> >> sameip        x
>> >>
>> >> We'd then need to figure out multipliers for single/all ports and IP
>> >> filters as well as other miscellaneous additions to the formula.
>> >>
>> >> So, if something like this has been done, please point me to it, but I
>> >> was unable to Google anything like this, and I figured that with all
>> >> of the seasoned Snort vets on this list we ought to be able to come up
>> >> with something.
>> >>
>> >> --Martin
>> >> _______________________________________________
>> >> Bleeding-sigs mailing list
>> >> Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
>> >>
>> http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs
>> >>
>> > _______________________________________________
>> > Bleeding-sigs mailing list
>> > Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
>> > http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs
>>
>> -- 
>> --------------------------------------------
>> Matthew Jonkman
>> Bleeding Edge Threats
>> 765-429-0398
>> 765-807-3060 fax
>> http://www.bleedingthreats.net
>> --------------------------------------------
>>
>> PGP: http://www.bleedingthreats.com/mattjonkman.asc
>>
>>
>> _______________________________________________
>> Bleeding-sigs mailing list
>> Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
>> http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs
>>
> _______________________________________________
> Bleeding-sigs mailing list
> Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
> http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs

-- 
--------------------------------------------
Matthew Jonkman
Bleeding Edge Threats
765-429-0398
765-807-3060 fax
http://www.bleedingthreats.net
--------------------------------------------

PGP: http://www.bleedingthreats.com/mattjonkman.asc