Seems like I have addressed some of the issues in a paper that you guys are
discussing here.
The paper addresses the question of what should be the evaluation strategy for
an incoming packet. It proposes a solution in which the evaluation strategy
is determined by collecting statistics of the traffic and the rule set.
With in the system, it decides whether a separate bin should be created for a
rule (or a group of rules ) by calculating the benefit of it as the "avg
number of rules that can be rejected". For example, if a ruleset has 10 rules
with dst port 80 then creating a separate bin for port 80 would result in the
rule getting rejected for all packets which do not have port 80 i.e.,
10 * fraction of packets which do not have port 80.
You can see a more detailed treatment in the paper.
Sushant Sinha, Farnam Jahanian, and Jignesh M. Patel, WIND: Workload-aware
INtrusion Detection, Recent Advances In Intrusion Detection (RAID), September
2006 [ pdf ]
You can download it from here.
http://www-personal.umich.edu/~sushant/
In my opinion, it is hard to say precisely what is the cost of addition of a
particular rule. Many times you won't see any measurable difference. But
rules with maximum content length 1 are costly to evaluate because they match
more often and make two byte shifts difficult. For rules which are general
or "any" in dst port, src port, and other attributes are difficult to
separate and become costly to evaluate and take more run time space.
Hope this helps,
-Sushant
On Thursday 11 January 2007 4:28 pm, Matt Jonkman wrote:
> Good points. But to measure load, you have to consider the traffic that
> the sensor will see, as well as the rules.
>
> If a high load rule never gets hit in the chain because no traffic
> appears that's close to it, then it's not a high load rule. It just
> makes the snort startup a few milliseconds longer. :)
>
> One way could be to use the performance stats stats in recent snorts.
> You could run a ruleset on average traffic in your environment, then add
> the rule in question and go for a similar period on similar traffic and
> measure the difference. That's the only real way I see to get stats that
> mean something to a specific environment, but even that's subjective as
> traffic will be slightly different in each period.
>
> Something I've considered doing before and found to be more work that I
> had resources to do, was to make an average traffic corpus and some kind
> of framework to do the testing. Tools like tomahawk exist, but aren't
> complete for this. There needs to be a very detailed statistical
> analysis behind the testing.
>
> But the key I think is the traffic corpus. Throwing packet generator
> traffic at snort is 100% wasted effort. Snort needs full streams, real
> conversations, and real attacks to be taxed. Companies that throw out
> claims of gigabit speed with a packet generator are wasting your time.
> It's absolutely meaningless.
>
> For an overall testing 'standard', something we could all use to compare
> ourselves and new products to, the corpus in use would have to have a
> wide sampling of all types of traffic. Plus, it'll need valid attacks
> interspersed, plus network anomaly crud (fragments, dupes, mangled
> packets, vlan tags, etc).
>
> THEN... you'd need to test on multiple architectures to find the
> bottlenecks in each technology so that you're not saying snort has Xmb/s
> limit with this ruleset when the bottleneck was actually the PCI bus, or
> the FSB. You'd need to measure all aspects of system performance, and at
> many different traffic levels, and then chart that all.
>
> Or, you can throw the sig into production and see when packets start to
> drop. :)
>
> Matt
>
> Martin Holste wrote:
> > Yes, I've seen that site, but I was unsatisfied with having to rely on
> > their results and with the lack of an API (for lack of a better term).
> > It seems worth it to me to figure out exactly what it takes (perhaps
> > down to count/size of memcpy's, etc?) for a rule op.
> >
> > Applications of this formula would be mainly in scripting to run
> > historical reports on rule matches per system load. Or, more
> > importantly for those of us who are always redlining our boxes,
> > answering questions like "how many pps will I drop with the addition
> > of this rule?" or "what is the rule capacity of this given setup?" I
> > think the latter has been more of an issue lately, especially when you
> > look at the exponential growth of the rule sets.
> >
> > Making truly informed decisions about the entirety of the rule set
> > seems like a worthwhile endeavor.
> >
> > On 1/11/07, Matt Jonkman
> > <jonkman-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx> wrote:
> >> Turbosnort.com is probably what you were thinking about. By the vigilant
> >> minds guys.
> >>
> >> Load has to consider traffic patterns, which this doesn't. Although it
> >> is *some* measure of performance. I use them here and there when unsure.
> >>
> >> Matt
> >>
> >> Gentoo-Wally wrote:
> >> > My math is not so good so... j/k
> >> >
> >> > Isn't there a site somewhere where you can copy a rule into it and it
> >> > would test it for performance?
> >> >
> >> > Wally
> >> >
> >> > On 1/11/07, Martin Holste
> >> > <martin.holste-Re5JQEeQqe8AvxtiuMwx3w@xxxxxxxxxxxxxxxx> wrote:
> >> >> It seems to me that it should be possible to create a somewhat simple
> >> >> (or at least straightforward) formula for how much effort it takes to
> >> >> process a given rule. It could be as simple as a unit of snort
> >> >> processing so that rule comparisons can be made, though not
> >> >> translated directly to actual CPU cycles or other measurable
> >> >> real-world events.
> >> >>
> >> >> I'm not sure if benchmarking or code-analysis (or some combination)
> >> >> would prove the most accurate. Obviously, different
> >> >> compiling/architecture options affect Snort's performance, but it
> >> >> seems to me that a theoretical performance formula is tangible based
> >> >> on the source code. I also think that it would be fair to assume
> >> >> that the ac-bnfa matching algorithm is used, since it is destined to
> >> >> be the default.
> >> >>
> >> >> So, to that end, do you think this mailling list could put together
> >> >> something to start a list of values where an oink = 1 unit of Snort
> >> >> processing load. Even estimated values would be helpful.
> >> >>
> >> >> Matching Oinks
> >> >> content x (per char)
> >> >> pcre x (per char)
> >> >> rawbytes x (per char)
> >> >> uricontent x (per char)
> >> >> isdataat x
> >> >> byte_test x
> >> >> byte_jump x
> >> >> ftpbounce x
> >> >> regex x (per char)
> >> >>
> >> >> Modifiers Oinks
> >> >> nocase x
> >> >> depth x
> >> >> offset x
> >> >> distance x
> >> >> within x
> >> >>
> >> >> Non-payload Oinks
> >> >> fragoffset x
> >> >> ttl x
> >> >> tos x
> >> >> id x
> >> >> ipopts x
> >> >> fragbits x
> >> >> dsize x
> >> >> flags x
> >> >> flow x
> >> >> flowbits x
> >> >> seq x
> >> >> ack x
> >> >> window x
> >> >> itype x
> >> >> icode x
> >> >> icmp_id x
> >> >> icmp_seq x
> >> >> rpc x
> >> >> ip_proto x
> >> >> sameip x
> >> >>
> >> >> We'd then need to figure out multipliers for single/all ports and IP
> >> >> filters as well as other miscellaneous additions to the formula.
> >> >>
> >> >> So, if something like this has been done, please point me to it, but
> >> >> I was unable to Google anything like this, and I figured that with
> >> >> all of the seasoned Snort vets on this list we ought to be able to
> >> >> come up with something.
> >> >>
> >> >> --Martin
> >> >> _______________________________________________
> >> >> Bleeding-sigs mailing list
> >> >> Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
> >>
> >> http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs
> >>
> >> > _______________________________________________
> >> > Bleeding-sigs mailing list
> >> > Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
> >> > http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sig
> >> >s
> >>
> >> --
> >> --------------------------------------------
> >> Matthew Jonkman
> >> Bleeding Edge Threats
> >> 765-429-0398
> >> 765-807-3060 fax
> >> http://www.bleedingthreats.net
> >> --------------------------------------------
> >>
> >> PGP: http://www.bleedingthreats.com/mattjonkman.asc
> >>
> >>
> >> _______________________________________________
> >> Bleeding-sigs mailing list
> >> Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
> >> http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs
> >
> > _______________________________________________
> > Bleeding-sigs mailing list
> > Bleeding-sigs-WwB1pFISwSkm7effSn6vN9HuzzzSOjJt@xxxxxxxxxxxxxxxx
> > http://lists.bleedingthreats.net/cgi-bin/mailman/listinfo/bleeding-sigs
|
