Re: DDOS on the appl level, timeouts and blacklisting

Hi there,

On Mon, Sep 18, 2006 at 03:18:51PM +0100, Ivan Ristic wrote:
> >A private exchange with Ryan and Ivan evolved. This directed me
> >away from mod_security.
> 
> No, no, no, we have to find a way to get you back now :)

Oh, you are on good tracks.
 
> >A single apache thread or process lacks the "whole picture" and
> >it can not do a proper assessment of the probability of an attack.
> 
> With ModSecurity 2.0 you actually get a persistent data store that is
> shared across processes and threads. But even that will not solve a
> problem if you have a cluster of web servers. (At least not until I
> release a cluster-ready version of ModSecurity, that is.)

This ability makes things a lot easier with modsecurity. I have
played with the ability and it is really cool. Too many ideas spring to
mind...

Still the basic problem with the attack i have described is for
me, that we never get into a position where modsecurity is able
to check something, as the attacker never reaches the post_read_request 
hook with his request. But you addressing this below.

> I believe it was Ryan who mentioned httpd_guardian in response to your
> recent email. On a single box httpd_guardian will connect to Apache
> via the piped logging mechanism. If you need to support a cluster it
> will work with the Spread toolkit to receive the required information
> in real time. This utility is at this point capable of detecting brute
> force attacks, but it can (and should) be extended to handle DoS
> attacks you describe.

I was planning to start with httpd_guardian as a base and extend
it in the direction described. Spread makes it a valuable tool.
It is good you included it from the start.

> Slightly off-topic but potentially useful to some: mod_backdoor
> (http://people.apache.org/~trawick/) keeps a separate process ready
> for access for cases like that.

good hint. thanks. One more apache module i did not know. :)

> >The problem is, that there is nothing like a close connection hook.
> >However, in http://hypermail.linklord.com/new-httpd/2004/Feb/3597.html
> >i have found a patch from Joe Orton, that brings a finish_connection hook.
> 
> But such hook might not be necessary. Each connection comes with its
> own memory pool, which is destroyed after the connection is closed.
> You can simply register a call-back with this memory pool and it will
> be invoked just before the pool is destroyed.

Thanks for pointing this out. I am not really a coder. :)

> I have some concerns (some of which you identified already):
> 
> 1. A large number of messages may be needed per connection. This could
> be a performance bottleneck for large installations.

Yes. I see this too.

> 2. I don't think the Apache error log is the right place to write
> these messages. The sheer volume of messages will hide other
> potentially interesting log entries.

In my testcode i have individual switches for the different
messages so they need not be running all the time in full mode. 
error_log is a starting point as i do not want to have different 
logfile for every possible threat... Still thinking about that.

> 3. It is not possible for such external checker to take care of large
> request bodies and file uploads.

? What do you mean by that? I do not want it to check the body.
I just want it to keep an eye on timeouts. It's modsecurity's
job to check that. And i would use modsecurity to authorise
the client to send a request body at all to this URL.

> How about this:
> 
> 1. We keep the log-oriented approach.
> 
> 2. In addition to working with Apache hooks as you did, we write an
> Apache input filter that looks at raw data. We look at the number of
> bytes received on every invocation and the time it took for the data
> to arrive. This could allow us to enforce:
> 
> 2.1 A time limit for the request headers to arrive.
> 
> 2.2 Minimum communication speed.
> 
> A log entry is made only when a problem is detected.
> 
> 3. Because log entries now appear only on attacks we can use the error
> log for this.

This reduces the logfile a lot. So this is clearly a positive
aspect. The analysis is within apache now, which is much more
efficient, but we can not tweak the rules so easily anymore in
case of an attack - it demands a graceful reload of the apache
now. Maybe even a not so graceful one when we want to free
the threads under attack.
This is modsecurity's problem anyways, but i think it is worth
noting, even if it is no big deal in real life.

Would you add such an input filter to modsecurity?

> One drawback of the filter approach, though, is that it is passive -
> it can only do its thing when the next batch of data arrives. This
> shouldn't be a big problem provided the Apache timeout value is low
> enough.

Hmm. We are thinking about setting up defcon scenarios where we would
set the timeout to 5 seconds or so in case of an attack.

What is if the attacker sends a TCP packet every second. Will this
trigger the input filter? As far as i have tested, the timeout is
never reached.

Thanks for your thoughts. We are advancing.

Christian


-- 
Given the choice between two theories, take the one which is funnier.
--- Blore's Razor, Author unknown

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV