Re: Rails App on Mongrel+Apache Stops Listening

Hey Zed,

First off, congrats on 1.0 and thanks for all your hard work.

On your list of items, I clearly left out some info that I should  
have included:

1. I'm using PostgreSQL and the C bindings.
2. I'm using the ActiveRecord store.
3. DNS... Okay, I'll take a look at this. I think we're okay here but  
I will double check.
4. Not using memcached.
5. Interesting. I'll check this one too.

I'll check out your list of debugging suggestions. I may just skip to  
#4 as that's worked for me in the past.

Cheers,
Hunter

On Jan 24, 2007, at 1:07 AM, Zed A. Shaw wrote:

> On Tue, 23 Jan 2007 21:53:39 -0800
> Hunter Hillegas <lists-HAWAbpnI61OZ1JSuHaJ1sQC/G2K4zDHf@xxxxxxxxxxxxxxxx> 
> wrote:
>
>> Howdy,
>>
>> I'm trying to debug a situation where a Rails app running on Mongrel
>> +Apache stops listening to requests. Restarting Mongrel and Apache
>> brings it back to life.
>>
>> First off, here's some info on versions, etc...:
>
> Here's the shortlist of stuff so far that can cause this:
>
> 1) Using the stock mysql.rb that comes with rails rather than using  
> the compiled mysql gem.
> 2) Use a pstore for sessions, or actually trying to negotiate file  
> locking in general.
> 3) Trying to resolve dns queries against a slow DNS server or  
> broken one.
> 4) Using memcached and putting keys in it with a space or \0 char  
> in them (memcached don't like that and will block the world).
> 5) An interesting new one is having Monit check the mongrel port on  
> sites with long running requests.  What happens is since rails  
> blocks further requests during these requests, and monit is  
> typically told to restart when they're slow, you'll get frequent  
> "slow" requests so monit will frequently restart mongrels for no  
> apparent reason.
>
> Ways to debug this include:
>
> 1) Using lsof -i -P to see what's open.
> 2) Attaching to the stuck process with gdb and using some of the  
> Ruby gdb scripts out there to force an exception to be thrown.   
> Then look in the mongrel.log to see what is logged.
> 3) Running strace or truss on the stuck process to see what system  
> call it's doing.  If it's repeatedly running select() then that  
> means it is most likely trying to write to a socket but getting  
> nothing back.  This happens in the memcached case especially.
> 4) Praying to some ancient god for redemption.
>
> I hear #4 works really well if you're bathed in the blood of 7  
> orphaned virgins, but I'm not that hardcore.
>
> -- 
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users-GrnCvJ7WPxnNLxjTenLetw@xxxxxxxxxxxxxxxx
> http://rubyforge.org/mailman/listinfo/mongrel-users