Subject: Re: Determining ideal number of Mongrels for an app? - msg#00257

List: lang.ruby.mongrel.general

Date: Prev Next Index Thread: Prev Next Index

On Tue, 28 Nov 2006 08:27:38 -0800 (PST)
Joe Ruby <joeat303-/E1597aS9LQAvxtiuMwx3w@xxxxxxxxxxxxxxxx> wrote:

> What's a rule of thumb for guesstimating how many
> Mongrels to use in a cluster for an app? I have an app
> that gets about 5000 unique visitors per day.

Pathetic. 5000/day? Sad. Oh wait, is that 5000 visitors doing 100k
request/second during the whole day, or 5000 visitors doing .005 request/second
at noon?

(I'm jokingly saying you should find out what their usage pattern is and
translate that to req/sec processed and kb/second transfered first.)

> I figured I'd give it plenty of Mongrels -- twenty to be
> specific. After running out of memory and hitting the
> swap periodically, I scaled it back to five and it
> still seems to serve up visitors fine. So, is there
> some super-secret math fomula I can use to determine
> the ideal number of Mongrels I should start in a
> cluster for an app?

Many folks pointed you at the semi-official tuning documentation. 20 is
totally over kill and will hurt you more than you know. I find it's about
2-4/CPU depending on the amount of ram you have and the speed of your CPU. BUT
YOU SHOULD TUNE EVERY APP EVERY TIME NO MATTER WHAT YOU CHANGED. Don't believe
anything as gospel, always test it yourself.

Now, lately I've been wanting to add a couple of qualifiers on the scaling
issue related to your resource usage. In some situations people are using up
tons of ram or tons of files. If you have 2-4 mongrels you quickly outstrip
the ram and files. For these folks I tell them to do a different test that
tries to do a lot of concurrent but slow connections to eat up as much ram and
file handles as they can. Then, they expand the mongrels out to handle the
number of connections they need. Many times, people in these situations have
to buy more machines.

The main culprit of this is RMagick. If you have RMagick you need to simulate
a ton of people concurrently slamming you with files to make sure you have the
resources.

This is also a reasonably good description of file limit defaults and HTTP
performance:

http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/node7.html

Play with these settings as well for each, maybe even just increasing the
defaults.

--
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Was this page helpful?

Thread at a glance:

Previous Message by Date: click to view message preview

Re: Uploading Large (100mb+) files

On Tue, 28 Nov 2006 14:17:37 -0600 cremes.devlist-ee4meeAH724@xxxxxxxxxxxxxxxx wrote: > > On Nov 28, 2006, at 1:26 PM, Rogelio Samour wrote: > > > I would like to combine what you guys are suggesting. Namely, have > > a merb app (mongrel) dealio... be the sole file uploader listener. > > I understand the ReWrite rules and ProxyPass magic. However, this > > poses a new question: how would I go about downloading these > > uploaded files? I'm assuming I would want this mongrel node to also > > be a file server, correct? > > > > My only experience with setting up mongrel is through > > mongrel_rails. cremes: are you saying this upload-only mongrel is > > it's own setup? if so, is there docs on setting up something like > > this? > > > > Well, there aren't any specific docs explaining how to do this > configuration. There are docs for setting up a standalone Mongrel. > Then there are other docs which show how to create a mongrel cluster > and proxy it with Apache. Just combine the two techniques and you're > done. It really is as simple as it sounds. > > If you have specific problems that you need the group's help to > overcome, post here and we'll try. Yep, it's not hard, the main thing is to read up on the Configurator and to create your own startup script. It's maybe 20-60 lines of code depending on how complex you want to get. Take a look at the mongrel_rails script and the Configurator documentation in the RDoc: http://mongrel.rubyforge.org/rdoc/ I've also already done this for a company and they've given me permission to write it up and release it for other people to use. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Next Message by Date: click to view message preview

Re: Uploading Large (100mb+) files

On Tue, 28 Nov 2006 14:20:00 -0600 (CST) Philip Hallstrom <mongrel-SUcgGwS4C16SUMMaM/qcSw@xxxxxxxxxxxxxxxx> wrote: > > If you use something like mongrel_upload_progress, the single-threadedness > > of Rails is no problem because Mongrel will intercept the upload and not > > hand it off to Rails until it's complete. That's what I'm doing at the > > moment, and my app runs just fine on one Mongrel. > > I haven't used this plugin, but just gave it a quick read through... one > thing I didn't see mentioned... can I skip the upload progress stuff and > just get the benefits of it not blocking mongrel/rails? Yep, actually that's an incredibly trivial thing to do. You'd probably be better off just starting with a small handler that moved the files where you wanted after the upload, and then expand it to your needs. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Previous Message by Thread: click to view message preview

Re: Determining ideal number of Mongrels for an app?

On Tue, 28 Nov 2006 15:23:14 -0600 (CST) Philip Hallstrom <mongrel-SUcgGwS4C16SUMMaM/qcSw@xxxxxxxxxxxxxxxx> wrote: > > On Tue, Nov 28, 2006 at 11:03:06AM -0600, Philip Hallstrom wrote: > >> Just for reference, we get about 500,000 pages a day and run 4 mongrel > >> processes... we've got about 20 servers[1], but 4 gives us the best > >> performance. > > > > I'm curious - do you have 4 * 20 Mongrels or just 4? How many > > requests/sec does Rails report in your logs? > > Just did a quick grep on one of the servers log files and for about 6000 > requests we are averaging 38 reqs/sec with a standard deviation of 113. Standard Deviation! YES! -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Next Message by Thread: click to view message preview

Re: Determining ideal number of Mongrels for an app?

On Nov 28, 2006, at 15:49 , Zed A. Shaw wrote: > Many folks pointed you at the semi-official tuning documentation. > 20 is totally over kill and will hurt you more than you know. I > find it's about 2-4/CPU depending on the amount of ram you have and > the speed of your CPU. BUT YOU SHOULD TUNE EVERY APP EVERY TIME NO > MATTER WHAT YOU CHANGED. Don't believe anything as gospel, always > test it yourself. Zed, Do dual-core chips count as two CPU's for this calculation? Also what is the recommended amount of ram per mongrel process or per CPU based on your experience? Thanks, ~Wayne

Loading Comments...