Subject: Re: POST with huge HTTP body - msg#00096

Previous Message by Date:

Re: Design flaw? - num_processors, accept/close

On 10/15/07, Zed A. Shaw <zedshaw-dd7LMGGEL7NBDgjK7y7TUQ@xxxxxxxxxxxxxxxx> wrote: >Tried Lucas Carlson's Dr. Proxy yet? Other solutions? Evented mongrel? HAProxy (and some other proxies smarter than mod_proxy_balancer) solves this problem by allowing to set the maximum number of requests outstanding to any node in the cluster. Setting it to 1 means that it will only ask a Mongrel instance to serve a request when it's not already doing so. Which makes perfect sense with Rails (single-threaded), especially if you do have something else to serve static content in this setup. Setting num_processors to 1 is only possible when you have a proxy that can restrict itself from sending more than one request per Mongrel. Otherwise, if I remember correctly, you replace occasional delays with HTTP 503s. Not a good trade-off. Setting num_processors low has a positive side effect of restricting how far your Mongrel will grow in memory when put under strain even for a short period. It grows in memory by allocating RAM to new threads (that then pile up on a Rails mutex). With, say, 10 Mongrels and a default num_processors = 1024, allocating memory for 1024 * 10 threads means hundreds of Megabytes of RAM. I usually set num_processors to something a bit bigger than 1 (say, 5), just so that monitoring can hit it at the same time when load balancer does. -- Alexey Verkhovsky CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com] RubyWorks [http://rubyworks.thoughtworks.com]

Next Message by Date:

Re: Design flaw? - num_processors, accept/close

On 16 Oct 2007, at 06:49, Zed A. Shaw wrote: > On Mon, 15 Oct 2007 16:43:34 -0700 > "Brian Williams" <bwillenator-Re5JQEeQqe8AvxtiuMwx3w@xxxxxxxxxxxxxxxx> wrote: > >> We recently ran into exactly this issue. Some rails requests were >> making >> external requests that were taking 5 minutes (networking issues >> out of our >> control). > > Now that's a design flaw. If you're expecting the UI user to wait > for a backend request that takes 5 minutes then you need to > redesign the workflow and interface. Do it like asynchronous email > where the use "sends a request", "awaits a reply", "reads the > reply", and doesn't deal with the backend processing chain of events. Zed, you're being obtuse. Of course that isn't what Brian means. What he's doing is giving a pathological example to illustrate just how badly the mod_proxy_balancer/mongrel/rails combination behaves when things go wrong. Yes, you can mask the problem to some extent by mucking about with your application (and in fact that's what we've done here), but that's missing the point. It is not unreasonable to expect that some actions performed by an application are "fast" and some are "slow". It's further not unreasonable to expect a very large difference between the fastest and the slowest actions (if one action takes 10ms and another takes 1s, that's not unreasonable - but it is a 2 order of magnitude difference). With the obvious setup, fast actions will be delayed behind slow actions. This is a Bad Thing. Furthermore, people are fallible. If I happen to accidentally introduce an action into my system which takes 10s, yes I've screwed up and should fix it. But is it reasonable for the fact that I have a single (possibly very rare) action which takes 10s to mean that all the other fast actions are affected? Even when most of my mongrels are idle? Of course, this isn't really a problem with Mongrel. It's a problem with Ruby (which doesn't know what the word "thread" means) and Rails (which doesn't even manage to successfully make use of the brain-dead version of threading which Ruby does support). -- paul.butcher->msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: https://www.linkedin.com/in/paulbutcher MSN: paul-q9qwgYRQEc9WkMItezQGDw@xxxxxxxxxxxxxxxx AIM: paulrabutcher Skype: paulrabutcher

Previous Message by Thread:

Re: POST with huge HTTP body

On Mon, 15 Oct 2007 07:37:57 -0700 (PDT) Tim Kuijsten <kuijsten-/E1597aS9LQAvxtiuMwx3w@xxxxxxxxxxxxxxxx> wrote: > Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe > it has something to do with your Railsdav plugin? Yes, I believe this is true as well. Railsdav pretty much blows. My experience with it is that none of the windows clients work well (all three) and it did some stupid stuff like used send_file in rails and processed the whole file on input. Ehem, there's this company called Xythos. They make a really good WebDAV server and client. Their WebDAV server is written in Java but it's not insanely architected. Instead it's very well done and using Jakarta Slide you can JRuby yourself a little Ruby lib that looks like Fileutils in about an hour. It's also a very flexible server, and very conforming since most of the people who worked on the RFCs works for Xythos. It's possible with Xythos to swap out nearly everything as well. For example, if you need to use your crappy "rails acts_as_lamely_authenticated" authentication, then you can write your own auth filter for Xythos and it'll go and look in your database. If you need to do something special to the files then you can do a filter for that (it's got a sample that uses Clam to do AV). They also give you all the source to their (ugly) web front end for free so you can see how every operation is done, even the weirdo admin stuff. It's a good product, and if you're dumb enough to work with WebDAV (since obviously you like Microsoft raping you over a barrel), then go check it out. Good luck, I've seen WebDAV kill off nearly every Rails project that's come near it. Not sure what it is, but it's like the black hole with a cache of gold in the center. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/

Next Message by Thread:

Design flaw? - num_processors, accept/close

This is a multi-part message in MIME format.Rails instances themselves are almost always single-threaded, whereas Mongrel, and it's acceptor, are multithreaded. In a situation with long-running Rails pages this presents a problem for mod_proxy_balancer. If num_processors is greater than 1 ( default: 950 ), then Mongrel will gladly accept incoming requests and queue them if its rails instance is currently busy. So even though there are non-busy mongrel instances, a busy one can accept a new request and queue it behind a long-running request. I tried setting num_processors to 1. But it looks like this is less than ideal -- I need to dig into mod_proxy_balancer to be sure. But at first glance, it appears this replaces queuing problem with a proxy error. That's because Mongrel still accepts the incoming request -- only to close the new socket immediately if Rails is busy. Once again, I do need to set up a test and see exactly how mod_proxy_balancer handles this... but... If I understand the problem correctly, then one solution might be moving lines 721 thru 734 into a loop, possibly in its own method, which does sth like this: def myaccept while true return @socket.accept if worker_list.length < num_processors ## check first to see if we can handle the request. Let client worry about connect timeouts. while @num_processors < reap_dead_workers sleep @loop_throttle end end end 720 @acceptor = Thread.new do 721 while true 722 begin * 723 client = @socket.accept * 724 725 if $tcp_cork_opts 726 client.setsockopt(*$tcp_cork_opts) rescue nil 727 end 728 729 worker_list = @workers.list 730 731 if worker_list.length >= @num_processors 732 STDERR.puts "Server overloaded with #{worker_list.length} processors (#@num_processors max). Dropping connection." * 733 client.close rescue Object* 734 reap_dead_workers("max processors") 735 else 736 thread = Thread.new(client) {|c| process_client(c) } 737 thread[:started_on] = Time.now 738 @workers.add(thread) 739 740 sleep @timeout/100 if @timeout > 0 741 end rob.vcf Description: Vcard