Re: Load problem with ims 5.2

Thanks to everyone who replied. Our disk service times don't look too 
bad - mostly under 15ms.

service.imap.maxthreads is 250
service.imap.maxsessions is 4000


The pstack output is quite large, so I've put them at 
http://www.pages.drexel.edu/~rjb38/ims-pstack.zip . It seems that lots 
of the threads are waiting for a mutex inside mboxlist_dofind.

We just moved dbtmpdir to tmpfs, but it doesn't really seem to have 
helped (load has climbed back to 200).

Rich

Chris Vervais wrote:
> Don't rely on top for gauging how slow/fast your I/O is going. It's 
> better to run a iostat -xzn 5 and look for service times on the disks 
> the store resides on to be below 20ms. Over 20ms the mailstore starts 
> hating life and chews on it's self.
>
> Can you post the following from configutil:
>
> service.imap.maxsessions
> service.imap.maxthreads
>
> Also, nab some pstacks of the IMAPD processes so we can see what 
> they're spinning on.
>
> Usually a corrupt database / index results in a crash, not a 
> performance issue. Something else to check is, is the ims-ms queue 
> building up? If it is that usually means the store is performing poorly.
>
> Also, what do you have your dbtmpdir set to?
>
> -Chris
>
> On Oct 15, 2007, at 6:35 PM, Rich Bishop wrote:
>
> > Hello,
> >
> > We're running IMS 5.2 (I know it's old!) on Solaris 8 and suddenly 
> > seeing some major load issues. It's one of two mailstores running on 
> > a Sunfire 280R with 6GB. We've been on this hardware for a couple of 
> > years and never noticed any problems until a couple of weeks ago.
> >
> > Suddenly on this one machine we see very high loads (up to 700). 
> > imapd processes are always consuming most of the CPU in top, and the 
> > activity is pretty evenly divided between user and kernel (ie. no or 
> > minimal iowait). Here's a top snapshot - load is low right now but it 
> > shows imapd chewing up all the cpu. At this time we had only 700 imap 
> > connections.
> >
> >
> >
> > last pid:  9266;  load averages: 26.25, 26.14, 
> > 44.50                                                              
> > 21:29:56
> > 86 processes:  78 sleeping, 6 running, 2 on cpu
> > CPU states:  0.0% idle, 58.4% user, 41.6% kernel,  0.0% iowait,  0.0% 
> > swap
> > Memory: 6144M real, 596M swap in use, 4513M swap free
> >
> >  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
> > 8248 mailsrv   33  53   -2  175M   43M run     3:06 51.89% imapd
> > 8219 mailsrv   16  50   -2  184M   41M run     3:37 12.44% imapd
> > 8231 mailsrv   28  50   -2  178M   42M run     3:58  8.79% imapd
> > 8300 mailsrv   23  58    0   28M   21M run     0:42  3.59% 
> > tcp_smtp_server
> > 8230 mailsrv   30  59   -2  232M   41M sleep   3:50  2.84% imapd
> > 23887 root      11  58    0  142M  141M sleep 105:58  0.97% bpbkar
> > 8269 mailsrv   10  50   -2  181M   54M run     0:12  0.86% mshttpd
> > 8272 mailsrv   13  59   -2  218M   61M sleep   0:23  0.59% mshttpd
> >
> > As this came on so suddenly without any configuration changes or 
> > major addition of users, I was wondering if we have some corrupt 
> > database / index somewhere, but I really have no idea where to look. 
> > Any suggestions would greatly appreciated.
> >
> > Thanks,
> >
> > Rich