[LONG] vmstat: What I/O is blocked and how to fix it?

[Originally sent to freebsd-questions, then I remembered this list (to 
which I am not subscribed, so please CC me on any replies.]

m

--- Begin Message ---

Subject:	[LONG] vmstat: What I/O is blocked and how to fix it?

On two occasions recently, vmstat has showed me that a
number of processes are blocked due to I/O. At the same
time, the number of disk transactions per second reported is
a small fraction of the disk's capability.

I am thinking some kind of VM config will help, but based on what
I have read, it is almost always best to let the default kernel
behavior do its thing.

Setup:

Dual PIII 1.0 GHz (256MB L1 and L2 cache)
3GB RAM
custom kernel (SMP + PF)
5.4-RELEASE
one intel fxp0 network interface used (second unused)


Case 1: (da0 36GB SCSI)

The first case is simple: running spamd-setup to load
the huge Composite Blacklist into spamd. The only other
things running are sshd and spamd itself.

When I run vmstat (vmstat -w 1 -c 10 output attached),
it shows ten processes blocked due to I/O, 1 runnable
and a total of four disk transactions during this
particular ten-second interval.

Accoring to postmark [1] da0 is capable of 250
transactions per second.

Interrupts look fine (200 of the 300/sec is the clock,
right?).

No swapping. No network traffic to speak of.

High CPU usage.

Paging looks really high to me. I have plenty of RAM
and swap--so why is it so high?

Is there a kernel param I can tweak to improve
performance?

Note: While this was running, I tried running vmstat
-m and got the kmem_map too small error. (I
read the FAQ entry and haven't yet rebuilt the
kernel using VM_KMEM_SIZE_MAX.)

Should I make VM_KMEM_SIZE_MAX 400MB as mentioned in the FAQ?

Any other kernel settings to switch from defaults when I
build the kernel?


Case 2: (mirror/gm0, da0 + da1 300GB SCSI)

This case is more complex and more important. This box
was heavily loaded, running SpamAssassin + ClamAv in one
jail, Courier-MTA + MySql in another, and dspamd + pf in
the jail host.

It was processing about one incoming email per second,
and had 20,000 entries in the spamd greylist and 4,000
in the whitelist. The black list was not loaded.

Courier-authlib was using MySQL to figure out where to
deliver the mail to for each of 20,000 virtual users.

Courier-MTA was calling the ClamAV daemon via TCP/IP
from a courier filter and calling SpamAssassin via
TCP/IP from maildrop.

Spamd (as in OpenBSD, not Apache) and courier were
hitting the disk hard--I watched top in "m" mode [2] and
over a five minute period these two apps had about
pretty much 50% each of the WRITES.

The only thing that jumps out at me is that when things
finally do get written to disk, the context switches go
really high.

Is this a VM issue of some sort? The disk transactions
per second are really low; this 300GB SCSI can do
730/sec according to postmark (well, this was testing on
a single disk not with the RAID1 gmirror).


Thanks for any tips.

m

[1] /usr/ports/benchmarks/postmark

[2] Where can I find docs on what the columns mean in top's
"m" mode?

vmstat.case1.out
Description: Text document

vmstat.case2.out
Description: Text document

_______________________________________________
freebsd-questions@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@xxxxxxxxxxx"
--- End Message ---

_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"