Hi,
I have a cluster of Ubuntu Dapper 6.06 LTS servers acting as
toasters, serving POP3, IMAP, IMAP proxy, webmail for a large mail
cluster.
During the last couple of weeks, I've had two serves in the cluster
unexpectedly stop doing any NFS operations to one of our NetApp
filers. Each server mounts three separate NetApp filers, and we've
got over 30 servers in this cluster mounting the same filers, and
running with exactly the same configuration (we kickstart all our
servers, so they're exactly the same, some just doing other tasks).
I see these errors in my syslog, but I suspect (based on graphs for
at what time I/O wait went up, and at what time these entries appears
in the logs) that they only appear here after the problem has already
started (because it can't do a stat on the NFS fs), so I doubt
they're of any use in solving my problem.
Oct 11 12:33:34 toaster01-mail kernel: [863316.270699] nfs_statfs:
statfs error = 512
Oct 11 12:33:37 toaster01-mail kernel: [863319.113889] nfs_statfs:
statfs error = 512
Oct 11 12:34:32 toaster01-mail kernel: [863374.713564] nfs_statfs:
statfs error = 512
Oct 11 12:34:35 toaster01-mail kernel: [863377.134229] nfs_statfs:
statfs error = 512
Oct 11 12:35:36 toaster01-mail kernel: [863438.108434] nfs_statfs:
statfs error = 512
Oct 11 12:35:45 toaster01-mail kernel: [863447.372073] nfs_statfs:
statfs error = 512
Oct 11 12:46:44 toaster01-mail kernel: [864105.539485] nfs_statfs:
statfs error = 512
I've checked each network interface from eth1 (backend storage
interface) through to the fibre gig port that connects to the NetApp,
and there's no errors, collisions, etc. I've also read the 32Mb
Netapp filer report, and can't spot anything unusual there. Also
tried doing a strace for say listing files on /mailspool/mail8, but
just get ... "lstat("/mailspool/mail8", <unfinished ...>".
I've got one server currently out of service, that's experiencing
this "condition", so I can get any stats/output from this server if
there's anything anyone can think of?
Here's some details:-
Kernel:
2.6.15-27-amd64-k8
linux-image-2.6.15-27-amd64-server
2.6.15-27.50 Linux kernel image for version 2.6.15 on
Ser
NFS client:
nfs-common
1.0.7-3ubuntu2 NFS support files common to client and
serve
Filer mount options:
10.1.25.212:/vol/vol0/mail8-export /mailspool/mail8 nfs
rw,hard,intr,timeo=600,retrans=2,rsize=32768,wsize=32768 0 0
eth1 interface output:
eth1: negotiated 100baseTx-FD, link ok
eth1 Link encap:Ethernet HWaddr 00:17:08:50:78:41
inet addr:10.1.25.141 Bcast:10.1.25.255 Mask:255.255.255.0
inet6 addr: fe80::217:8ff:fe50:7841/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1468254714 errors:0 dropped:0 overruns:0 frame:0
TX packets:995375684 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1677649220750 (1.5 TiB) TX bytes:155688040628
(144.9 GiB)
Interrupt:193
nfsstat output:
Client rpc stats:
calls retrans authrefrsh
460431933 104 0
Client nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 114645897 24% 2066807 0% 78704007 17% 151704157 32%
0 0%
read write create mkdir symlink mknod
53366969 11% 11421861 2% 6827536 1% 8280 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
10806338 2% 233 0% 2897780 0% 3458142 0% 5434425 1% 19085492 4%
fsstat fsinfo pathconf commit
68 0% 4 0% 0 0% 3933 0%
Any ideas, things to try?
Cheers,
Jaco
--
bje@xxxxxxxxxxxxxxxxxx
the faculty of making fortunate discoveries
|
