bad (2 times) gfs read performance under x86_64 compared to i386 WAS: Re: AW: GNBD multipath with devicemapper? -- possible solution

Hi

I have done some futher testing on this issue, and I noticed an 
interesting behavior.
We have an installation with RHEL4 on i386 hardware (old SAN) and one under
new x86_64 (dual CPU each hyptherthreading).

On i386 read performance is better than write performance (under ext3 
and gfs)
On x86_64 read performance is better the write performance under ext3
but two times bader under gfs (Filsystem created with lock_nolock)

Here a short summary:
i386 ext3
write:    1:16
read:    0:55
read:    0:43 (blockdev --setra 8192)

i386 ext3
write:    1:30
read:    1:08
read:    1:08 (blockdev --setra 8192)


x86_64 ext3
write:    0:35
read:    0:27
read:    0:18 (blockdev --setra 8192)

x86_64 gfs
write:    0:27
read:    1:03
read:    1:03 (blockdev --setra 8192)


The gnbd tests I did last week were under x86_64 to, so that might not 
be an gnbd issue, but an gfs issue under
x86_64.
The tests where done under
2.6.9-5.0.3.ELsmp and  2.6.9-6.38.ELsmp with current GFS from  CVS 
(RHEL4 TAG) with no difference.

Can anyone reproduce this behavior?

Greetings

Hansjörg


Here the detailed tests


[root@chianti sda1]# uname -a
Linux chianti.itsd.de 2.6.9-6.38.EL #1 Wed Apr 13 01:36:09 EDT 2005 i686 
athloni386 GNU/Linux


[root@chianti ~]# mkfs.ext3 /dev/sda1
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 0, rsv_gdb = 1024
Dateisystem-Label=
OS-Typ: Linux
BlockgröÃe=4096 (log=2)
FragmentgröÃe=4096 (log=2)
5341184 Inodes, 10673176 Blöcke
533658 Blöcke (5.00%) reserviert für den Superuser
erster Datenblock=0
Maximum filesystem blocks=12582912
326 Blockgruppen
32768 Blöcke pro Gruppe, 32768 Fragmente pro Gruppe
16384 Inodes pro Gruppe
Superblock-Sicherungskopien gespeichert in den Blöcken:
       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 
2654208,
       4096000, 7962624

Schreibe Inode-Tabellen: erledigt
inode.i_blocks = 98312, i_size = 4243456
Erstelle Journal (8192 Blöcke): erledigt
Schreibe Superblöcke und Dateisystem-Accountinginformationen: erledigt

Das Dateisystem wird automatisch alle 28 Mounts bzw. alle 180 Tage 
überprüft,
je nachdem, was zuerst eintritt. Veränderbar mit tune2fs -c oder -t .



[root@chianti ~]# mount /dev/sda1 /mnt/sda1/
[root@chianti ~]# cd /mnt/sda1
[root@chianti sda1]# free
            total       used       free     shared    buffers     cached
Mem:        515736     164504     351232          0      43544      25104
-/+ buffers/cache:      95856     419880
Swap:       786424        152     786272


[root@chianti sda1]# time mkfile 1000M a

real    1m16.242s
user    0m0.009s
sys     0m16.273s
[root@chianti sda1]# time mkfile 1000M b

real    1m17.738s
user    0m0.009s
sys     0m15.511s
[root@chianti sda1]# time cat a > /dev/null

real    0m55.025s
user    0m0.193s
sys     0m6.529s

[root@chianti sda1]# time cat b > /dev/null

real    0m54.241s
user    0m0.220s
sys     0m6.331s


[root@chianti sda1]# blockdev --setra 8192 /dev/sda1
[root@chianti sda1]# time cat a > /dev/null

real    0m43.565s
user    0m0.189s
sys     0m6.014s

[root@chianti sda1]# time cat b > /dev/null

real    0m43.214s
user    0m0.196s
sys     0m6.727s

[root@chianti ~]# gfs_mkfs -p lock_nolock -j 3 /dev/sda1
This will destroy any data on /dev/sda1.
 It appears to contain a EXT2/3 filesystem.

Are you sure you want to proceed? [y/n] y
Device:                    /dev/sda1
Blocksize:                 4096
Filesystem Size:           10573560
Journals:                  3
Resource Groups:           162
Locking Protocol:          lock_nolock
Lock Table:

Syncing...
All Done

[root@chianti sda1]# time mkfile 1000M a

real    1m27.421s
user    0m0.010s
sys     0m12.202s
[root@chianti sda1]# time mkfile 1000M b

real    1m35.009s
user    0m0.006s
sys     0m12.513s
[root@chianti sda1]# time cat a > /dev/null

real    1m12.609s
user    0m0.153s
sys     0m9.980s
[root@chianti sda1]# time cat b > /dev/null

real    1m7.989s
user    0m0.154s
sys     0m10.427s


[root@chianti sda1]# blockdev --setra 256 /dev/sda1
[root@chianti sda1]# time cat a > /dev/null

real    1m8.402s
user    0m0.082s
sys     0m8.841s
[root@chianti sda1]# time cat b > /dev/null

real    1m8.647s
user    0m0.124s
sys     0m9.565s
[root@chianti sda1]# blockdev --setra 8192 /dev/sda1
[root@chianti sda1]# time cat a > /dev/null

real    1m8.419s
user    0m0.115s
sys     0m9.262s



[root@rmvbs02 ~]# uname -a
Linux rmvbs02.cluster.robotic.dlr.de 2.6.9-5.0.3.ELsmp #1 SMP Sat Feb 19 
15:45:14 CST 2005 x86_64 x86_64 x86_64 GNU/Linux


[root@rmvbs02 ~]# mkfs.ext3 /dev/sdb1
mke2fs 1.35 (28-Feb-2004)
max_blocks 4294967295, rsv_groups = 131072, rsv_gdb = 977
Dateisystem-Label=
OS-Typ: Linux
BlockgröÃe=4096 (log=2)
FragmentgröÃe=4096 (log=2)
97615872 Inodes, 195221872 Blöcke
9761093 Blöcke (5.00%) reserviert für den Superuser
erster Datenblock=0
Maximum filesystem blocks=4294967296
5958 Blockgruppen
32768 Blöcke pro Gruppe, 32768 Fragmente pro Gruppe
16384 Inodes pro Gruppe
Superblock-Sicherungskopien gespeichert in den Blöcken:
       32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 
2654208,
       4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
       102400000

Schreibe Inode-Tabellen: erledigt
inode.i_blocks = 140696, i_size = 4243456
Erstelle Journal (8192 Blöcke): erledigt
Schreibe Superblöcke und Dateisystem-Accountinginformationen: erledigt

Das Dateisystem wird automatisch alle 38 Mounts bzw. alle 180 Tage 
überprüft,
je nachdem, was zuerst eintritt. Veränderbar mit tune2fs -c oder -t .


[root@rmvbs02 tmp]# free
            total       used       free     shared    buffers     cached
Mem:       1026364      65960     960404          0        684      15544
-/+ buffers/cache:      49732     976632
Swap:      2739040       1360    2737680
[root@rmvbs02 tmp]# time mkfile 2000M a

real    0m35.560s
user    0m0.005s
sys     0m7.596s
[root@rmvbs02 tmp]# time mkfile 2000M b

real    0m30.917s
user    0m0.005s
sys     0m7.707s
[root@rmvbs02 tmp]# time mkfile 2000M c

real    0m40.663s
user    0m0.010s
sys     0m7.806s
[root@rmvbs02 tmp]# time cat a > /dev/null

real    0m28.787s
user    0m0.122s
sys     0m2.818s
[root@rmvbs02 tmp]# time cat b > /dev/null

real    0m27.468s
user    0m0.124s
sys     0m2.678s
[root@rmvbs02 tmp]# blockdev --getra /dev/sdb1
128
[root@rmvbs02 tmp]# blockdev --setra 8192 /dev/sdb1
[root@rmvbs02 tmp]# time cat c > /dev/null

real    0m18.541s
user    0m0.105s
sys     0m2.064s
[root@rmvbs02 tmp]# time cat a > /dev/null

real    0m18.464s
user    0m0.117s
sys     0m2.035s


[root@rmvbs02 ~]# gfs_mkfs -p lock_nolock -j 3 /dev/sdb1
This will destroy any data on /dev/sdb1.
 It appears to contain a EXT2/3 filesystem.

Are you sure you want to proceed? [y/n] y

Device:                    /dev/sdb1
Blocksize:                 4096
Filesystem Size:           195108656
Journals:                  3
Resource Groups:           2978
Locking Protocol:          lock_nolock
Lock Table:

Syncing...
All Done
[root@rmvbs02 ~]# mount -t gfs /dev/sdb1 /mnt/tmp/
[root@rmvbs02 ~]# cd /mnt/tmp/
[root@rmvbs02 tmp]# time mkfile 2000M a

real    0m24.616s
user    0m0.004s
sys     0m4.892s
[root@rmvbs02 tmp]# time mkfile 2000M b

real    0m27.023s
user    0m0.005s
sys     0m5.027s
[root@rmvbs02 tmp]# time mkfile 2000M c

real    0m29.205s
user    0m0.005s
sys     0m5.163s


[root@rmvbs02 tmp]# time cat a > /dev/null

real    1m4.698s
user    0m0.120s
sys     0m6.138s
[root@rmvbs02 tmp]# time cat b > /dev/null

real    1m2.958s
user    0m0.132s
sys     0m6.175s
[root@rmvbs02 tmp]# time cat c > /dev/null

real    1m2.867s
user    0m0.109s
sys     0m6.079s
[root@rmvbs02 tmp]# blockdev --getra  /dev/sdb1
8192
[root@rmvbs02 tmp]# blockdev --setra 256  /dev/sdb1
[root@rmvbs02 tmp]# time cat a > /dev/null

real    1m2.931s
user    0m0.101s
sys     0m6.073s





Benjamin Marzinski wrote:

> On Thu, Apr 21, 2005 at 07:58:49AM +0200, Hansjörg Maurer wrote:
>  
>
> > Hi
> >
> > thank you very much,
> > Do you have any idea what causes the speed difference between vanilla 
> > and RHEL4 kernel?
> > Should I officially file a bug in bugzilla, or do you take care of the 
> > problem?
> >    
>
> Go ahead and file a bug on gnbd if you are interested in tracking the status
> of this. I'll put it on my list of things to do.
>
> -Ben
>
>  
>
> > Greetings
> >
> >
> > Hansjörg
> >
> >
> >
> >
> >
> >    
> >
> > > I got those numbers on a vanilla 2.6.11 kernel.
> > > Running on a 2.6.9-6.24.EL kernel, I get
> > > 45.6608 MB/sec writes 
> > > 35.8578 MB/sec reads
> > >
> > > Not as pronouced as your numbers, but a noticeable speed difference.
> > >
> > > -Ben
> > >
> > >
> > >      
> > >
> > > > Thank you very much
> > > >
> > > > Hansjörg
> > > >
> > > >  
> > > >
> > > >        
> > > >
> > > > >    
> > > > >
> > > > >          
> > > > --
> > > > _________________________________________________________________
> > > >
> > > > Dr.  Hansjoerg Maurer           | LAN- & System-Manager
> > > >                             |
> > > > Deutsches Zentrum               | DLR Oberpfaffenhofen
> > > > f. Luft- und Raumfahrt e.V.   |
> > > > Institut f. Robotik             |
> > > > Postfach 1116                   | Muenchner Strasse 20
> > > > 82230 Wessling                  | 82234 Wessling
> > > > Germany                         |
> > > >                             |
> > > > Tel: 08153/28-2431              | E-mail: Hansjoerg.Maurer@xxxxxx
> > > > Fax: 08153/28-1134              | WWW: http://www.robotic.dlr.de/
> > > > __________________________________________________________________
> > > >
> > > >
> > > > There are 10 types of people in this world, 
> > > > those who understand binary and those who don't.
> > > >
> > > > --
> > > > Linux-cluster mailing list
> > > > Linux-cluster@xxxxxxxxxx
> > > > http://www.redhat.com/mailman/listinfo/linux-cluster
> > > >  
> > > >
> > > >        
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster@xxxxxxxxxx
> > > http://www.redhat.com/mailman/listinfo/linux-cluster
> > >
> > >
> > >
> > >      
> > --
> > _________________________________________________________________
> >
> > Dr.  Hansjoerg Maurer           | LAN- & System-Manager
> >                               |
> > Deutsches Zentrum               | DLR Oberpfaffenhofen
> > f. Luft- und Raumfahrt e.V.   |
> > Institut f. Robotik             |
> > Postfach 1116                   | Muenchner Strasse 20
> > 82230 Wessling                  | 82234 Wessling
> > Germany                         |
> >                               |
> > Tel: 08153/28-2431              | E-mail: Hansjoerg.Maurer@xxxxxx
> > Fax: 08153/28-1134              | WWW: http://www.robotic.dlr.de/
> > __________________________________________________________________
> >
> >
> > There are 10 types of people in this world, 
> > those who understand binary and those who don't.
> >
> > --
> > Linux-cluster mailing list
> > Linux-cluster@xxxxxxxxxx
> > http://www.redhat.com/mailman/listinfo/linux-cluster
> >    
>
> --
> Linux-cluster mailing list
> Linux-cluster@xxxxxxxxxx
> http://www.redhat.com/mailman/listinfo/linux-cluster
>
>