Subject: RE: Multiple threads, hard-linking performance
problem - msg#00119

Previous Message by Date:

Multiple threads, hard-linking performance problem

Hi, I'm seeing a performance problem with creating hundreds of hard links through multiple threads, occasionally while large file writes are going on. First i'll explain my configuration and then get into the problem. NFS Server: RedHat Linux 7.2, kernel 2.4.18-26.7 IBM x335 2.0Ghz cpu, 1GB RAM Gigabit ethernet Disk info: IBM EXP300 with 14 Ultra320 SCSI spindles, IBM ServeRAID-4(?) running RAID-10, chunk size 64k single ext3 partition mounted rw,noatime exported as rw,no_root_squash nfsv3 NFS Client: RedHat Linux 7.2, kernel 2.4.18-26.7 IBM x360 4x1.6Ghz cpu, 1GB RAM Gigabit ethernet connection to NFS Server mounts nfs partition as rw,hard,intr,rsize=32768,wsize=32768 nfsv3 Usage behavior: The NFS client reads from a separate NFS mount (same nfs parameters, identical drives except for the noatime) and will do the following through the aforementioned NFS mount: 1) Multiple threads (consider 3 max) checking file existence and timestamp, and doing a file copy through Java file copy. 2) Multiple threads (consider 3 max) creating soft link to files created by #1 3) Multiple threads (consider 3 max) creating hundreds (200,600,2000 for example) hard links to this soft link. I'm aware of portability concerns with this behavior. These links are created, 2 per subdirectory in a directory tree with hundreds of directories (500, 1500, 5000 corresponding to the above example). Statistics: All tests done with 1.46GB file: Local EXP300 to EXP300 file copy - 48.91 sec Copy from NFS Client's local drive to NFS share - 68.23 sec Copy between NFS Client's 2 mounted NFS partitions (on different servers) - 120 sec Same as above, but with noatime on NFS Server's exported partition - 102 sec Java file copy throughput between 2 mounted partitions (along with other processing overhead) 150 sec Issue description: While #1 and #2 are going in the background, #3 - linking 200 files appeared to take approx. 7 minutes per thread. This is approx 84 hard links created per minute with 3 concurrent threads, which seems like a possible issue. Going directly to the filesystem and creating hard links manualy appeared to be fast, and an extremely slow "ls" command from the NFS client on the NFS-mounted partition appeared to be slow. Another test, with #1 at 2 threads, #2 at 3 threads, and #3 at 5 threads, with 600 links per thread revealed about 20-25 minutes per thread to complete with 5 concurrent threads. This still seems low to me. I have read through the mailing list archives, and a couple of the notes seemed as if they may have relevance to this issue. In particular, poor handling of large numbers of small files by ext3, and also Alan Powell's post about ext3 directory indexing: http://sourceforge.net/mailarchive/forum.php?thread_id=1671309&forum_id=4930 I may consider trying XFS or ReiserFS out of curiosity, but since this architecture will soon change, I just was wondering if there were any obvious configuration changes I should make, and also thought that it might be good to share this experience with the list. Thanks, Gordon Luk System Administrator Sequoia Broadband, Inc. http://www.sequoiabroadband.com

Next Message by Date:

wsize & PAGE_SIZE issues on IA64 clients.

Hi Folks, I've been chasing a very poor performance problem from a SuSE Sles-8 (2.4.19) system on an IA64 box to an AIX NFS server. Writes to the same AIX server on an IA32 box also running Sles-8 performed as expected. These are v3 mounts with a 4K rsize/wsize. Eg. mount -t nfs -o nfsvers=3,udp,rsize=4096,wsize=4096,hard rs75:/linux_test /linux_test I tracked the problem down to the default PAGE_SIZE on ia64, which is 16K, versus the wsize which was 4K. In nfs_updatepage() if the wsize is smaller than the page size the write is performed synchronously. It appears though that the block size is then reduced further, in this case to 512 bytes. From what I could work out, it was cp_new_stat() in linux/fs/stat.c that determined the new preferred block size from the remote filesystem. On ia32 both the page size and the wsize were 4K, and no problem was seen. Can someone give me rough explanation of why that logic is used? ie. Why the page is written synchronously if its smaller than the page size? and Why even when its written synchronously the wsize wasn't used, but the remote filesystems block size was used? The fix/workaround was to either increase wsize to 16K, or reduce PAGE_SIZE to 4K, obviously increasing wsize to 16K makes more sense. However this leads to a problem between Linux clients and servers where the maximum supported block size on the server is 8K and the page size on the IA64 client is 16K. Is increasing the maximum blocksize for the server as simple as changing NFSSVC_MAXBLKSIZE (linux/include/linux/nfsd/const.h) to 16K or 32K ? Or is more porting work required? Cheers, Mark. -- Mark Price IBM Linux Change Team (503)-578-7524 ------------------------------------------------------- This SF.net email is sponsored by: The Definitive IT and Networking Event. Be There! NetWorld+Interop Las Vegas 2003 -- Register today! http://ads.sourceforge.net/cgi-bin/redirect.pl?keyn0001en _______________________________________________ NFS maillist - NFS@xxxxxxxxxxxxxxxxxxxxx https://lists.sourceforge.net/lists/listinfo/nfs

Previous Message by Thread:

Multiple threads, hard-linking performance problem

Hi, I'm seeing a performance problem with creating hundreds of hard links through multiple threads, occasionally while large file writes are going on. First i'll explain my configuration and then get into the problem. NFS Server: RedHat Linux 7.2, kernel 2.4.18-26.7 IBM x335 2.0Ghz cpu, 1GB RAM Gigabit ethernet Disk info: IBM EXP300 with 14 Ultra320 SCSI spindles, IBM ServeRAID-4(?) running RAID-10, chunk size 64k single ext3 partition mounted rw,noatime exported as rw,no_root_squash nfsv3 NFS Client: RedHat Linux 7.2, kernel 2.4.18-26.7 IBM x360 4x1.6Ghz cpu, 1GB RAM Gigabit ethernet connection to NFS Server mounts nfs partition as rw,hard,intr,rsize=32768,wsize=32768 nfsv3 Usage behavior: The NFS client reads from a separate NFS mount (same nfs parameters, identical drives except for the noatime) and will do the following through the aforementioned NFS mount: 1) Multiple threads (consider 3 max) checking file existence and timestamp, and doing a file copy through Java file copy. 2) Multiple threads (consider 3 max) creating soft link to files created by #1 3) Multiple threads (consider 3 max) creating hundreds (200,600,2000 for example) hard links to this soft link. I'm aware of portability concerns with this behavior. These links are created, 2 per subdirectory in a directory tree with hundreds of directories (500, 1500, 5000 corresponding to the above example). Statistics: All tests done with 1.46GB file: Local EXP300 to EXP300 file copy - 48.91 sec Copy from NFS Client's local drive to NFS share - 68.23 sec Copy between NFS Client's 2 mounted NFS partitions (on different servers) - 120 sec Same as above, but with noatime on NFS Server's exported partition - 102 sec Java file copy throughput between 2 mounted partitions (along with other processing overhead) 150 sec Issue description: While #1 and #2 are going in the background, #3 - linking 200 files appeared to take approx. 7 minutes per thread. This is approx 84 hard links created per minute with 3 concurrent threads, which seems like a possible issue. Going directly to the filesystem and creating hard links manualy appeared to be fast, and an extremely slow "ls" command from the NFS client on the NFS-mounted partition appeared to be slow. Another test, with #1 at 2 threads, #2 at 3 threads, and #3 at 5 threads, with 600 links per thread revealed about 20-25 minutes per thread to complete with 5 concurrent threads. This still seems low to me. I have read through the mailing list archives, and a couple of the notes seemed as if they may have relevance to this issue. In particular, poor handling of large numbers of small files by ext3, and also Alan Powell's post about ext3 directory indexing: http://sourceforge.net/mailarchive/forum.php?thread_id=1671309&forum_id=4930 I may consider trying XFS or ReiserFS out of curiosity, but since this architecture will soon change, I just was wondering if there were any obvious configuration changes I should make, and also thought that it might be good to share this experience with the list. Thanks, Gordon Luk System Administrator Sequoia Broadband, Inc. http://www.sequoiabroadband.com

Next Message by Thread:

wsize & PAGE_SIZE issues on IA64 clients.

Hi Folks, I've been chasing a very poor performance problem from a SuSE Sles-8 (2.4.19) system on an IA64 box to an AIX NFS server. Writes to the same AIX server on an IA32 box also running Sles-8 performed as expected. These are v3 mounts with a 4K rsize/wsize. Eg. mount -t nfs -o nfsvers=3,udp,rsize=4096,wsize=4096,hard rs75:/linux_test /linux_test I tracked the problem down to the default PAGE_SIZE on ia64, which is 16K, versus the wsize which was 4K. In nfs_updatepage() if the wsize is smaller than the page size the write is performed synchronously. It appears though that the block size is then reduced further, in this case to 512 bytes. From what I could work out, it was cp_new_stat() in linux/fs/stat.c that determined the new preferred block size from the remote filesystem. On ia32 both the page size and the wsize were 4K, and no problem was seen. Can someone give me rough explanation of why that logic is used? ie. Why the page is written synchronously if its smaller than the page size? and Why even when its written synchronously the wsize wasn't used, but the remote filesystems block size was used? The fix/workaround was to either increase wsize to 16K, or reduce PAGE_SIZE to 4K, obviously increasing wsize to 16K makes more sense. However this leads to a problem between Linux clients and servers where the maximum supported block size on the server is 8K and the page size on the IA64 client is 16K. Is increasing the maximum blocksize for the server as simple as changing NFSSVC_MAXBLKSIZE (linux/include/linux/nfsd/const.h) to 16K or 32K ? Or is more porting work required? Cheers, Mark. -- Mark Price IBM Linux Change Team (503)-578-7524 ------------------------------------------------------- This SF.net email is sponsored by: The Definitive IT and Networking Event. Be There! NetWorld+Interop Las Vegas 2003 -- Register today! http://ads.sourceforge.net/cgi-bin/redirect.pl?keyn0001en _______________________________________________ NFS maillist - NFS@xxxxxxxxxxxxxxxxxxxxx https://lists.sourceforge.net/lists/listinfo/nfs