Software raid5 inefficiencies

I've been playing with software raid5 on a heavily loaded newsserver.

This machine is part of a diablo setup, and basically it's a
database server for usenet articles.

All articles are stored in multiple multi-megabyte sized files on
the filesystem. When a client asks for data (an article, average
300 KB) the location is looked up in a (fast, cached in memory)
database, the file is opened, the server seeks to the offset of
the article and the 300KB is served to the client.

Because many clients are connecting to the server simultaneously,
it's better to use a large stripe size - a stripe size smaller
than 600 KB means that to serve one request, multiple disks need
to seek, and since the clients are higly parallel and the data
is spread randomly over the disks, that is bad. You want to serve
each read request from one disk.

So I'm using a stripe size of 4 MB.

Now it works fine but I'm hitting a few bottlenecks:

1. In this case, for every 4K read I need a stripe_head. So the
  standard NR_STRIPES = 256 is way to low. I increased
  NR_STRIPES to 1024 and that helps, but with 7 disks it
  uses 29 MB of unswappable kernel memory (and 2048 is even
  better but uses 58 MB). With 2 RAID5 devices this adds up.

2. With a heavy read load, on this PIV/3Ghz system time used
  reaches 95% just reading from the raid5 device, and I can't
  read faster than about 50-60 MB/sec. From oprofile, the
  bottleneck appears to be raid5.c::copy_data()

The solution, I think, is:

1. Do not allocate sh->dev[ALL].page when requesting a stripe_head-
  just allocate them for the devices we actually need to read/write.
  Keep the pages on a seperate LRU.

2. Do not copy read data into the stripe_head at all when just
  reading data - just remap the BIOs like dm does. You only need
  to copy the data from/to the stripe_head when the same part of
  the device is being written to, or is in degraded mode.

This should boost performance in normal circumstances a lot, I
think. Is anyone working on this yet ? Comments, flames ?

[please keep the cc, I'm not on the list]

Mike.

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html