Subject: Speeding up truncate - msg#00092

Hello,
I've been trying to improve unlink speed under ext3, because we are
getting huge slowdowns at the end of scientific compute jobs on Lustre,
when a thousand nodes each delete a bunch of files 10-50MB in size.

I noticed the blocks_for_truncate() fix from 2.5 has just made it into
2.4.21-pre5 (I was on the verge of sending the identical patch just now).

This won't fix our unlink speed problem though, because these inodes
would need the maximum transaction size for truncates anyways. At most
we would do a bit of extra reservation at the end of the truncate.

For the unlink speed issue I'm creating a thread which runs in the
background that gets handed inodes to delete/truncate (after the name
has been removed, and the inode added to the orphan list) and inodes
are deleted asynchronously. This lets the applications proceed without
waiting for 500GB of data to be unlinked/journaled, yet is safe from any
crash issues because the inode has alreay been linked into the orphan list
(it would make for a long recovery on crash, but that is no different
than before). Most of the scientific apps are bursty writers, so deferring
the unlinks to a background thread is an overall win.


What I would _really_ like to do is reduce the amount of data that
truncate needs to write, instead of (or in addition to) only pushing
the truncate latency into the background. The issue with truncate is
that it can possibly need huge transaction sizes, and we need to be
able to restart the transaction in the middle if we can't get a single
transaction large enough to do the entire truncate.

However, it is also possible to do the minimum amount of updates required
to keep a consistent filesystem in a single transaction and write out a
lot less data if we _can_ get a single transaction large enough to hold
the data, and we know how large that transaction needs to be. Basically,
we would need to count the number of bitmaps that were being modified,
the group desciptor blocks, add an inode table block, and the superblock,
and get a transaction of that size and we can do a consistent truncate.
We don't need to zero all of the {d,t}indirect blocks that we are dropping
entirely, and that is by far the largest amount of data to be written
(1/1024 of the size of the file, vs ~1/32768 for the bitmaps).

Depending on the journal size, there is obviously an upper limit to
the size of a file (and/or number of allocated blocks) for which this
is practical (after which we would need to go to the slow method),
but I think it can make a huge difference in performance. For Lustre,
we are of course willing to have huge journals in order to encompass
the truncate of large files in a single operation if it means we need
to journal 1/32 as much data to disk. The fringe side benefit would be
that we can leave the allocated block lists the inode (like ext2 does),
and we could get undelete for ext3 working again.

My only worry is the number of headaches I'll get looking at
extN_free_branches() and worrying about how to get partial truncate right
(which can be optimized similarly, with the addition of at most a single
indirect, dindirect, and tindirect block to the transaction size). I
think the performance hit of reading all of the block allocation lists
to count bitmaps/descriptors will be significantly less than the cost of
writing them all out again. We will need to read all of these allocation
lists anyways, so it is mostly just a pre-read for the real truncate.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/



-------------------------------------------------------
This SF.net email is sponsored by:Crypto Challenge is now open!
Get cracking and register here for some mind boggling fun and
the chance of winning an Apple iPod:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en