Re: [announcement] ntfs-3g: open source read-write driver

On Sat, 15 Jul 2006, Szakacsits Szabolcs wrote:
> On Fri, 14 Jul 2006, Anton Altaparmakov wrote:
> > On Fri, 14 Jul 2006, Szakacsits Szabolcs wrote:
> > > As part of the Linux-NTFS project, I'm happy to announce my contribution
> > > to ntfsmount and libntfs which resulted ntfs-3g, a read-write ntfs driver,
> > > capable for unlimited file creation and deletion.
> > 
> > Wow, well done!  Although "unlimited" is a bit of an exagerration as you 
> > yourself admit below.
> 
> Practically the limits the ones I documented, they occure quite rarely.

True but still...

> > I had a quick look at the code and it chickens out of a lot of corner 
> > cases which are quite hard...
> 
> There are four such corner cases and they are not so hard. I have code 
> for some of them but there was no time to massively test so they aren't 
> included.
>  
> > Also I may be wrong as I did not spend enough time on it but if you run 
> > out of space on the partition in the middle of a complex add or delete 
> > operation you appear to leave a corrupt directory.  In general the error 
> > handling appears to leave things corrupt and does not attempt to roll back 
> > / correct things.  Is this a fair description?
> 
> Partly. Thanks, I forgot to document this. When it was possible I tried to 
> make the changes that things stay consistent whenever the free space runs 
> out or whatever error happens. Chkdsk must be always able to fix such 
> inconsistencies. I was thinking to refuse serving requests when free space 
> was getting low but currently the MFT extension limit fills this role 
> usually.
> 
> Please also note that "full disk" handling is not so simple as just rolling 
> back things because lower layers can report success when later they can 
> fail which point you already released all the needed in-memory info which 
> would be needed to rollback the state. It's not only a simply rollback 
> problem.

I don't understand that...  At least in the kernel driver what you 
describe can never happen.  All allocations are done a-priori thus you 
cannot run out of space once you start doing anything to the metadata.  I 
admit that my driver will not cope with disk write error as that is indeed 
asynchronous.  But that is completely orthogonal to "full disk" handling.

> > And of course you use recursion which you cannot use in kernel context so 
> > the code could never be moved to a kernel driver (even ignoring the fact 
> > that it does not do locking or 
> 
> Currently I'm not interested in the kernel driver. It's a lost case for 
> over a decade. Full read-write could be done in user space pretty fast and 

Why a lost case?  Just a harder to do one.  But I am sure it will be 
faster than the user space driver.  Just the userspace/kernel 
communication slow down would make sure of that...

> I can't see drawbacks, only benefits:
> 
>       - NTFS is huge and complex, not for the kernel. Crash in kernel 
>           (hw error, corrupt ntfs, etc) and game is over. Crash in user
>         space then just restart the service.

So I make sure it does not crash.  (-;

>       - kernel has a lot of limitations, restrictions which are all gone.
> 
>       - fedora/redhat users have never ending hassles with installing
>         the driver. Instead they could install ntfs-3g once and forget 
>           the issue forever.

Yes, I personally do not care for fedora/redhat users.  My opinion is that 
we should not provide rpms for them and instead tell them to go and 
complain to the redhat/fedora management and ask them to provide the ntfs 
driver compiled in the kernel.  And if they refuse they should switch to 
SuSE (or whatever).  If a hundred thousand people do that and tell 
redhat/fedora admins about this fact I am pretty sure that redhat/fedora 
will rethink their position and allow ntfs into their distros...

> > endianness handling).
> 
> That's planned in a later phase as part of a major cleanup by abstracting 
> access to metadata.

Fair enough.

> > Wow!  Enjoy it!  That sounds like an amazing experience!  I hope you bring 
> > back some nice photos for us all to wonder at!  (-;
> 
> Thanks :-) I planned video, we'll see!

Even better!  (-:

> > > Version  1.03    ------Sequential Create------ --------Random 
> > > Create--------
> > >                  -Create-- --Read--- -Delete-- -Create-- --Read--- 
> > > -Delete--
> > >            files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
> > > %CP
> > > reiserfs     16k 21459  99 +++++ +++ 17856  96 20172  98 +++++ +++ 16414  
> > > 96
> > > jfs          16k  7015  13 +++++ +++  5868  10  3068  14 +++++ +++  1075  
> > >  3
> > > ntfs-3g      16k  3021  99 14291  99  5226  99  3548  99 16149  99  5223  
> > > 99
> > > xfs          16k  2401  17 +++++ +++  2095  15  2301  20 +++++ +++   347  
> > >  2
> > > ext3         16k  1862  96 +++++ +++ +++++ +++  1914  96 +++++ +++  9695  
> > > 98
> > > minix        16k  1450  97 +++++ +++ 18148  94  1694  97 +++++ +++  4847  
> > > 98
> > > fat32        16k   366  97 +++++ +++  1809  97   428  97 +++++ +++  1361  
> > > 97
> > > paragon ntfs 16k    58  98  1259  99   245  99    55  99 +++++ +++   832  
> > > 99
> > > captive ntfs    Always crashes early on file creations and loses files.
> > >           This was also confirmed and fix denied by its developer.
> > > 
> > > We estimate the ntfs-3g driver performance could be increased by several 
> > > times.
> > 
> > Wow, nice looking numbers.  I haven't done anything like that for my code.  
> > I wonder how it would compare 
> 
> I wonder, if it would actually work ;) It's also a very good function test 
> for the directory operation. For example I had problems with FAT32 failing 
> sometimes and Captive always crashing after creation of 60-130 files.

I have done stress tests just not with a benchmark...

> > given it is in the kernel and IMHO a lot better optimized than your 
> > version...  E.g. I use binary search where you use sequential and I never 
> > have to repeat a search twice during a single operation as that is all 
> > cached the first time although I admit that the cache overhead may 
> > introduce a delay in itself so the benefit in my code may not be as much 
> > as one would expect initially...
> 
> As I wrote, functionality comes first, especially as I don't have more than 
> max a few hours in the nights to work on it (started only a few weeks ago). 
> 
> But being twice as fast as ext3 is enough currently. It's even faster than 
> xfs what I use. Sometimes I'm quite suprised why something is so fast when
> I realize that it was actually ntfs-3g :-)

(-:

> > > KNOWN ISSUES, TROUBLESHOOTING
> > > =============================
> > > 
> > > Problem:    "No such file or directory" message when accessing some files:
> > > Answer:     This is a known problem in libntfs inode attribute 
> > > enumeration 
> > >             when the same type of attributes are distributed across
> > >             multiply mft records. Occures only very rarely.
> > > Workaround: If the file is a hard link (usually yes) then first delete
> > >             some other files hard linking to the same file then you'll
> > >             be able to delete the problematic file as well.
> > > Status:     High priority work.
> > 
> > Interesting.  This is the first time I hear of this problem.  Why did you 
> > not report it?  Do you have an example where this happens?
> 
> I identified this problem before yesterday. Sometimes I can reproduce it by
> 
>       cp -a /usr ntfs_volume
>       rm -rf ntfs_volume/usr
> 
> Most ntfs volumes have also such cases usually, so if you rm -rf / then you 
> can find them easily. 

Weird.  I wonder if it is something libntfs specific and not in the 
kernel as I have never seen this problem...

> > > Problem:    Why doesn't the driver work on 64-bit and bigendian systems?
> > > Answer:     We have no resource for that. Neither hardware, nor workforce.
> > > Status:     Low priority.
> > 
> > You do not need hardware.  If the little endian version works, then you 
> > can just run sparse over appropriately annotated code and it will tell you 
> > if you have screwed up somewhere.  
> 
> Sorry, if code can't be tested and validate in real then that's broken.

It is not, it's life.  I certainly have no interest of having 20 computers 
at home so I can test my code on all architectures.  I don't have the time 
to do that either.  That's what users are for.  (-;  Anyway all code can 
be validated by simply reading it and making sure it is logically correct.  
(There is of course a requirement for good concentration abilities and a 
lot of caffeine...)

> > Workforce: fair enough but in this state it cannot be added to ntfsprogs 
> > because it would break ntfsprogs on all but i386/x86_64 which is not 
> > acceptable.  
> 
> Sure, but the libntfs fixes must be applied. There were quite a few. 
> Actually finding and fixing them took most of the time. They can be 
> figured out trivially by diff.

Again interesting...  I am way too busy (just moved house, my wife is 
pregnant with number 3, ...) so I will not be doing it for a while but 
when I have time I am looking forward to seeing that diff.

> > It is a shame.  I hope you or someone fixes this one day...  ntfsprogs 
> > are in use on big endian architectures by people today and I am not 
> > willing to see that all broken...
> 
> Sure, I wouldn't even have time to integrate it, and actually I think the 
> driver should be a separate package, so we don't have to wait months to 
> synchronize with everything and install would be also much easier. So, 
> should this be a separate subproject then? I'm also reviewing and planing
> major cleanups to libntfs-3g.

It would be a shame to have two versions of libntfs...  Very confusing for 
everyone not least for all the utilities...  (Or do you literally call it 
libntfs-3g?)  Perhaps you should compile libntfs-3g static into 
ntfsmount-3g so libntfs-3g only exists as a concept in the sources but not 
in installed binaries?

Best regards,

        Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642