Re: [announcement] ntfs-3g: open source read-write driver

Hi,

On Fri, 14 Jul 2006, Anton Altaparmakov wrote:
> On Fri, 14 Jul 2006, Szakacsits Szabolcs wrote:
> > As part of the Linux-NTFS project, I'm happy to announce my contribution
> > to ntfsmount and libntfs which resulted ntfs-3g, a read-write ntfs driver,
> > capable for unlimited file creation and deletion.
> 
> Wow, well done!  Although "unlimited" is a bit of an exagerration as you 
> yourself admit below.

Practically the limits the ones I documented, they occure quite rarely.
 
> I had a quick look at the code and it chickens out of a lot of corner 
> cases which are quite hard...

There are four such corner cases and they are not so hard. I have code 
for some of them but there was no time to massively test so they aren't 
included.
 
> Also I may be wrong as I did not spend enough time on it but if you run 
> out of space on the partition in the middle of a complex add or delete 
> operation you appear to leave a corrupt directory.  In general the error 
> handling appears to leave things corrupt and does not attempt to roll back 
> / correct things.  Is this a fair description?

Partly. Thanks, I forgot to document this. When it was possible I tried to 
make the changes that things stay consistent whenever the free space runs 
out or whatever error happens. Chkdsk must be always able to fix such 
inconsistencies. I was thinking to refuse serving requests when free space 
was getting low but currently the MFT extension limit fills this role 
usually.

Please also note that "full disk" handling is not so simple as just rolling 
back things because lower layers can report success when later they can 
fail which point you already released all the needed in-memory info which 
would be needed to rollback the state. It's not only a simply rollback 
problem.
 
> And of course you use recursion which you cannot use in kernel context so 
> the code could never be moved to a kernel driver (even ignoring the fact 
> that it does not do locking or 

Currently I'm not interested in the kernel driver. It's a lost case for 
over a decade. Full read-write could be done in user space pretty fast and 
I can't see drawbacks, only benefits:

        - NTFS is huge and complex, not for the kernel. Crash in kernel 
          (hw error, corrupt ntfs, etc) and game is over. Crash in user
          space then just restart the service.

        - kernel has a lot of limitations, restrictions which are all gone.

        - fedora/redhat users have never ending hassles with installing
          the driver. Instead they could install ntfs-3g once and forget 
          the issue forever.

> endianness handling).

That's planned in a later phase as part of a major cleanup by abstracting 
access to metadata.
 
> Wow!  Enjoy it!  That sounds like an amazing experience!  I hope you bring 
> back some nice photos for us all to wonder at!  (-;

Thanks :-) I planned video, we'll see!

> > Version  1.03    ------Sequential Create------ --------Random Create--------
> >                  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> >            files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> > reiserfs     16k 21459  99 +++++ +++ 17856  96 20172  98 +++++ +++ 16414  96
> > jfs          16k  7015  13 +++++ +++  5868  10  3068  14 +++++ +++  1075   3
> > ntfs-3g      16k  3021  99 14291  99  5226  99  3548  99 16149  99  5223  99
> > xfs          16k  2401  17 +++++ +++  2095  15  2301  20 +++++ +++   347   2
> > ext3         16k  1862  96 +++++ +++ +++++ +++  1914  96 +++++ +++  9695  98
> > minix        16k  1450  97 +++++ +++ 18148  94  1694  97 +++++ +++  4847  98
> > fat32        16k   366  97 +++++ +++  1809  97   428  97 +++++ +++  1361  97
> > paragon ntfs 16k    58  98  1259  99   245  99    55  99 +++++ +++   832  99
> > captive ntfs    Always crashes early on file creations and loses files.
> >             This was also confirmed and fix denied by its developer.
> > 
> > We estimate the ntfs-3g driver performance could be increased by several 
> > times.
> 
> Wow, nice looking numbers.  I haven't done anything like that for my code.  
> I wonder how it would compare 

I wonder, if it would actually work ;) It's also a very good function test 
for the directory operation. For example I had problems with FAT32 failing 
sometimes and Captive always crashing after creation of 60-130 files.

> given it is in the kernel and IMHO a lot better optimized than your 
> version...  E.g. I use binary search where you use sequential and I never 
> have to repeat a search twice during a single operation as that is all 
> cached the first time although I admit that the cache overhead may 
> introduce a delay in itself so the benefit in my code may not be as much 
> as one would expect initially...

As I wrote, functionality comes first, especially as I don't have more than 
max a few hours in the nights to work on it (started only a few weeks ago). 

But being twice as fast as ext3 is enough currently. It's even faster than 
xfs what I use. Sometimes I'm quite suprised why something is so fast when
I realize that it was actually ntfs-3g :-)

> > KNOWN ISSUES, TROUBLESHOOTING
> > =============================
> > 
> > Problem:    "No such file or directory" message when accessing some files:
> > Answer:     This is a known problem in libntfs inode attribute enumeration 
> >             when the same type of attributes are distributed across
> >             multiply mft records. Occures only very rarely.
> > Workaround: If the file is a hard link (usually yes) then first delete
> >             some other files hard linking to the same file then you'll
> >             be able to delete the problematic file as well.
> > Status:     High priority work.
> 
> Interesting.  This is the first time I hear of this problem.  Why did you 
> not report it?  Do you have an example where this happens?

I identified this problem before yesterday. Sometimes I can reproduce it by

        cp -a /usr ntfs_volume
        rm -rf ntfs_volume/usr

Most ntfs volumes have also such cases usually, so if you rm -rf / then you 
can find them easily. 
 
> > Problem:    Why doesn't the driver work on 64-bit and bigendian systems?
> > Answer:     We have no resource for that. Neither hardware, nor workforce.
> > Status:     Low priority.
> 
> You do not need hardware.  If the little endian version works, then you 
> can just run sparse over appropriately annotated code and it will tell you 
> if you have screwed up somewhere.  

Sorry, if code can't be tested and validate in real then that's broken.

> Workforce: fair enough but in this state it cannot be added to ntfsprogs 
> because it would break ntfsprogs on all but i386/x86_64 which is not 
> acceptable.  

Sure, but the libntfs fixes must be applied. There were quite a few. 
Actually finding and fixing them took most of the time. They can be 
figured out trivially by diff.

> It is a shame.  I hope you or someone fixes this one day...  ntfsprogs 
> are in use on big endian architectures by people today and I am not 
> willing to see that all broken...

Sure, I wouldn't even have time to integrate it, and actually I think the 
driver should be a separate package, so we don't have to wait months to 
synchronize with everything and install would be also much easier. So, 
should this be a separate subproject then? I'm also reviewing and planing
major cleanups to libntfs-3g.

Best regards,
                Szaka


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642