Re: [announcement] ntfs-3g: open source read-write driver

Hi Szaka, hi Anton,

[Sorry for late reply, hope you will read it before you will go to the
trip.]

First of all, great work! Cool that you found time to implement full
index operations support.

On Sat, 15 Jul 2006 at 01:40 +0300, Szakacsits Szabolcs wrote: 
> On Fri, 14 Jul 2006, Anton Altaparmakov wrote:
> > I had a quick look at the code and it chickens out of a lot of corner 
> > cases which are quite hard...
> 
> There are four such corner cases and they are not so hard. I have code 
> for some of them but there was no time to massively test so they aren't 
> included.

Can you list them, please?

> > And of course you use recursion which you cannot use in kernel context so 
> > the code could never be moved to a kernel driver (even ignoring the fact 
> > that it does not do locking or 
> 
> Currently I'm not interested in the kernel driver. It's a lost case for 
> over a decade. Full read-write could be done in user space pretty fast and 
> I can't see drawbacks, only benefits:
> 
>       - NTFS is huge and complex, not for the kernel. Crash in kernel 
>           (hw error, corrupt ntfs, etc) and game is over. Crash in user
>         space then just restart the service.
> 
>       - kernel has a lot of limitations, restrictions which are all gone.
> 
>       - fedora/redhat users have never ending hassles with installing
>         the driver. Instead they could install ntfs-3g once and forget 
>           the issue forever.

I can add support of FreeBSD (and all OSes on which FUSE will be ported,
maybe one day we will be able to use ntfsmount even under windows ;-)

> > Wow!  Enjoy it!  That sounds like an amazing experience!  I hope you bring 
> > back some nice photos for us all to wonder at!  (-;
> 
> Thanks :-) I planned video, we'll see!

Have a nice time! BTW, give us link when video will ready.
(BTW, maybe all developers can add photos/video to their wiki pages?)

> > > Version  1.03    ------Sequential Create------ --------Random 
> > > Create--------
> > >                  -Create-- --Read--- -Delete-- -Create-- --Read--- 
> > > -Delete--
> > >            files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
> > > %CP
> > > reiserfs     16k 21459  99 +++++ +++ 17856  96 20172  98 +++++ +++ 16414  
> > > 96
> > > jfs          16k  7015  13 +++++ +++  5868  10  3068  14 +++++ +++  1075  
> > >  3
> > > ntfs-3g      16k  3021  99 14291  99  5226  99  3548  99 16149  99  5223  
> > > 99
> > > xfs          16k  2401  17 +++++ +++  2095  15  2301  20 +++++ +++   347  
> > >  2
> > > ext3         16k  1862  96 +++++ +++ +++++ +++  1914  96 +++++ +++  9695  
> > > 98
> > > minix        16k  1450  97 +++++ +++ 18148  94  1694  97 +++++ +++  4847  
> > > 98
> > > fat32        16k   366  97 +++++ +++  1809  97   428  97 +++++ +++  1361  
> > > 97
> > > paragon ntfs 16k    58  98  1259  99   245  99    55  99 +++++ +++   832  
> > > 99
> > > captive ntfs    Always crashes early on file creations and loses files.
> > >           This was also confirmed and fix denied by its developer.
> > > 
> > > We estimate the ntfs-3g driver performance could be increased by several 
> > > times.
> > 
> > Wow, nice looking numbers.  I haven't done anything like that for my code.  
> > I wonder how it would compare 

Agree, very nice. BTW, what optimisations you did except noatime option
by default?

> > > KNOWN ISSUES, TROUBLESHOOTING
> > > =============================
> > > 
> > > Problem:    "No such file or directory" message when accessing some files:
> > > Answer:     This is a known problem in libntfs inode attribute 
> > > enumeration 
> > >             when the same type of attributes are distributed across
> > >             multiply mft records. Occures only very rarely.
> > > Workaround: If the file is a hard link (usually yes) then first delete
> > >             some other files hard linking to the same file then you'll
> > >             be able to delete the problematic file as well.
> > > Status:     High priority work.
> > 
> > Interesting.  This is the first time I hear of this problem.  Why did you 
> > not report it?  Do you have an example where this happens?
> 
> I identified this problem before yesterday. Sometimes I can reproduce it by
> 
>       cp -a /usr ntfs_volume
>       rm -rf ntfs_volume/usr
> 
> Most ntfs volumes have also such cases usually, so if you rm -rf / then you 
> can find them easily. 

I reproduced this. Metadata that illustrates this issue:
http://data.linux-ntfs.org/attr_lookup_bug.img.bz2

Anton, it seems problem in your code (AFAICS in ntfs_external_attr_find)
Just run "ntfsinfo attr_lookup_bug.img -F a/b/russecurity -v". There
will be 2 attributes listed in attribute list, but only one of them will
printed later by ntfsinfo.
 
> > It is a shame.  I hope you or someone fixes this one day...  ntfsprogs 
> > are in use on big endian architectures by people today and I am not 
> > willing to see that all broken...
> 
> Sure, I wouldn't even have time to integrate it, and actually I think the 
> driver should be a separate package, so we don't have to wait months to 
> synchronize with everything and install would be also much easier. So, 
> should this be a separate subproject then? I'm also reviewing and planing
> major cleanups to libntfs-3g.

I do not like idea of sub-project and rename. I do not see any benefits,
only drawbacks:
      * It will be hell to merge every time changes in libntfs and
        libntfs-3g.
      * ntfsmount already has it name. It was difficult to explain
        difference between it and kernel driver. This will only start
        mess once again.
      * Merging will give changes split on smaller patches that are
        easier for review. (BTW, some of your changes not directly
        needed for index operations are arguable. Dad that you posted
        all as full source, not as patches serial. I have 180kb diff of
        your changes against current libntfs)

My current plan to merge your code and at least fix endianess problem.
Will see if I will have time for more.

> Problem:    Error message when mounting:
>             fusermount: failed to open /dev/fuse: No such file or directory
> Answer:     The FUSE kernel module isn't loaded. 
> Workaround: Run 'modprobe fuse' as root.
> Status:     High priority work.

This is FUSE problem. Miklos should fix it.

> Problem:    "Directory not empty" message when deleting "empty" directories: 
> Answer:     If locale isn't set correctly then glibc can't convert
>             some filenames, so they aren't visible and the directory 
>             indeed isn't empty. Occures very rarely.
> Workaround: Set your locale properly according to the used characters
> Status:     High priority work.

What can we do about this?

> Problem:    Windows chkdsk complains about free space being marked in use.
> Answer:     chkdsk sometimes optimizes the NTFS layout and later finds
>             its own bugs (e.g. when index root attributes are moved from 
>             extent mft records to the base one). This is not a problem 
>             with ntfs-3g.
> Workaround: No need, it's a chkdsk bug and it fixes itself.
> Status:     Low priority work: we will optimize our algorithms too.

Do you have full list (or close to full) when such things happens?

> Problem:    Windows chkdsk may report minor inconsistency.
> Answer:     The allocation size of sparse files isn't set always correctly.
> Workaround: No need.
> Status:     Normal priority work.

I thought that I fixed all problems with sparse files. How can I
reproduce this?

> Problem:    Why doesn't the driver work on 64-bit and bigendian systems?
> Answer:     We have no resource for that. Neither hardware, nor workforce.
> Status:     Low priority.

It work on 64-bit systems. Guy from local irc network tested it on
amd64. Also see ml.

-- 
Cheers,
        Yura



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642