Re: ZLIB 2.0 / Blead 25366 on VMS

Paul Marquess wrote:
> From: John E. Malmberg [mailto:wb8tyw@xxxxxxx]
>  
>
> > Paul Marquess wrote:
>  

> Sorry if I'm showing my ignorance of VMS, but does this mean that when you
> run Perl on VMS you will/do have the choice of running the C library under
> UNIX emulation mode or in VMS native mode? 

It is quite a bit more grey than that.  There a number of feature 
settings that will determine how UNIXish or VMSish that the C library 
will behave.  In most cases the defaults are for backward compatibility 
with existing applications.

> Abe, does your copy of Perl run in UNIX emulation mode?

I do not think it will run too well.  Too much of Perl is assuming that 
when $^O eq 'VMS' that it will always see a VMSish file specification.

I only have about 1/3 of the fixes checked into blead right now.

> If the problem is in my code I want to know about it and I'll certainly help
> out. If it's someplace else I'll still try to help if I can.

Thanks.

> > There are a number of places that the VMS specific code is failing
> > silently because of a bug, and this is not detected because there is
> > some fallback code that compensates.  But there are also consequences to
> > fixing those bugs as they expose others.
>
> Ugh!

Not to mention, in some cases the VMS specific code is apparently 
depending on UNIX code failing before it falls back to the VMS code, and 
as the C library or the underlying VMS support code gets fixed, some of 
these failures will stop automatically happening.  I have already seen a 
case of VMS specific code depending on access to '/dev/tty' failing, and 
in the future that may not fail.

> Although it would be good to have these tests running on VMS, I'm not sure
> if it is really necessary right now if you are going to be guaranteed to
> have Compress::Zlib available in the core. The code it is exercising will
> only kick in if Compress::Zlib is unavailable.

I am going to ignore that for now, but it shows that as I fix things in 
VMS, I have to make sure that tests are not inadvertently skipped simply 
because they are running on VMS.

> > > > ext/Compress/Zlib/t/04def.t  fails on test 102 and dies.
> > > > not ok 102 -   create IO::Gzip
> > > > #     Failed test ([-.ext.compress.zlib.t]04def.t at line 322)
> > > > Can't call method "fileno" on an undefined value at
> > > > [-.ext.compress.zlib.t]04def.t line 324.
> > > > # Looks like you planned 1781 tests but only ran 102.
> > > > # Looks like your test died just after 102.
> > > > %SYSTEM-F-ABORT, abort
>
> This code is writing to a file via a *FH style filehandle.

I do not have access to the system i am running the tests on at the 
moment, but did look at this a bit last night.  The *FH filehandle was 
opened, and then something tried
an operation to set a variable named $x.

That failed for a reason that I have not determined.  I was stepping 
through the code in the debugger, and suddenly was in sub destroy.

This leaves the method $x->fileno undefined as was in the subsequent 
error message.

There may or may not be useful information in the screen buffer of my 
terminal emulator.

> > ext/Compress/Zlib/t/05examples.t
> > not ok 21 -   exit status is 0
> > #     Failed test ([-.ext.compress.zlib.t]05examples.t at line 152)
> > #          got: '512'
> > #     expected: '0'
> > not ok 22 -   no stderr
> > #     Failed test ([-.ext.compress.zlib.t]05examples.t at line 153)
> > #          got: '71'
> > #     expected: '0'
> > ok 23 - gzcat
> > ok 24 -   exit status is 0
> > not ok 25 -   content is ok
> > #     Failed test ([-.ext.compress.zlib.t]05examples.t at line 158)
> > #          got: '%RMS-E-FNF, file not found
> > # '
> > #     expected: 'helloext/Compress/Zlib/t/06gzsetp.t
> > <snip>
> > # Looks like you failed 3 tests of 25.
>
>
> Again this code is using a filehandle of the form *FH. 

So I need to find out why this is failing.

> > On VMS, all filenames have a period to delimit where the name part ends
> > and the extension (also known as file type).  If there is a NULL type,
> > then the period still is returned.
> >
> > There is another special rule about specifying filenames with a NULL
> > type.  If a utility gets a file with out the type specified, it will
> > append a default type, unless the NULL type is explicitly specified by a
> > trailing period.
> >
> > But this behavior could be considered a bug in the VMS code because it
> > is leaving the trailing period in glob, but it is removing it with
> > readdir() on Perl.  Not consistent behavior.
>
>
> So is the trailing period issue something that will be fixed at your end, or
> do I need to rework my tests to always use filenames with extensions/file
> type?

Yes or no.  It is a bit grey.  And I am not sure how to fix it and keep 
everyone happy.

I think that you may need to always use a non-null file type.

This is the problem:

If the filenames are being presented in VMS mode, then the trailing dot 
should be present.  If they are presented in UNIX mode it should not be.

And also according to both the Perl (most routines) and VMS C library 
routines manual, if you supply a VMS format name as input, you should 
get a VMS format name on output, and if you put in a UNIX format path as 
input, you should get a UNIX format path as output.

In the default mode of the C Library, an ambiguous name is considered 
VMS.  In what I have been calling UNIX mode, it is treated as UNIX.
And coming soon is a POSIX mode that changes things a little bit more 
behind the scenes.

Now with the ODS-2 volumes, UNIX "foo." and VMS "foo." are aways the 
same file.

On ODS-5 volumes, UNIX "foo." is a VMS "foo^..", a significant difference.

So that makes it a problem of what to do with unixify("foo.").

Also in UNIX mode, routines that return paths with out a path being 
input will return them in UNIX mode.

Unfortunately there are some historical bugs.  I seem to remember at 
least one test scripts in Perl that depend on readdir() always returning 
filenames in UNIX mode even when the input directory is in VMS mode.

And some of the file manipulation modules are accepting VMS format names 
and returning UNIX format names.

And it gets even worse, but I will leave that for another topic.

The summary is that currently everything that is manipulating VMS format 
filenames is at least a little bit broken, and more broken if it is 
converting them to UNIX format, and even more broken if they are 
converting them back to VMS again.

Much of the fixes that I have been putting in, or are planning to put in 
are removing the conversion of UNIX names to/from VMS where it is 
absolutely not needed.

> For now, at least, I've changed all the tests that hit glob to make sure
> they have an extensions/file type.
>  
> > > > t/14gzopen......
> > > >
> > > > #     Failed test (t/14gzopen.t at line 421)
> > > >
> > > > #     Failed test (t/14gzopen.t at line 424)
> > > >
> > > > #     Failed test (t/14gzopen.t at line 425)
> > > > #  gzerrno 0
> > > > # Looks like you failed 3 tests of 186.
> > >
> > >
> > > OK, those failing tests are checking that gzopen can detect a non-
> > > writable file. At the moment I'm creating one with this
> > >
> > >    my $name = "test.gz" ;
> > >    ...
> > >    writeFile($name, "abc");
> > >    chmod 0, $name ;
> > >
> > > I imagine the "chmod 0, $name" is the non-portable part that I'm
> > > tripping over.
> >
> > > Can you try changing line 419 of t/14gzopen.t to this please and report
> > > if that makes any difference?
> > >
> > >         chmod 0444, $name ;
> >
> > Unless a UNIX compatibility logical is set, the protection masks have
> > different meanings on VMS.  That feature is also only available on newer
> > versions of VMS.
>
> There are other instances of testing non-writable files in the test harness
> that seem to be passing -- they use "chmod 0444, $name". I'll update my test
> harness.

There may be an Access Control List on the directory that is also 
overriding the chmod setting.  If an ACL permits write access, chmod can 
not deny it.  I do not know if VMS ACLs can be manipulated by Perl 
scripts directly.

I probably have an ACL on my build directory that always grants me 
delete access to a file.

> > > > t/16top.........
> > > > glob failed (can't start child: invalid argument) at /P5M/
> > > > COMPRESS-ZLIB-2_000_01
> > > > /blib/lib/Compress/Zlib/Common.pm line 232.
> > > > legal or die??? at /P5M/COMPRESS-ZLIB-2_000_01/blib/lib/Compress/Zlib/
> > > > Common.pm
> > > > line 237.
> > >
> > >
> > > That is tripping over file globbing. The test is failing when I try to
> > > execute this
> > >
> > >         glob "in[1234]"
> > >
> > > and there are 4 matching files "in1", "in2", "in3" and "in4".
> > >
> > > Hmm, just had a read at perlvms.pod  -- is VMS complaining because
> >
> > "[1234]"
> >
> > > looks like a directory or does it not support '[1234]' as a wildcard
> > > character class?
> >
> > Most likely.  Currently in most places that Perl is trying to determine
> > if a filename is in UNIX format or VMS, it is only looking for
> > m{[]:<>;}.  The caret character should also be added to that mix as that
> > is used on escape extended characters.
> >
> > The other test that is used to determine if a file is UNIX is to see if
> > it contains a '/' character.
> >
> > Also glob on VMS is done in C code, not by passing the glob through to
> > the shell.
>
>
> I realise that VMS perl is trying to steer a middle course and have glob
> work both with Unix filenames and VMS native filenames, but the parameter to
> glob isn't a filename - it's a cut down regular expression. I assume thought
> with the "[]" characters you are faced with a major culture clash -- it
> can't be both a VMS directory and a character class at the same time.

Yep.  With ODS-5 it is even worse. "/" is a legal character in a VMS 
format file specification.

Some rules need to be documented and enforced on how path names are to 
be interpreted.  If both UNIX and VMS path names are to be mixed, then 
some rule is needed to decide the ambiguous cases.

The fun is doing it with out breaking any existing Perl scripts.

> Assuming character classes are supported, is it possible to either make VMS
> think that the glob is a Unix filename or to escape the [] characters so
> that they will be used as character class delimiters?
>
> For example, if prefix the globs with "./" like this
>
>     glob "./in[1234]"
>
> will it go into VMS mode or Unix mode?

I do not know with out trying it, or code inspection.  It depends on 
what particular check that the code is making.  If it is checking for a 
"/" then it will go into UNIX mode.

> I'll try to get an updated version of my module out tonight or tomorrow with
> all of the fixes applied.

Please let me know what files you change and where I should put them in 
blead because that is where I am set up to test them.

ZLIB is now the only test failing that I do not know a potential 
solution for, and what I want to do next is activate both the "case 
preserved" and "case sensitive" modes in vms.c.

Of course as I progress, it may require running the test suite on VMS 
multiple times for the different options available.

-John
wb8tyw@xxxxxxx
Personal Opinion Only