Re: making v6 test suite its own distro

At 6:53 PM +0300 7/10/06, Gaal Yahas wrote:
> On Sat, Jul 08, 2006 at 12:47:17AM -0700, Darren Duncan wrote:
> >  As briefly discussed on #perl6 ...
>
> As briefly replied there before being jethandled...

As further discussed there ...

> Perhaps we need a baby-Perl Test::Simple for new implementations, and
> by convention have 01-sanity use only those functions present there. The
> drawback being somewhat duplicated effort and another distribution worry.

I believe that the current Test.pm already qualifies as a baby-Perl 
implementation, as overall its functionality is quite simple, and it 
uses very little of the language (especially since its use of 
junctions was removed a few weeks ago).

Have a look at 01-sanity again.  Since I moved 08-test out of that 
folder (and into 02-test-pm) none of those 01-sanity tests use 
Test.pm at all.

My impression of 01-sanity is that it checks that the Perl 6 
implementation has the basic language functions that Test.pm has as a 
prerequisite, such as being able to print, knowing basic 
conditionals, supporting basic subroutines and modules.

If 01-sanity runs, then the normal Test.pm should be useable, and 
02-test-pm checks that Test.pm itself actually works.  When that 
runs, then the other tests can be performed as usual.

> The first obvious question that this enumeration presents is: where to
> cut? Does every implementation use its own Test.pm? Pugs was able to
> start using one after 23 days, but that was a simpler module then than
> it is today, and a fresh implementation may find it hard to get started
> if its first objective is to implement so much Perl 6 simultaneously.

I would say for starters that anything which is written in Perl 6 
will be shared by all implementations, and that includes Test.pm.

Considering its few needs, any Perl that can't run the current 
Test.pm is hardly useful, so they would be getting up to the level of 
supporting Test.pm very quickly.  And in the process of getting to 
that level, the 01-sanity tests can be tried without Test.pm.

> The other question that comes to mind is how to manage SKIP and TODO
> tests for release mangement. Currently Pugs, assuming globality, simply
> plunges into every t/ file before a release and marks forsaken (for this
> release) tests TODO, and commits. But surely different implementations
> have their different bugs. In the case of TODO we can fix that easily:
> we already have a mechanism for force_todoing tests "at a distance",
> though currently that distance is at the head of the test file (but it's
> a simple matter to make that a distro-level file). The deeper technical
> challenge is how to maintain SKIPs, which after all, are often used
> before hanging tests. skip_all we can manage; skip we cannot, until at
> least when we have a powerful debugger.

All unconditional skips/todos in the test suite should be eliminated 
entirely, and be allowed to fail.  If hangs are a concern, then the 
test harness (presumably not written in Perl 6) can just have a 
timeout function and kill any tests that run for too long (the time 
threshold can be changed per user in config.yml).

The proper way to have skips/todos is for them to be conditional.

Ideally, one of the first things that any Perl 6 implementation would 
provide is something akin to a %FEATURES (or some other named) global 
or environmental or config variable, which a Perl 6 script can query 
at runtime to see whether that implementation CLAIMS to support 
certain features.  Only features that are claimed to be supported are 
tested, to yield a ok/not ok, and only those features claimed to be 
unsupported, either explicitly, or implicity by not being mentioned 
at all, will be skipped.

A brand new Perl 6 implementation will declare an empty %FEATURES, 
meaning that everything is unsupported, and so everything conditional 
skips (which would be most tests, ideally).  As the implementation 
matures, it starts adding items to %FEATURES, and so skips are 
replaced by passes or fails.

Of course, certain sanity items will have to work before %FEATURES 
does, such as the ability to print, basic conditionals and boolean 
expressions, the ability to read from a variable, etc.  But no 
support for subroutines or external modules will be needed in order 
to do this, so the threshold is lower than for any type of Test.pm.

The %FEATURES variable would be populated in the logical fashion, 
derived partly or entirely from explicit declarations in the 
implementation code itself, and partly or not at all from user 
configuration or environment as is applicable.  Entirely from code 
may be better so that this doesn't make the CONFIG or ENV or whatever 
vars redundant, or they could possibly overlap as makes sense, but my 
emphasis is on the code.

If we are going to go with this idea, and I give it my highest level 
of recommendation, it would stand to reason that the possible 
features to be tested for will be standardized and documented such as 
in a Synopsis.

I could start enumerating such a list if no one else wants to, but it 
would be long and would stand for a thought out design, like the 
tests themselves.

More broadly speaking, I think that a revisiting of the test 
hierarchy with a goal to it being able to scale better (see TASKS of 
Pugs), and a capabilities/features list could be done somewhat hand 
in hand.  Features that tend to be implemented together can also be 
grouped together for testing, and tests could be grouped generally in 
dependency order, such that if feature A needs to be implemented 
before feature B can be implemented, feature A will be tested first. 
A trivial case of such organizing has started with 01-sanity and 
02-test-pm being checked first, but that's as far as it goes so far.

> As a Pugs hacker I can say that the immediacy of access to t/ is a
> great benefit, and I would hate to lose that. But what's the best way
> of granting that benefit to other implementations without crippling it
> for anyone?

From the point of view of a *user* of Perl 6, one that gets it from 
CPAN, getting this to work is just a matter of packing the tests in 
such a way that they can be "installed" some how and be visible to 
the packages for Pugs or v6 or whatever.

This may require something unusual, because afaik, only "modules" 
typically get installed and become visible to other packages, while 
test scripts don't.

(I get around this perceived limitation with Rosetta by implementing 
all the backend-shared tests as Rosetta::Validator::* modules, which 
install with the framework and can be invoked like a module.  But I 
don't know if something like that is feasible for a general Perl 6 
test, so something else may be required.)

As for hackers working from version control, where immediacy of 
access to and seamless integration of updates of otherwise separately 
distributed components is important, I don't know exactly what to 
suggest as a solution but perhaps something that lets you checkout 
multiple SVN into a shared space where the pieces can see each other 
as if they were one big distro, and you can just test how the parts 
work together naturally.  I don't know how to implement this, though.

> As you can see, there are possibly wider and wider scopes of problems to
> solve here and we can certainly choose not to attack every problem now.
> But yes, let's get the discussion going (thanks, Darren, for starting
> this thread) and come up with a plan, so that we can be ambitious and
> realistic together.

I think it is very important to share whatever resources can be 
shared, for easier development and better quality results.  If this 
causes a few hiccups short term, like test fails rather than skips, 
or all tests temporarily skipping, I think it is still worth it.

-- Darren Duncan