Re: [Puppet-dev] [Puppet-users] State of Development

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[cross-posting/moving to -dev]

On Friday 26 October 2007, Luke Kanies wrote:
> On Oct 26, 2007, at 2:27 AM, David Schmitt wrote:
> > That's good to hear!  My regular tests are already running again on
> >
> > http://puppettests.black.co.at/
> >
> > The testresults are put into a directory named after the time the
> > test ran.
> > I run the following tests:
> >
> > * rake test, output in /rake_tests
>
> You're definitely getting failures I'm not getting, at least on Debian.
>
> I've only got one failure on Debian and none on OS X; I'm about to
> test my other OSes.

Hmm. This is running on a pretty clean etch VServer, with only my default 
configs applied. Actually, its configuration comes from my puppet repo:

http://git.black.co.at/?p=manifests.git;a=blob;f=manifests/site.pp;h=d7a265ea397b98b4a9e57de9d960ce4d6ccaf8cd;hb=HEAD#l397

 397 node 'git-test.black.co.at' {
 398 
 399         $mta = ssmtp
 400         $nagios_parent = "ic.black.co.at"
 401 
 402         include dbp
 403         include puppet::test_from_git::on_debian
 404 
 405 }


The class on #403 coming from the module at 
http://git.black.co.at/?p=manifests.git;a=tree;f=modules/puppet;hb=HEAD

I would discourage its use on valueable installations though, it's pretty 
rough to all things ruby and puppet when running the tests as well as the 
rake tests which are not very restrained either.

Which already brings me to my first problem when testing from git: I get 
conflicts with the system's puppet version:

| root@git-test:/tests/puppet-trunk/test# rake test
| (in /tests/puppet-trunk/test)
| [...]
| Could not
| autoload "/usr/lib/ruby/1.8/puppet/parser/ast/resourceoverride.rb":
| superclass mismatch for class ResourceOverride  

The testscript thus removes before puppet for the time of the testing.


The script runs from cron which cleans most of the environment. running rake 
test from a shell fixed the cron failure: 

./ral/types/cron.rb:467:in `test_default_user' ENV["USER"] is used, which is 
empty under cron. I used the Etc.getpwuid function instead.

./ral/providers/service.rb uses "hddtemp" as example service, which is not 
installed on this host. I changed these tests to use "sysklogd with a 
pattern.

./ral/types/mount.rb assumes that /etc/fstab contains some mounts. Since 
mounts cannot be changed from within a vserver, it'd make sense to not run 
these tests at all on this host. 

One way to check this would be a non-zero XID:
root@git-test:/tests/puppet-trunk# cat /proc/$$/vinfo 
XID:    5

I could just subtract 3 from the error count too ;)


./executables/puppetd.rb is missing a --no-daemonize

With the changes as published in 
http://git.black.co.at/?p=puppet-testsuite-fixes;a=summary or 
git://git.black.co.at/puppet-testsuite-fixes the failure count goes down to:

940 tests, 10240 assertions, 3 failures, 0 errors

two of which were mount errors and the third:

test_period_by_number(TestSchedule)
    [./ral/types/schedule.rb:218:in `test_period_by_number'
     /tests/puppet-trunk/test/lib/mocha/test_case_adapter.rb:19:in `__send__'
     /tests/puppet-trunk/test/lib/mocha/test_case_adapter.rb:19:in `run']:
matched minus an hour.
<false> is not true.

which happens probably because local time is 00:40.


> > * puppetmaster + puppetd on a empty site.pp. config, output and vardir
> > in /minimal
> >
> > * puppetmaster + puppetd on my full configuration. config, output
> > and vardir
> > in /trunk
>
> Can we work at getting these tests, at least the minimal one, into
> the test framework?  Otherwise it's difficult for me to work at
> reproducing them.

I believe the minimal one is a subset of executable/puppetd.rb only run from 
the command line and implemented in shell.

> > These tests are currently run 06:30 UTC+2, which should be around
> > the end of
> > your workday? Please tell me if you'd like them run more often or at a
> > different time.
>
> The time is fine.  Now we just need a script that emails failures to
> puppet-dev (preferably only if the failures are different than the
> day before).  Care to take responsibility for that?

Sure. There is currently a full testrun in the works, so I get the output with 
my fixes to the website. I'll use that as a testcase. For starters I'll just 
use a plain diff and we can work on from there.

> > I even whipped up a little munin plugin which tracks rake test
> > results.
> > http://www.edv-bus.at/munin/black.co.at/git-test.black.co.at-
> > puppettests.html
>
> That's pretty darn slick.

Yeah, munin really rules for this kind of stuff. See 
http://git.black.co.at/?p=manifests.git;a=blob;f=modules/puppet/files/munin.puppettest
 
for the plugin's code.


> We really need to set up some kind of autobuild system, preferably
> backed by multiple OS instances.  Maybe something on EC2, with a
> community member taking responsible for each instance?  E.g., have a
> Solaris image maintained by someone familiar with it, same for
> Debian, Ubuntu, etc.
>
> Anyone want to volunteer to head up this effort?  Anyone know much
> about autobuild systems, especially those that will generate nice
> graphs like this?

'cuse me, what do you mean by autobuilding? Which functionality/report are you 
missing? What additional metrics would you like to see? I could try for the 
runtime of the tests, but that will have a high variance, since there are 
other services on this machine too ...

Just for fun I added lines of code for lib/ and test/ subdirectories to the 
munin plugin.

> > As you said, the git repo currently is a bit off track. Also the
> > tests are run
> > within a vserver container, so some tests (e.g. the mount stuff)
> > will always
> > fail due to missing permissions.
>
> We need to get the tests to the point where they always pass on all
> systems.  If you've got tests that never pass, then the tests need to
> be fixed so they pass or they need to be changed so they aren't run.
>
> Are you in a position to take a crack at your consistently-failing
> tests?

The only two still failing due to external reasons are the mount tests, and 
I'm not really sure how they should be handled. Please advise!

Regards, David
- -- 
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHInj2/Pp1N6Uzh0URAhFdAJ9fn6yzge2CHBs+4cQB2maMl+bIywCgobJH
+f5VPqSXPhd7vTA2H1J0ces=
=776p
-----END PGP SIGNATURE-----