Re: SPF test suite - volunteers needed

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> Tasks:
>
> 1) Decide text file format.  Suggestions were XAML, JSON, CSV.  Bind is
> not an option because it is tricky to parse and the point is *not* to
> use a real DNS server.  My current vote is JSON - but then I don't know
> what XAML is or looks like.

s/XAML/YAML/.  http://www.yaml.org/refcard.html describes YAML in itself as 
an example.

> 2) Write script to extract DNS records referenced by current suite to
> text database.  It is a one-off script, so we don't care what language
> you use - we just want the result.  It may need to run multiple times
> and/or merge results to handle TEMP errors in the DNS servers holding
> the data.

What DNS zone are you talking about?

BTW, Shevek offered to provide a lot of test records in the "spftest.org." 
zone (as well as delegating the zone to a name server of our choice in 
case we want to host part of the test suite in DNS online).

The "spf1-test.mailzone.com." zone used by the Mail::SPF::Query test 
script[1] is far from comprehensive and in part outright archaic.

> 3) Write a driver for your favorite SPF implementation (I will do
> pyspf). Run the suite, and review failures to determine whether the test
> suite or your implementation needs updating.  Submit test suite patches
> to me.  Post tricky cases to spf-discuss.  Suggest changes to the test
> suite syntax that would make driving your implementation easier.

I think we should set up an issue tracker for the test suite mini-project.  
Koen offered to set up additional queues for SPF-related tasks on his RT 
installation, which already runs the contact form support queue.  
Comments?  Should such an issue tracker be publicly readable?   (I think 
it should.)

I'll care for Mail::SPF, which needs to be developed a bit further from its 
current state (and documented!) anyway.

I think we should define a coverage strategy for the test suite:

  * We should create at least one positive and one negative test for each
    and every RFC 2119 keyword ("MUST", "MAY", etc.) in RFC 4408 that is
    relevant to SPF implementations (some aren't).  Thus we would have to
    create an index of the spec's authoritative statements to determine
    which are relevant and how to best test them.

  * Besides, as Wayne did in his test suite, we should test for common
    mistakes and tricky corner cases.  We should look at the existing test
    data to see what's already there.

  * Should we create tests for "strict vs. lax" implementation behavior?
    Those tests could be used in special modes (i.e. "strict"/"lax") of the
    test suite.

Comments and additions?

References:
 1. http://search.cpan.org/src/JMEHNLE/Mail-SPF-Query-1.999.1/t/test.dat

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEcilBwL7PKlBZWjsRAlijAJ9kSgrn0JpLYgoxAvxy21FYuDCKTACfbGcH
DNARZz6NMC9Q6BusiW7Sxw8=
=5uAa
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your 
subscription, 
please go to http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx