Subject: Re: Test Suite Zone Data - msg#00089

Previous Message by Date: click to view message preview

Re: Test Suite Zone Data

> The new test suite will foremost and above all be an offline test suite, > the rationale being that often software, when being built and tested > automatically (such as by the Debian project's package builders system), > has no access to the Internet for practical and/or security reasons. IOW, > SPF implementations should be able to be tested offline. > Wow, knock me over with a feather. I would think that would not be the exception, not the rule. Was this a shift along the way? I mean, why the effort to update the dns for mailzone.com and spftest.org? > You are not supposed to enter the zone data from the test suite into real > DNS. Any implementation should use a so-called adapter or test driver to > read the zonedata data from the test suite file instead of from real DNS. Again, I'm just so very surprised. So, every single implementation has to construct their own dns driver just to implement the unit tests? This just requires _so much more code_ from everyone to create a test arena. Moreover, it also make the testing process more fallible. DNS already works, and we've already written code to use it to support the framework. If someone/something doesn't have access to the Internet for an automated test suite, then let them install a local dns server for their test suite. I strongly urge that you guys rethink that one. Make it the exception, not the rule. It just seems like I'm being punished or something: the test suite is my cross to bear, and I have to carry it with me inside the framework libary code too ;-) Please do not impose this on the implementations. It should be setup for the implementations to use already -- it's part of the environment that should already be in place. Each implementation shouldn't have to construct it's own environment, unless they need to for their own reasons. Imposing this will be a turnoff for developers on future implementations. Respectfully submitted, Monte Hansen "Julian Mehnle" <julian@xxxxxxxxxx> wrote in message news:200607300825.53749.julian@xxxxxxxxxxxxx > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Monte Hansen wrote: >> I've tried to catch up on all the recent posts and didn't see any mention >> about anything related to the zone data with the new test suite. For >> instance, with the various live tests, I could run them without setup >> since they worked against live dns records in the mailzone.com tld. Not >> so in the new suite, it appears (example.com). >> >> I'm not clear what the plan is, but wouldn't it make sense to use that >> model. This way _each_ tester isn't required to stage an environment in >> accordance with every "zonedata:" test case from the yaml records. It >> would only need to be setup once for everyone to receive the benefit of >> it, and aligns with the Development Roadmap 1 c. > > The new test suite will foremost and above all be an offline test suite, > the rationale being that often software, when being built and tested > automatically (such as by the Debian project's package builders system), > has no access to the Internet for practical and/or security reasons. IOW, > SPF implementations should be able to be tested offline. > > You are not supposed to enter the zone data from the test suite into real > DNS. Any implementation should use a so-called adapter or test driver to > read the zonedata data from the test suite file instead of from real DNS. > > However, we do plan on mirroring the test data in real DNS once a solid > release of the offline test suite has been finished and we thus no longer > need to concentrate our efforts on getting the test suite done ASAP. Then > implementations can test themselves using the live DNS if they want > (althouth that will not be the preferred modus operandi of testing.) > >> That said, has anyone ported the live test cases to the new yaml layout? > > No, not yet. Some of the old tests will be ported, some will be > considered > obsolete. Also, some new tests will be developed. > > Everyone who wants to contribute valuable tests is invited to do so. > Please post tests to spf-devel in the new test suite schema (I'll provide > an update shortly). > >> I think I'm ready to release the COM version of SPF, I'm just trying to >> shake it down against as much test data I can gather, as well as >> implementing the new unit test model. Live data would help (so I don't >> hafta write code to add the dns records =). > > Great to hear you're doing a COM implementation of SPF! Please keep us > posted on the status. > > The new test suite is far from ready (although it might not take all that > long). You can use some of the older test data[1] until it is ready. > > References: > 1. http://www.openspf.org/svn/project/test-suite/contrib/ > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.3 (GNU/Linux) > > iD8DBQFEzG0RwL7PKlBZWjsRAqhpAKChjsRZRD1K21kmlM7sDGJjC4DjXgCdHVOA > 2C8RG+Aqlp8RhQHhRxgEAyM= > =l5KI > -----END PGP SIGNATURE----- > > ------- > To unsubscribe, change your address, or temporarily deactivate your > subscription, > please go to > http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx > ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx

Next Message by Date: click to view message preview

Re: Test Suite Zone Data

> What effort to update the DNS for mailzone.com and spftest.org? I know of > no such effort. The records exist to suit the live test cases, and are acknowledged as line item 1(c) on the Development Roadmap. This is the only hint on the http://new.openspf.org/Test_Suite page that infers an online/offline methodology, and that speaks online to me. Julian, I'll spare you the inline rebuts =) and sum up like so: To impose an offline style test suite IS more work for everyone than it is for an online style, merely to suit a condition that doesn't apply to everyone. True? That's both dns administration and code to handle the offline methodology (for all existing and non-existing implementations). An online methodology means only 1 person takes the hit of setting up the live records, for the benefit of a larger audience, no hooks or dns drivers needed in the implementation. Maybe to dns admins it's easy to stage the environment, but not every author that implements spf is a dns admin class developer (like me). This would also be locked down in (managed) corporate environments (running XP, which most of the corprorate world runs). Not every workstation is capable of doing so either. I'm a hellofa developer, and crappy dns admin =). But as a developer I say to you it is considerably more effort to program hooks to handle a dns driver, and every developer that assumes this task for future implementations will have to do the same (if they want to use the entire test suite). If you make the zone records live, then the existing dns drivers in place will still work -- best of both worlds. If I'm the only one that feels this way, then I'll go to the corner and chaw on my humble pie. Monte Hansen "Julian Mehnle" <julian@xxxxxxxxxx> wrote in message news:200607301605.23453.julian@xxxxxxxxxxxxx > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Monte Hansen wrote: >> > The new test suite will foremost and above all be an offline test >> > suite, the rationale being that often software, when being built and >> > tested automatically (such as by the Debian project's package builders >> > system), has no access to the Internet for practical and/or security >> > reasons. IOW, SPF implementations should be able to be tested >> > offline. >> >> Wow, knock me over with a feather. I would think that would not be the >> exception, not the rule. Was this a shift along the way? I mean, why the >> effort to update the dns for mailzone.com and spftest.org? > > What effort to update the DNS for mailzone.com and spftest.org? I know of > no such effort. > >> > You are not supposed to enter the zone data from the test suite into >> > real DNS. Any implementation should use a so-called adapter or test >> > driver to read the zonedata data from the test suite file instead of >> > from real DNS. >> >> Again, I'm just so very surprised. So, every single implementation has >> to construct their own dns driver just to implement the unit tests? > > No. Like before, any implementation is free to implement testing on its > own. It's not like we were _removing_ anything from existence. > > What we are doing is creating a _new_, _official_ test suite, which > happens > to be developed in an offline form. Don't use it if you don't like it. > >> This just requires _so much more code_ from everyone to create a test >> arena. > > I doubt it. > > Anyway, I don't understand why you're making such a fuss about this. As > soon as the offline test data is ready, anyone is free to copy it to a > real DNS zone of their choosing and exercise their implementation against > it. (Heck, even _we_ might do it, but who cares if _anybody_ can?) > > > The rest is redundant, or tangential only, but here we go: > >> Moreover, it also make the testing process more fallible. > > On the contrary. Offline testing eliminates all outside sources of prob- > lems in order to concentrate on the SPF core functionality. An implemen- > tation hardly needs an officially approved test suite in order to test > plain DNS functionality. > >> If someone/something doesn't have access to the Internet for an automated >> test suite, then let them install a local dns server for their test >> suite. > > Right, as if all the implementations were going to add bind9 (or even just > djbdns) to their build dependencies. No, that's not a good idea at all. > Besides, you are really missing the fact that you can always load the zone > data into your own DNS zone. > >> Please do not impose this on the implementations. > > We're not imposing anything that hasn't been imposed all the time already. > We're not removing anything from existence. > >> Each implementation shouldn't have to construct it's own environment, >> unless they need to for their own reasons. Imposing this will be a >> turnoff for developers on future implementations. > > I doubt it. Loading zone data into your own zone is trivial (but I'm > repeating myself), and writing an adapter for your implementation that > directly reads the offline data should be almost trivial as well. > Certainly a lot simpler than writing the SPF implementation itself anyway. > > I guess you just need to think the situation over once more, then it > should > all become clear. ;-) > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.3 (GNU/Linux) > > iD8DBQFEzNjDwL7PKlBZWjsRAjNPAJ4oRx2qGUkxLm6gC/hfWbhrPLwifgCgmRCB > HxNcKUKz9ewyhSsxCA0B3h4= > =1IAL > -----END PGP SIGNATURE----- > > ------- > To unsubscribe, change your address, or temporarily deactivate your > subscription, > please go to > http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx > ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx

Previous Message by Thread: click to view message preview

Re: Test Suite Zone Data

> The new test suite will foremost and above all be an offline test suite, > the rationale being that often software, when being built and tested > automatically (such as by the Debian project's package builders system), > has no access to the Internet for practical and/or security reasons. IOW, > SPF implementations should be able to be tested offline. > Wow, knock me over with a feather. I would think that would not be the exception, not the rule. Was this a shift along the way? I mean, why the effort to update the dns for mailzone.com and spftest.org? > You are not supposed to enter the zone data from the test suite into real > DNS. Any implementation should use a so-called adapter or test driver to > read the zonedata data from the test suite file instead of from real DNS. Again, I'm just so very surprised. So, every single implementation has to construct their own dns driver just to implement the unit tests? This just requires _so much more code_ from everyone to create a test arena. Moreover, it also make the testing process more fallible. DNS already works, and we've already written code to use it to support the framework. If someone/something doesn't have access to the Internet for an automated test suite, then let them install a local dns server for their test suite. I strongly urge that you guys rethink that one. Make it the exception, not the rule. It just seems like I'm being punished or something: the test suite is my cross to bear, and I have to carry it with me inside the framework libary code too ;-) Please do not impose this on the implementations. It should be setup for the implementations to use already -- it's part of the environment that should already be in place. Each implementation shouldn't have to construct it's own environment, unless they need to for their own reasons. Imposing this will be a turnoff for developers on future implementations. Respectfully submitted, Monte Hansen "Julian Mehnle" <julian@xxxxxxxxxx> wrote in message news:200607300825.53749.julian@xxxxxxxxxxxxx > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Monte Hansen wrote: >> I've tried to catch up on all the recent posts and didn't see any mention >> about anything related to the zone data with the new test suite. For >> instance, with the various live tests, I could run them without setup >> since they worked against live dns records in the mailzone.com tld. Not >> so in the new suite, it appears (example.com). >> >> I'm not clear what the plan is, but wouldn't it make sense to use that >> model. This way _each_ tester isn't required to stage an environment in >> accordance with every "zonedata:" test case from the yaml records. It >> would only need to be setup once for everyone to receive the benefit of >> it, and aligns with the Development Roadmap 1 c. > > The new test suite will foremost and above all be an offline test suite, > the rationale being that often software, when being built and tested > automatically (such as by the Debian project's package builders system), > has no access to the Internet for practical and/or security reasons. IOW, > SPF implementations should be able to be tested offline. > > You are not supposed to enter the zone data from the test suite into real > DNS. Any implementation should use a so-called adapter or test driver to > read the zonedata data from the test suite file instead of from real DNS. > > However, we do plan on mirroring the test data in real DNS once a solid > release of the offline test suite has been finished and we thus no longer > need to concentrate our efforts on getting the test suite done ASAP. Then > implementations can test themselves using the live DNS if they want > (althouth that will not be the preferred modus operandi of testing.) > >> That said, has anyone ported the live test cases to the new yaml layout? > > No, not yet. Some of the old tests will be ported, some will be > considered > obsolete. Also, some new tests will be developed. > > Everyone who wants to contribute valuable tests is invited to do so. > Please post tests to spf-devel in the new test suite schema (I'll provide > an update shortly). > >> I think I'm ready to release the COM version of SPF, I'm just trying to >> shake it down against as much test data I can gather, as well as >> implementing the new unit test model. Live data would help (so I don't >> hafta write code to add the dns records =). > > Great to hear you're doing a COM implementation of SPF! Please keep us > posted on the status. > > The new test suite is far from ready (although it might not take all that > long). You can use some of the older test data[1] until it is ready. > > References: > 1. http://www.openspf.org/svn/project/test-suite/contrib/ > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.3 (GNU/Linux) > > iD8DBQFEzG0RwL7PKlBZWjsRAqhpAKChjsRZRD1K21kmlM7sDGJjC4DjXgCdHVOA > 2C8RG+Aqlp8RhQHhRxgEAyM= > =l5KI > -----END PGP SIGNATURE----- > > ------- > To unsubscribe, change your address, or temporarily deactivate your > subscription, > please go to > http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx > ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx

Next Message by Thread: click to view message preview

Re: Test Suite Zone Data

> What effort to update the DNS for mailzone.com and spftest.org? I know of > no such effort. The records exist to suit the live test cases, and are acknowledged as line item 1(c) on the Development Roadmap. This is the only hint on the http://new.openspf.org/Test_Suite page that infers an online/offline methodology, and that speaks online to me. Julian, I'll spare you the inline rebuts =) and sum up like so: To impose an offline style test suite IS more work for everyone than it is for an online style, merely to suit a condition that doesn't apply to everyone. True? That's both dns administration and code to handle the offline methodology (for all existing and non-existing implementations). An online methodology means only 1 person takes the hit of setting up the live records, for the benefit of a larger audience, no hooks or dns drivers needed in the implementation. Maybe to dns admins it's easy to stage the environment, but not every author that implements spf is a dns admin class developer (like me). This would also be locked down in (managed) corporate environments (running XP, which most of the corprorate world runs). Not every workstation is capable of doing so either. I'm a hellofa developer, and crappy dns admin =). But as a developer I say to you it is considerably more effort to program hooks to handle a dns driver, and every developer that assumes this task for future implementations will have to do the same (if they want to use the entire test suite). If you make the zone records live, then the existing dns drivers in place will still work -- best of both worlds. If I'm the only one that feels this way, then I'll go to the corner and chaw on my humble pie. Monte Hansen "Julian Mehnle" <julian@xxxxxxxxxx> wrote in message news:200607301605.23453.julian@xxxxxxxxxxxxx > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Monte Hansen wrote: >> > The new test suite will foremost and above all be an offline test >> > suite, the rationale being that often software, when being built and >> > tested automatically (such as by the Debian project's package builders >> > system), has no access to the Internet for practical and/or security >> > reasons. IOW, SPF implementations should be able to be tested >> > offline. >> >> Wow, knock me over with a feather. I would think that would not be the >> exception, not the rule. Was this a shift along the way? I mean, why the >> effort to update the dns for mailzone.com and spftest.org? > > What effort to update the DNS for mailzone.com and spftest.org? I know of > no such effort. > >> > You are not supposed to enter the zone data from the test suite into >> > real DNS. Any implementation should use a so-called adapter or test >> > driver to read the zonedata data from the test suite file instead of >> > from real DNS. >> >> Again, I'm just so very surprised. So, every single implementation has >> to construct their own dns driver just to implement the unit tests? > > No. Like before, any implementation is free to implement testing on its > own. It's not like we were _removing_ anything from existence. > > What we are doing is creating a _new_, _official_ test suite, which > happens > to be developed in an offline form. Don't use it if you don't like it. > >> This just requires _so much more code_ from everyone to create a test >> arena. > > I doubt it. > > Anyway, I don't understand why you're making such a fuss about this. As > soon as the offline test data is ready, anyone is free to copy it to a > real DNS zone of their choosing and exercise their implementation against > it. (Heck, even _we_ might do it, but who cares if _anybody_ can?) > > > The rest is redundant, or tangential only, but here we go: > >> Moreover, it also make the testing process more fallible. > > On the contrary. Offline testing eliminates all outside sources of prob- > lems in order to concentrate on the SPF core functionality. An implemen- > tation hardly needs an officially approved test suite in order to test > plain DNS functionality. > >> If someone/something doesn't have access to the Internet for an automated >> test suite, then let them install a local dns server for their test >> suite. > > Right, as if all the implementations were going to add bind9 (or even just > djbdns) to their build dependencies. No, that's not a good idea at all. > Besides, you are really missing the fact that you can always load the zone > data into your own DNS zone. > >> Please do not impose this on the implementations. > > We're not imposing anything that hasn't been imposed all the time already. > We're not removing anything from existence. > >> Each implementation shouldn't have to construct it's own environment, >> unless they need to for their own reasons. Imposing this will be a >> turnoff for developers on future implementations. > > I doubt it. Loading zone data into your own zone is trivial (but I'm > repeating myself), and writing an adapter for your implementation that > directly reads the offline data should be almost trivial as well. > Certainly a lot simpler than writing the SPF implementation itself anyway. > > I guess you just need to think the situation over once more, then it > should > all become clear. ;-) > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.3 (GNU/Linux) > > iD8DBQFEzNjDwL7PKlBZWjsRAjNPAJ4oRx2qGUkxLm6gC/hfWbhrPLwifgCgmRCB > HxNcKUKz9ewyhSsxCA0B3h4= > =1IAL > -----END PGP SIGNATURE----- > > ------- > To unsubscribe, change your address, or temporarily deactivate your > subscription, > please go to > http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx > ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?listname=spf-devel@xxxxxxxxxxxxxx