Subject: Re: Potentially interesting (and definitely useful) TODO task for revamping master_sites - msg#00335

Previous Message by Date: click to view message preview

[New port] scmxx - Data exchange utility for Siemens mobile phones

Hi List! A port for scmxx is available. Bugzilla #480 Max

Next Message by Date: click to view message preview

Re: Potentially interesting (and definitely useful) TODO task for revamping master_sites

On Wednesday, April 23, 2003, at 06:29PM, Jordan Hubbard <jkh@xxxxxxxxxxxxxx> wrote: >Hmmm. > >Clearly, everyone has leaped on the lowest-ping function which I tried >my best to make clear was just one _possibility_ and probably nowhere >near as useful as some number of other helper functions. I was >largely just trying to illustrate how the helpers could be combined if >they were true functions vs simply being magic variables floating >around in the global namespace. If the idea of sorting by lowest ping >bothers people then they should simply toss that section of my email >away and look purely at the first part of it which deals with master >site collections. :-) I didn't say not to do the other stuff.. I said don't go by ping :) [at least I don't think I said not to do the other stuff.] DO THE OTHER STUFF! :) Dave > >- Jordan > >On Wednesday, April 23, 2003, at 04:09 PM, Thomas Vincent wrote: > >> Why not use geography like CPAN? >> Or determine how much bandwidth/server resources each master site has >> and produce a score. The majority of servers that DP references are >> relatively few in reality. The information shouldn't be hard to >> compile. >> In addition you could resolve the machine's domain name and if a >> mirror server's domain name matched, the server could be given more >> weight. For example this would work well with sites with large mirrors >> such as mit.edu or unc.edu, >> >> There are too many mis-configured networks out there to rely on pings. >> >> Cheers, >> Tom >> >> On Wednesday, April 23, 2003, at 01:18 PM, David Leimbach wrote: >> >>> I don't think sorting by lowest ping is a good idea. >>> >>> At work our firewall filters all incoming pings so we can send out >>> but not hear >>> the response. This would mean none of my OS X boxes can use >>> Darwinports. >>> >>> I think this might even be a fairly common configuration as there is >>> that "ping of death" >>> DDOS attack out there right? >>> >>> Dave >>> On Wednesday, April 23, 2003, at 02:59PM, Jordan Hubbard >>> <jkh@xxxxxxxxxxxxxx> wrote: >>> >>>> If anyone's looking for an interesting (but still rather easy) Tcl >>>> task, I might suggest that it's time to revamp our usage of >>>> master_sites somewhat through the addition of some useful helper >>>> functions and at least one global set of "canned site lists" which >>>> I'll >>>> explain in a moment. >>>> >>>> What I have in mind is probably best illustrated with a little >>>> pseudo-Tcl code (and easier to write than real Tcl code :-). >>>> >>>> First, we look at the way that FreeBSD defines its nifty canned >>>> master >>>> site classes (in /usr/ports/Mk/bsd.sites.mk): >>>> >>>> MASTER_SITE_GNU+= \ >>>> ftp://ftp.gnu.org/gnu/%SUBDIR%/ \ >>>> ftp://gatekeeper.dec.com/pub/GNU/%SUBDIR%/ \ >>>> ... >>>> >>>> And we convert that to something more aligned with Tcl vs make(1) >>>> since >>>> it's just as easy for us to have ${subdir} or whatever we choose to >>>> call it expand using normal variable expansion: >>>> >>>> set _master_sites(gnu) { >>>> ftp://ftp.gnu.org/gnu/${subdir}/ >>>> ftp://gatekeeper.dec.com/pub/GNU/${subdir}/ >>>> ... >>>> } >>>> >>>> Then we write a little helper function for expanding variable >>>> references in these name lists and returning the expanded list: >>>> >>>> # Expand all variable references in each site variable, passing back >>>> new expanded list >>>> proc expand-site-vars {sites} { >>>> initialize new list x >>>> foreach element in sites { >>>> eval append x $element >>>> } >>>> return x >>>> } >>>> >>>> And then we write a function for accessing a canned list of master >>>> sites so that you can say something like this in your Portfile: >>>> >>>> master_sites [master-sites-for gnu] >>>> >>>> Which looks something like this: >>>> >>>> # For a given master site type, e.g. "gnu" or "x11", check to see if >>>> there's a pre-registered >>>> # set of sites (somewhere suitable in the DP infrastructure) and, if >>>> so, return them >>>> proc master-sites-for {arg} { >>>> global _master_sites >>>> >>>> if [!exists _master_sites($arg)] { >>>> ui_puts "No master sites on file for class $arg" >>>> return {} >>>> } >>>> return [expand-site-vars $_master_sites($arg) ] >>>> } >>>> >>>> Now that we've introduced the notion of helper functions which return >>>> lists of sites for master-sites' consumption, we can get >>>> (theoretically >>>> anyway) even more creative: >>>> >>>> # Given a list of URLs, return a list which is sorted by lowest ping >>>> time >>>> proc lowest-ping {args} { >>>> foreach element in args { >>>> set times($element) [extract-ping-time [ip-from-url >>>> $element]] >>>> } >>>> sort array times >>>> foreach element in times { >>>> lappend res $element >>>> } >>>> return res >>>> } >>>> >>>> Which might allow you to say: >>>> >>>> master_sites [lowest-ping [master-sites-for gnu] http://my.own.site >>>> $darwinports_master_site] >>>> >>>> And offer a wide variety of choices to the Portfile which are then >>>> sorted according to whichever site actually makes the most sense for >>>> the end-user. Note that this particular function is likely to be >>>> somewhat difficult to implement without some C code since our ping(1) >>>> command doesn't actually lend itself well to this kind of usage (it >>>> takes too long to time out on an unreachable site) but that's not to >>>> say that other helper functions for sorting / pruning lists of master >>>> sites wouldn't be equally helpful and easier to write. >>>> >>>> Any takers? This would be a great and rather self-contained "excuse >>>> to >>>> learn Tcl" task for someone out there. :-) >>>> >>>> - Jordan >>>> >>>> _______________________________________________ >>>> Darwinports mailing list >>>> Darwinports@xxxxxxxxxxxxxx >>>> http://www.opendarwin.org/mailman/listinfo/darwinports >>>> >>>> >>> _______________________________________________ >>> Darwinports mailing list >>> Darwinports@xxxxxxxxxxxxxx >>> http://www.opendarwin.org/mailman/listinfo/darwinports >> >> _______________________________________________ >> Darwinports mailing list >> Darwinports@xxxxxxxxxxxxxx >> http://www.opendarwin.org/mailman/listinfo/darwinports >> >-- >Jordan K. Hubbard >Engineering Manager, BSD technology group >Apple Computer > >_______________________________________________ >Darwinports mailing list >Darwinports@xxxxxxxxxxxxxx >http://www.opendarwin.org/mailman/listinfo/darwinports > >

Previous Message by Thread: click to view message preview

Re: Potentially interesting (and definitely useful) TODO task for revamping master_sites

Why not use geography like CPAN? Or determine how much bandwidth/server resources each master site has and produce a score. The majority of servers that DP references are relatively few in reality. The information shouldn't be hard to compile. In addition you could resolve the machine's domain name and if a mirror server's domain name matched, the server could be given more weight. For example this would work well with sites with large mirrors such as mit.edu or unc.edu, There are too many mis-configured networks out there to rely on pings. Cheers, Tom On Wednesday, April 23, 2003, at 01:18 PM, David Leimbach wrote: I don't think sorting by lowest ping is a good idea. At work our firewall filters all incoming pings so we can send out but not hear the response. This would mean none of my OS X boxes can use Darwinports. I think this might even be a fairly common configuration as there is that "ping of death" DDOS attack out there right? Dave On Wednesday, April 23, 2003, at 02:59PM, Jordan Hubbard <jkh@xxxxxxxxxxxxxx> wrote: If anyone's looking for an interesting (but still rather easy) Tcl task, I might suggest that it's time to revamp our usage of master_sites somewhat through the addition of some useful helper functions and at least one global set of "canned site lists" which I'll explain in a moment. What I have in mind is probably best illustrated with a little pseudo-Tcl code (and easier to write than real Tcl code :-). First, we look at the way that FreeBSD defines its nifty canned master site classes (in /usr/ports/Mk/bsd.sites.mk): MASTER_SITE_GNU+= \ ftp://ftp.gnu.org/gnu/%SUBDIR%/ \ ftp://gatekeeper.dec.com/pub/GNU/%SUBDIR%/ \ ... And we convert that to something more aligned with Tcl vs make(1) since it's just as easy for us to have ${subdir} or whatever we choose to call it expand using normal variable expansion: set _master_sites(gnu) { ftp://ftp.gnu.org/gnu/${subdir}/ ftp://gatekeeper.dec.com/pub/GNU/${subdir}/ ... } Then we write a little helper function for expanding variable references in these name lists and returning the expanded list: # Expand all variable references in each site variable, passing back new expanded list proc expand-site-vars {sites} { initialize new list x foreach element in sites { eval append x $element } return x } And then we write a function for accessing a canned list of master sites so that you can say something like this in your Portfile: master_sites [master-sites-for gnu] Which looks something like this: # For a given master site type, e.g. "gnu" or "x11", check to see if there's a pre-registered # set of sites (somewhere suitable in the DP infrastructure) and, if so, return them proc master-sites-for {arg} { global _master_sites if [!exists _master_sites($arg)] { ui_puts "No master sites on file for class $arg" return {} } return [expand-site-vars $_master_sites($arg) ] } Now that we've introduced the notion of helper functions which return lists of sites for master-sites' consumption, we can get (theoretically anyway) even more creative: # Given a list of URLs, return a list which is sorted by lowest ping time proc lowest-ping {args} { foreach element in args { set times($element) [extract-ping-time [ip-from-url $element]] } sort array times foreach element in times { lappend res $element } return res } Which might allow you to say: master_sites [lowest-ping [master-sites-for gnu] http://my.own.site $darwinports_master_site] And offer a wide variety of choices to the Portfile which are then sorted according to whichever site actually makes the most sense for the end-user. Note that this particular function is likely to be somewhat difficult to implement without some C code since our ping(1) command doesn't actually lend itself well to this kind of usage (it takes too long to time out on an unreachable site) but that's not to say that other helper functions for sorting / pruning lists of master sites wouldn't be equally helpful and easier to write. Any takers? This would be a great and rather self-contained "excuse to learn Tcl" task for someone out there. :-) - Jordan _______________________________________________ Darwinports mailing list Darwinports@xxxxxxxxxxxxxx http://www.opendarwin.org/mailman/listinfo/darwinports _______________________________________________ Darwinports mailing list Darwinports@xxxxxxxxxxxxxx http://www.opendarwin.org/mailman/listinfo/darwinports

Next Message by Thread: click to view message preview

Re: Potentially interesting (and definitely useful) TODO task for revamping master_sites

On Wednesday, April 23, 2003, at 06:29PM, Jordan Hubbard <jkh@xxxxxxxxxxxxxx> wrote: >Hmmm. > >Clearly, everyone has leaped on the lowest-ping function which I tried >my best to make clear was just one _possibility_ and probably nowhere >near as useful as some number of other helper functions. I was >largely just trying to illustrate how the helpers could be combined if >they were true functions vs simply being magic variables floating >around in the global namespace. If the idea of sorting by lowest ping >bothers people then they should simply toss that section of my email >away and look purely at the first part of it which deals with master >site collections. :-) I didn't say not to do the other stuff.. I said don't go by ping :) [at least I don't think I said not to do the other stuff.] DO THE OTHER STUFF! :) Dave > >- Jordan > >On Wednesday, April 23, 2003, at 04:09 PM, Thomas Vincent wrote: > >> Why not use geography like CPAN? >> Or determine how much bandwidth/server resources each master site has >> and produce a score. The majority of servers that DP references are >> relatively few in reality. The information shouldn't be hard to >> compile. >> In addition you could resolve the machine's domain name and if a >> mirror server's domain name matched, the server could be given more >> weight. For example this would work well with sites with large mirrors >> such as mit.edu or unc.edu, >> >> There are too many mis-configured networks out there to rely on pings. >> >> Cheers, >> Tom >> >> On Wednesday, April 23, 2003, at 01:18 PM, David Leimbach wrote: >> >>> I don't think sorting by lowest ping is a good idea. >>> >>> At work our firewall filters all incoming pings so we can send out >>> but not hear >>> the response. This would mean none of my OS X boxes can use >>> Darwinports. >>> >>> I think this might even be a fairly common configuration as there is >>> that "ping of death" >>> DDOS attack out there right? >>> >>> Dave >>> On Wednesday, April 23, 2003, at 02:59PM, Jordan Hubbard >>> <jkh@xxxxxxxxxxxxxx> wrote: >>> >>>> If anyone's looking for an interesting (but still rather easy) Tcl >>>> task, I might suggest that it's time to revamp our usage of >>>> master_sites somewhat through the addition of some useful helper >>>> functions and at least one global set of "canned site lists" which >>>> I'll >>>> explain in a moment. >>>> >>>> What I have in mind is probably best illustrated with a little >>>> pseudo-Tcl code (and easier to write than real Tcl code :-). >>>> >>>> First, we look at the way that FreeBSD defines its nifty canned >>>> master >>>> site classes (in /usr/ports/Mk/bsd.sites.mk): >>>> >>>> MASTER_SITE_GNU+= \ >>>> ftp://ftp.gnu.org/gnu/%SUBDIR%/ \ >>>> ftp://gatekeeper.dec.com/pub/GNU/%SUBDIR%/ \ >>>> ... >>>> >>>> And we convert that to something more aligned with Tcl vs make(1) >>>> since >>>> it's just as easy for us to have ${subdir} or whatever we choose to >>>> call it expand using normal variable expansion: >>>> >>>> set _master_sites(gnu) { >>>> ftp://ftp.gnu.org/gnu/${subdir}/ >>>> ftp://gatekeeper.dec.com/pub/GNU/${subdir}/ >>>> ... >>>> } >>>> >>>> Then we write a little helper function for expanding variable >>>> references in these name lists and returning the expanded list: >>>> >>>> # Expand all variable references in each site variable, passing back >>>> new expanded list >>>> proc expand-site-vars {sites} { >>>> initialize new list x >>>> foreach element in sites { >>>> eval append x $element >>>> } >>>> return x >>>> } >>>> >>>> And then we write a function for accessing a canned list of master >>>> sites so that you can say something like this in your Portfile: >>>> >>>> master_sites [master-sites-for gnu] >>>> >>>> Which looks something like this: >>>> >>>> # For a given master site type, e.g. "gnu" or "x11", check to see if >>>> there's a pre-registered >>>> # set of sites (somewhere suitable in the DP infrastructure) and, if >>>> so, return them >>>> proc master-sites-for {arg} { >>>> global _master_sites >>>> >>>> if [!exists _master_sites($arg)] { >>>> ui_puts "No master sites on file for class $arg" >>>> return {} >>>> } >>>> return [expand-site-vars $_master_sites($arg) ] >>>> } >>>> >>>> Now that we've introduced the notion of helper functions which return >>>> lists of sites for master-sites' consumption, we can get >>>> (theoretically >>>> anyway) even more creative: >>>> >>>> # Given a list of URLs, return a list which is sorted by lowest ping >>>> time >>>> proc lowest-ping {args} { >>>> foreach element in args { >>>> set times($element) [extract-ping-time [ip-from-url >>>> $element]] >>>> } >>>> sort array times >>>> foreach element in times { >>>> lappend res $element >>>> } >>>> return res >>>> } >>>> >>>> Which might allow you to say: >>>> >>>> master_sites [lowest-ping [master-sites-for gnu] http://my.own.site >>>> $darwinports_master_site] >>>> >>>> And offer a wide variety of choices to the Portfile which are then >>>> sorted according to whichever site actually makes the most sense for >>>> the end-user. Note that this particular function is likely to be >>>> somewhat difficult to implement without some C code since our ping(1) >>>> command doesn't actually lend itself well to this kind of usage (it >>>> takes too long to time out on an unreachable site) but that's not to >>>> say that other helper functions for sorting / pruning lists of master >>>> sites wouldn't be equally helpful and easier to write. >>>> >>>> Any takers? This would be a great and rather self-contained "excuse >>>> to >>>> learn Tcl" task for someone out there. :-) >>>> >>>> - Jordan >>>> >>>> _______________________________________________ >>>> Darwinports mailing list >>>> Darwinports@xxxxxxxxxxxxxx >>>> http://www.opendarwin.org/mailman/listinfo/darwinports >>>> >>>> >>> _______________________________________________ >>> Darwinports mailing list >>> Darwinports@xxxxxxxxxxxxxx >>> http://www.opendarwin.org/mailman/listinfo/darwinports >> >> _______________________________________________ >> Darwinports mailing list >> Darwinports@xxxxxxxxxxxxxx >> http://www.opendarwin.org/mailman/listinfo/darwinports >> >-- >Jordan K. Hubbard >Engineering Manager, BSD technology group >Apple Computer > >_______________________________________________ >Darwinports mailing list >Darwinports@xxxxxxxxxxxxxx >http://www.opendarwin.org/mailman/listinfo/darwinports > >