Subject: Re: Regression testing (was Re: Performance issue) - msg#00082

Previous Message by Date: click to view message preview

Regression testing (was Re: Performance issue)

This thread makes me wonder if there is value in runing performance tests on a regular basis. This would give an early warning of any peformance loss and can be a useful forensic tool (one can pinpoint when some performance curve changed discontinuously even though at the time of change it may be too small to be noticed). Over a period of time one can gain a view of how the performance evolves. This would not be a single metric but a set of low and high level measures: such as syscall overhead, interrupt overhead, specific h/w devices, disk and fs performance for various filesystems and file sizes, networking data and pkt throughput, routing performance, VM, other subsystems, effect of SMP, various threading libraries, scaling with number of users/programs/cpus/memory, typical applications under normal and stressed loads, compile time for the system and kernel etc. etc. etc. The setup would allow for easy addition of new benchmarks (the only way anything like this can be bootstrapped). Of course, one would need to record disk/processor/memory speed and capacities + kernel config options, system build tools and their options to interpret the results as best as possible. For the results to be useful the setup has to remain as stable as possible for a long time. [While I am dreaming...] A follow on project would be to create visualization tools -- mainly graphing and comparing graphs. It would be neat if one can click on a performance graph to zoom in or see commits made during some selected period. Such a detailed look, combined with profiling can help people focus on specific hotspots & feel good about any improvements they are making. This can be a great way to rope in new people;-) _______________________________________________ freebsd-performance@xxxxxxxxxxx mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Next Message by Date: click to view message preview

Re: Regression testing (was Re: Performance issue)

This sounds somewhat similar to Solaris dtrace stuff? Pete Bakul Shah wrote: This thread makes me wonder if there is value in runing performance tests on a regular basis. This would give an early warning of any peformance loss and can be a useful forensic tool (one can pinpoint when some performance curve changed discontinuously even though at the time of change it may be too small to be noticed). Over a period of time one can gain a view of how the performance evolves. This would not be a single metric but a set of low and high level measures: such as syscall overhead, interrupt overhead, specific h/w devices, disk and fs performance for various filesystems and file sizes, networking data and pkt throughput, routing performance, VM, other subsystems, effect of SMP, various threading libraries, scaling with number of users/programs/cpus/memory, typical applications under normal and stressed loads, compile time for the system and kernel etc. etc. etc. The setup would allow for easy addition of new benchmarks (the only way anything like this can be bootstrapped). Of course, one would need to record disk/processor/memory speed and capacities + kernel config options, system build tools and their options to interpret the results as best as possible. For the results to be useful the setup has to remain as stable as possible for a long time. [While I am dreaming...] A follow on project would be to create visualization tools -- mainly graphing and comparing graphs. It would be neat if one can click on a performance graph to zoom in or see commits made during some selected period. Such a detailed look, combined with profiling can help people focus on specific hotspots & feel good about any improvements they are making. This can be a great way to rope in new people;-) _______________________________________________ freebsd-performance@xxxxxxxxxxx mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx" _______________________________________________ freebsd-performance@xxxxxxxxxxx mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Previous Message by Thread: click to view message preview

Regression testing (was Re: Performance issue)

This thread makes me wonder if there is value in runing performance tests on a regular basis. This would give an early warning of any peformance loss and can be a useful forensic tool (one can pinpoint when some performance curve changed discontinuously even though at the time of change it may be too small to be noticed). Over a period of time one can gain a view of how the performance evolves. This would not be a single metric but a set of low and high level measures: such as syscall overhead, interrupt overhead, specific h/w devices, disk and fs performance for various filesystems and file sizes, networking data and pkt throughput, routing performance, VM, other subsystems, effect of SMP, various threading libraries, scaling with number of users/programs/cpus/memory, typical applications under normal and stressed loads, compile time for the system and kernel etc. etc. etc. The setup would allow for easy addition of new benchmarks (the only way anything like this can be bootstrapped). Of course, one would need to record disk/processor/memory speed and capacities + kernel config options, system build tools and their options to interpret the results as best as possible. For the results to be useful the setup has to remain as stable as possible for a long time. [While I am dreaming...] A follow on project would be to create visualization tools -- mainly graphing and comparing graphs. It would be neat if one can click on a performance graph to zoom in or see commits made during some selected period. Such a detailed look, combined with profiling can help people focus on specific hotspots & feel good about any improvements they are making. This can be a great way to rope in new people;-) _______________________________________________ freebsd-performance@xxxxxxxxxxx mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Next Message by Thread: click to view message preview

Re: Regression testing (was Re: Performance issue)

On Tue, May 10, 2005 at 01:35:40PM -0500, Jonathan Noack wrote: > Sounds great! When do you begin? ;-) > > This has been proposed before and has been (to my knowledge) universally > accepted as a Good Idea. If you have the interest and time to devote to > it, I would urge you to work on it. The benefit to the community would > be huge. Does FreeBSD have any facilities setup for peoople wishing to work on 'sub projects' collaboratively? Something akin to http://pgfoundry.org for PostgreSQL? Several of us have been using pgFoundry as a means to work on things that will eventually go into PostgreSQL itself, and it's been a big benefit when it comes to bringing people together to work on something. -- Jim C. Nasby, Database Consultant decibel@xxxxxxxxxxx Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?" _______________________________________________ freebsd-performance@xxxxxxxxxxx mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"