[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Performance Testing - request for comments

A few thoughts:

1. Gradle can intelligently build only the dependencies necessary for a task, so it shouldn't build all of Python if the test suite, if you only specify the task you're interested in. I'm not sure of the command for "build all of the dependencies of my tests but don't run my tests"; maybe "./gradlew mytests -x mytests" ?

2. Some tasks in the build are not yet cacheable for various reasons. So you may see them getting rebuilt on the second execution even on success, which would then be included in your overall build timing. Information about which tasks were used from the build cache is available in the Gradle build scan (--scan).

Another idea for measuring the execution time of just your tests would be to pull this out of Gradle's build report.  Adding the --profile flag generates a report in $buildDir/reports/profile, which should have the timing info for just the task you're interested in: https://docs.gradle.org/current/userguide/command_line_interface.html 

On Tue, May 8, 2018 at 8:23 AM Łukasz Gajowy <lukasz.gajowy@xxxxxxxxx> wrote:
Hi Beam Devs, 

currently PerfkitBenchmarker (a tool used to invoke performance tests) has two phases that run gradle commands:
  • Pre-build phase: this is where all the beam repo is build. This phase is to prepare the necessary artifacts so that it doesn't happen when executing tests.
  • Actual test running phase. After all necessary code is built we run the test and measure it's execution time. The execution time is displayed on the PerfKit dashboard [1].
After the recent mvn - gradle migration we noticed that we are unable to "Pre build" the code[2]. Because one of the python related tasks fails, the whole "preBuild" phase fails silently and the actual building happens in the "test running" phase which increases the execution time (this is visible in the plots on the dashboard). 

This whole situation made me wonder about several things, and I'd like to ask you for opinions. I think:
  • we should skip all the python related tasks while building beam for java performance tests in PerfKit. Those are not needed anyway when we are running java tests. Is it possible to skip them in one go (eg. the same fashion we skip all checks using -xcheck option)?
  • the same goes for Python tests: we should skip all java related tasks when building beam for python performance tests in PerfKit. Note that this bullet is something to be developed in the future, as beam_PerformanceTests_Python job (the only Python Performance test job) is failing for 4 months now and seems abandoned. IMO it should be done when someone will bring the test back to life. For now the job should be disabled. 
  • we should modify Perfkit so that when the prebuild phase fails for some reason, the test is not executed. Now we don't do this and the test execution time depends on whether "gradle integrationTest" command builds something or just runs the test. IMO when the command has to build anything the execution time should not be included in the Dashboards, because it's a false result.
[2] https://issues.apache.org/jira/browse/BEAM-4256

Best regards, 
Łukasz Gajowy