[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Documenting Github PR jenkins trigger phrases

+1 to adding a doc for this, along with some PR conventions about when to use these. Some questions that I would love to see documented:

* What tests are run automatically and when (pre-commits, all languages, on every commit)
* What other test suites exist, and how should they be used (post-commits and performance tests run on a schedule and validate already-merged code; if a merge breaks post-commits we should prefer rollback over rollforward to keep master in a healthy state; if you think your change could affect one of these suites you can invoke them manually on your PR)
* How do reviewers use Jenkins signals (i.e. do reviewers expect pre-commits to be green before reviewing PR's?)
* What should you do if you think a flaky test caused a failure on your PR? (re-run tests using magical Jenkins incantation)

I also like the idea of automatically scraping the string from the source-of-truth rather than manually maintaining. However, I suspect it might be more trouble than it's worth (the groovy files are in a different git repo than beam-site; the site is currently built from simple markdown files which get compiled to HTML and checked-in). If sourcing the strings from the source proves difficult, I'm in favor of going the simple route and manually maintaining the list; I think there's sufficient value to justify the maintenance cost of.

Another idea would be to see if we could query Jenkins via _javascript_ on page load for the set of beam jobs and trigger phrases. This could be a simpler integration point and has the benefit of always being up-to-date regardless of which commit was used to generate the jobs.

On Fri, May 11, 2018 at 7:11 AM Alexey Romanenko <aromanenko.dev@xxxxxxxxx> wrote:
+1 to add such reference guide of Jenkins commands to Testing guide. It should be extremely useful, especially for those who were not aware about this before. 


On 11 May 2018, at 02:44, Ankur Goenka <goenka@xxxxxxxxxx> wrote:

In my experience affect of white space in commit is inconsistent for certain commands they do matter while for others they don't.

On Thu, May 10, 2018 at 5:43 PM Valentyn Tymofieiev <valentyn@xxxxxxxxxx> wrote:
+1 to writing a Beam Jenkins spellbook. 
I have observed that Jenkins commands sometimes don't work for the first time, why could this be? Do end of lines at the end of command matter?

On Thu, May 10, 2018 at 1:24 PM, Andrew Pilloud <apilloud@xxxxxxxxxx> wrote:
It would be great to have the set of "Run {Java,Python,Go} PreCommit" documented in the contributors guide as well. Those match up to the jobs auto run on every PR and are the ones I use most. There is no security, anyone can run them including 'Run Seed Job'. That one seems like a good one to document in testing, because it is the one that loads changes to the rest.


On Thu, May 10, 2018 at 12:27 PM Huygaa Batsaikhan <batbat@xxxxxxxxxx> wrote:
Hi devs,

We can run various jenkins commands (precommit, postcommit, performance tests) directly from Github Pull Request UI by commenting phrases such as "retest this please". Unfortunately, this tool is not documented. I am adding a brief documentation in https://beam.apache.org/contribute/testing/ and I need some help.
  1. What are the most common phrases used?
  2. Can anyone run these commands? Are there any permission issues?
  3. Does it make sense to categorize the commands as Performance tests, Precommit, Postcommit, and Release Validation?
Let me know what you think,