osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Automatically create JIRA tickets for failing post-commit tests


I think this looks good, we should enable the plugin and try it out. Concrete details of the follow-up tasks (auto-assignment, triage, and dashboarding) will probably depend on how functional the plugin is and what the test failures data looks like.

Regards,
Anton

On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin <migryz@xxxxxxxxxx> wrote:
@Yifan Zou 

I believe that we should test-drive the system with tickets + PR first and decide on email notification later. We already have tests failure emails sent to commits@, I believe most people filter out or not signed up for that list though.

It creates only one ticket, and keeps it for recurring test failures.

Thank you for the suggestion. I'll add it to design doc.

--Mikhail



On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou <yifanzou@xxxxxxxxxx> wrote:
+1 to Andrew's concerns. Leaving the tickets unassigned will cause the ticket being ignored and no actions being taken.

I can see the challenges on ticket assignment. Like Mikhail mentioned, the plugin does not support dynamic assignments. We have to implement custom script to determine the assignees and do some tricks to the jenkins job. Also, the post-commits tests usually cover tons of stuffs that it is difficult to find which part was broken and ask the right person to look into within the Auto JIRA process. Some naive thoughts: Are we able to send emails to the dev@ to ask people to take care of the JIRA issues? Are we able to find component leads and ask them triage the test failure tickets?

Another nitpick comment. Does the jenkins job file the JIRA issue in every test failure? Sometimes the test continuously fails in a time period due to the same reason. In this case, we will get some duplicate issues filed by Jenkins. I think it could be better if we can avoid filing issues if the previous one has not been resolved.

Thanks.
Yifan


On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud <apilloud@xxxxxxxxxx> wrote:
That sounds great. You should add this detail to the doc.

On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin <migryz@xxxxxxxxxx> wrote:
We already have component for this purpose: "test-failures". All tickets created will go to that component. As an option, we can add link to view list of open JIRA tickets to PR template. 

We also would want to create graph on dashboard with amount of unassigned and assigned bugs. 

I believe that we can also add counter of unassigned bugs to PR template. This way it will be easier for everyone to know when there's some tests issue not attended.

--Mikhail


On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud <apilloud@xxxxxxxxxx> wrote:
So it sounds like you will want to create a component for untriaged issues so they are easy to find. I like the idea of distributing the work of triaging post commit failures to new PR authors as a condition of merging. I feel like we will just be filling JIRA with spam if the issues are automatically created without a plan for triage.

Andrew

On Wed, Jul 11, 2018 at 4:12 PM Rui Wang <ruwang@xxxxxxxxxx> wrote:
Maybe this is also a good thread to start the discussion that if we want to enforce postcommit test for every PR. 

Can we afford the cost of longer waiting time to catch potential bugs? 

-Rui 

On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin <migryz@xxxxxxxxxx> wrote:
That's a valid point. 

Unfortunately, the JiraTestResultReporter plugin does not have features to dynamically assign owners. Additionally, I don't think it is always easy to find proper owner for post-commit tests at first glance, since they usually cover broad specter of issues.

My assumption is that we need someone to triage new issues. 

Ideally, any contributor, who sees failing test, should check unassigned tickets and either do triage, or assign them to someone who can. I strongly encourage this approach.

We have couple other ready-made options to consider:
1. We can configure JIRA component owner who would be assigned to created tickets.
2. JiraTestReporterPlugin can assign tickets to specific user. This is configured per Jenkins job. We can utilize this if someone volunteers.
3. Dynamic assignment will most likely require custom solution.

--Mikhail


On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud <apilloud@xxxxxxxxxx> wrote:
Hi Mikhail,

I like the proposal! Hopefully this can replace the constant stream of build failure emails. I noticed one detail seems to be missing:  How will new issues be assigned to the proper owner? Will the tool do this automatically or will we need someone to triage new issues?

Andrew

On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin <migryz@xxxxxxxxxx> wrote:
Hi everyone,

I want to add an automatic JIRA tickets creation for failing post-commit tests.

I wrote up design proposal doc with more details on this:

Quick summary:
I suggest to utilize JiraTestResultReporter plugin. 
Since this plugin is not installed on our Jenkins yet, we have to request to Infra team to add it.

Please, comment if this approach sounds good to you.

Best regards,
--Mikhail