[release] making releases fast again (was: decentralising release approvals)
Mark Goddard wrote:
> As kolla PTL and ironic release liaison I've proposed a number of
> release patches recently. Generally the release team is good at churning
> through these, but sometimes patches can hang around for a while.
> Usually a ping on IRC will get things moving again within a day or so
> (thanks in particular to Sean who has been very responsive).
I agree we've seen an increase in processing delay lately, and I'd like
to correct that. There are generally three things that would cause a
perceptible delay in release processing...
1- wait for two release managers +2
This is something we put in place some time ago, as we had a lot of new
members and thought that would be a good way to onboard them. Lately it
created delays as a lot of those were not as active.
2- stable releases
Two subcases in there... Eitherthe deliverable is under stable policy
and there are *significant* delays there as we have to pause to give a
chance to stable-maint-core people to voice an opinion. Or the
deliverable is not under stable policy, but we do a manual check on the
changes, as a way to educate the requester on semver.
3- waiting for PTL/release liaison to approve
That can take a long time, but the release management team is not really
at fault there.
Could you describe where you've seen "sometimes patches can hang around
for a while"? I suspect they belong in the (2) category?
> I have a few questions for the release team about these reviews.
> * What manual checks do you do beyond those that are currently automated?
> * Could the above checks be automated?
We aggressively automate everything that can be. Like I'm currently
working to automate the check that the release was approved by the PTL
or release liaison.
> * What issues have you caught that were not caught by CI jobs?
It's generally semver violations, or timing issues (like requesting a
release during a freeze). Sometimes it's corner cases not handled (yet)
by automation, like incompatibility between the release version asked
and the deliverable release model. You can look at the history of
releases for examples.
> Hopefully I haven't offended anyone here. There's often more involved
> with these things than you first suspect.
Decentralizing would be a lot of work to create new systems and
processes... and I don't think we can automate everything. It's
unreasonable to expect everyone to know the release process by heart and
respect timing and freezes. And releases are the only thing we produce
that we can't undo.
I would rather eliminate the issue by making sure release processing is
back to fast. So here is my proposal:
- go back to single release manager approval
- directly approve stable releases after a cursory semver check, not
waiting for stable-maint-core approval.
That should make sure all releases are processed within a couple of
days, which I think is a good trade-off between retaining some releases
for 10+ days and not having a chance to catch odd cases before releases
Thierry Carrez (ttx)