[qa][openstackclient] Debugging devstack slowness
On 8/6/19 11:34 AM, Ben Nemec wrote:
> On 8/6/19 10:49 AM, Clark Boylan wrote:
>> On Tue, Aug 6, 2019, at 8:26 AM, Ben Nemec wrote:
>>> Just a reminder that there is also
>>> which was intended to address this same issue.
>>> I toyed around with it a bit for TripleO installs back then and it did
>>> seem to speed things up, but at the time there was a bug in our client
>>> plugin where it was triggering a prompt for input that was problematic
>>> with the server running in the background. I never really got back to it
>>> once that was fixed. :-/
>> I'm not tied to any particular implementation. Mostly I wanted to show
>> that we can take this ~5 minute portion of devstack and turn it into a
>> 15 second portion of devstack by improving our use of the service APIs
>> (and possibly even further if we apply it to all of the api
>> interaction). Any idea how difficult it would be to get your client as
>> a service stuff running in devstack again?
> I wish I could take credit, but this is actually Dan Berrange's work. :-)
>> I do not think we should make a one off change like I've done in my
>> POC. That will just end up being harder to understand and debug in the
>> future since it will be different than all of the other API
>> interaction. I like the idea of a manifest or feeding a longer lived
>> process api update commands as we can then avoid requesting new tokens
>> as well as pkg_resource startup time. Such a system could be used by
>> all of devstack as well (avoiding the "this bit is special" problem).
>> Is there any interest from the QA team in committing to an approach
>> and working to do a conversion? I don't want to commit any more time
>> to this myself unless there is strong interest in getting changes
>> merged (as I expect it will be a slow process weeding out places where
>> we've made bad assumptions particularly around plugins).
>> One of the things I found was that using names with osc results in
>> name to id lookups as well. We can avoid these entirely if we remember
>> name to id mappings instead (which my POC does). Any idea if your osc
>> as a service tool does or can do that? Probably have to be more
>> careful for scoping things in a tool like that as it may be reused by
>> people with name collisions across projects/users/groups/domains.
> I don't believe this would handle name to id mapping. It's a very thin
> wrapper around the regular client code that just makes it persistent so
> we don't pay the startup costs every call. On the plus side that means
> it basically works like the vanilla client, on the minus side that means
> it may not provide as much improvement as a more targeted solution.
> IIRC it's pretty easy to use, so I can try it out again and make sure it
> still works and still provides a performance benefit.
It still works and it still helps. Using the osc service cut about 3
minutes off my 21 minute devstack run. Subjectively I would say that
most of the time was being spent cloning and installing services and
I guess the downside is that working around the OSC slowness in CI will
reduce developer motivation to fix the problem, which affects all users
too. Then again, this has been a problem for years and no one has fixed
it, so apparently that isn't a big enough lever to get things moving
>>> On 7/26/19 6:53 PM, Clark Boylan wrote:
>>>> Today I have been digging into devstack runtime costs to help Donny
>>>> Davis understand why tempest jobs sometimes timeout on the
>>>> FortNebula cloud. One thing I discovered was that the keystone user,
>>>> group, project, role, and domain setup  can take many minutes
>>>>  (in the examples here almost 5).
>>>> I've rewritten create_keystone_accounts to be a python tool  and
>>>> get the runtime for that subset of setup from ~100s to ~9s .Â I
>>>> imagine that if we applied this to the other create_X_accounts
>>>> functions we would see similar results.
>>>> I think this is so much faster because we avoid repeated costs in
>>>> openstack client including: python process startup, pkg_resource
>>>> disk scanning to find entrypoints, and needing to convert names to
>>>> IDs via the API every time osc is run. Given my change shows this
>>>> can be so much quicker is there any interest in modifying devstack
>>>> to be faster here? And if so what do we think an appropriate
>>>> approach would be?
>>>>  https://review.opendev.org/#/c/673108/
>>>> Note the jobs compared above all ran on rax-dfw.