[oslo][oslo-messaging][nova] Stein nova-api AMQP issue running under uWSGI
Is anyone running Stein nova-api under uWSGI? I'm trying to do this, in
exactly the same way as I did with Rocky, but encountering a new problem...
The symptom is intermittent failure of API operations that involve RPC
calls - notably getting the noVNC console URL for an instance. The
nova-api log shows RPC timeouts and "connection reset by peer" for AMQP.
I've ruled out environmental issues, as a Rocky installation on the same
hosts works fine. With oslo debug enabled, I see that the first API call
causes a AMQP connections to be established, and the RPC call gets its
reply, and all is well... but there are no heartbeats logged after that.
3 minutes later, rabbitmq-server reports that it has received no
heartbeats, and drops the connection. On a subsequent API call (but not
before), nova-api logging "connection reset by peer" and the RPC call
times out. With the Rocky code, once the first API call causes the AMQP
connections to be established, regular heartbeats start happening, the
connections are maintained, and subsequent API calls work as expected.
It seems like a lockup in a thread that's supposed to maintain the AMQP
connection, or something. I tried to use SIGUSR2 to trigger guru
meditation, but it seems that uWSGI has its own interpretation of
SIGUSR2, so nova-api never sees it (anyone know a way around this?).
Running nova-api as a standalone service, instead of under uWSGI, the
problem does not occur (I see the heartbeats, and everything works fine).
I've tried updating oslo-messaging (and python-amqp) on a Rocky
installation to the Stein version, but the problem does not reproduce.