[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached.

On 8/6/2019 7:18 AM, Mark Goddard wrote:
> We do use a larger timeout for glance-api 
> (haproxy_glance_api_client_timeout 
> and haproxy_glance_api_server_timeout, both 6h). Perhaps we need 
> something similar for cinder-api.

A 6 hour timeout for cinder API calls would be nuts IMO. The thing that 
was failing was a volume attachment delete/create from what I recall, 
which is the newer version (as of Ocata?) for the old 
initialize_connection/terminate_connection APIs. These are synchronous 
RPC calls from cinder-api to cinder-volume to do things on the storage 
backend and we have seen them take longer than 60 seconds in the gate CI 
runs with the lvm driver. I think the investigation normally turned up 
lvchange taking over 60 seconds on some concurrent operation locking out 
the RPC call which eventually results in the MessagingTimeout from 
oslo.messaging. That's unrelated to your gateway timeout from HAProxy 
but the point is yeah you likely want to bump up those timeouts since 
cinder-api has these synchronous calls to the cinder-volume service. I 
just don't think you need to go to 6 hours :). I think the keystoneauth1 
default http response timeout is 10 minutes so maybe try that.