[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[kolla][nova][cinder] Got Gateway-Timeout error on VM evacuation if it has volume attached.

Hi Mark, thanks for suggestion.

I think this, too. Cinder-api may normal but HAproxy could be very busy
since one controller down.
I'll try to increase the value about cinder-api timeout.

Mark Goddard <mark at stackhpc.com> æ?¼ 2019å¹´8æ??7æ?¥ é?±ä¸? ä¸?å??12:06寫é??ï¼?

> On Tue, 6 Aug 2019 at 16:33, Matt Riedemann <mriedemos at gmail.com> wrote:
>> On 8/6/2019 7:18 AM, Mark Goddard wrote:
>> > We do use a larger timeout for glance-api
>> > (haproxy_glance_api_client_timeout
>> > and haproxy_glance_api_server_timeout, both 6h). Perhaps we need
>> > something similar for cinder-api.
>> A 6 hour timeout for cinder API calls would be nuts IMO. The thing that
>> was failing was a volume attachment delete/create from what I recall,
>> which is the newer version (as of Ocata?) for the old
>> initialize_connection/terminate_connection APIs. These are synchronous
>> RPC calls from cinder-api to cinder-volume to do things on the storage
>> backend and we have seen them take longer than 60 seconds in the gate CI
>> runs with the lvm driver. I think the investigation normally turned up
>> lvchange taking over 60 seconds on some concurrent operation locking out
>> the RPC call which eventually results in the MessagingTimeout from
>> oslo.messaging. That's unrelated to your gateway timeout from HAProxy
>> but the point is yeah you likely want to bump up those timeouts since
>> cinder-api has these synchronous calls to the cinder-volume service. I
>> just don't think you need to go to 6 hours :). I think the keystoneauth1
>> default http response timeout is 10 minutes so maybe try that.
> Yeah, wasn't advocating for 6 hours - just showing which knobs are
> available :)
>> --
>> Thanks,
>> Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190808/679def98/attachment.html>