osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Nova] migration issue


Hi,

is this the first migration test or have there already been some  
successful migrations? If it's the first I would suggest to check if  
the nova user can access the other node via ssh without password. Is  
/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova  
user?

Regards,
Eugen


Zitat von Budai Laszlo <laszlo.budai at gmail.com>:

> Dear all,
>
> we are testing the cold migratoin of instances that are using local  
> ephemeral storage and it fails with the following error:
>
>
> 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver  
> [req-356fce31-5e98-425f-89d6-8a98664d31ad  
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -  
> default default] [instance: 5a6dae
> 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after  
> 3 seconds.
> 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-]  
> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed  
> successfully.
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager  
> [req-356fce31-5e98-425f-89d6-8a98664d31ad  
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -  
> default default] [instance: 5a6dae58-
> 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR:  
> ProcessExecutionError: Unexpected error while running command.
> Command: scp -r  
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0  
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph
>
> Exit code: 1
> Stdout: u''
> Stderr:  
> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured  
> syst
> em and your actions will be logged along   *\n* with identifying  
> information. Disconnect immediately if you are not an     *\n*  
> authorized user of this system.                                       
>       *
> \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No  
> suc
> h file or directory\n'
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call  
> last):
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line  
> 75
> 18, in _error_out_instance_on_exception
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     yield
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line  
> 42
> 75, in _resize_instance
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     timeout, retry_interval)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",  
> lin
> e 8200, in migrate_disk_and_power_off
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     shared_storage)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line  
> 220
> , in __exit__
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     self.force_reraise()
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line  
> 196
> , in force_reraise
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     six.reraise(self.type_,  
> self.value, self.tb)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",  
> lin
> e 8185, in migrate_disk_and_power_off
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py",  
> line
>  226, in copy_image
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs
> .py", line 110, in copy_file
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs
> .py", line 196, in copy_file
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     on_execute=on_execute,  
> on_completion=on_completion)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py",  
> line 231, in exec
> ute
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     return  
> processutils.execute(*cmd, **kwargs)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py"
> , line 424, in execute
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]     cmd=sanitized_cmd)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError:  
> Unexpected error while running command.
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r  
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0  
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u''
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr:  
> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or  
> directory\n'
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49]
> 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager  
> [req-356fce31-5e98-425f-89d6-8a98664d31ad  
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -  
> default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]  
> Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held  
> by migration da66c416-2636-421b-99cb-470bd07dad4c for instance
> 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager  
> [req-356fce31-5e98-425f-89d6-8a98664d31ad  
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -  
> default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]  
> Successfully reverted task state from resize_migrating on failure  
> for instance.
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server  
> [req-356fce31-5e98-425f-89d6-8a98664d31ad  
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -  
> default default] Exception during message handling:  
> ProcessExecutionError: Unexpected error while running command.
> Command: scp -r  
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0  
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> Exit code: 1
> Stdout: u''
> Stderr:  
> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or  
> directory\n'
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server  
> Traceback (most recent call last):
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in  
> _process_incoming
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> res = self.dispatcher.dispatch(message)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in  
> dispatch
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> return self._do_dispatch(endpoint, method, ctxt, args)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in  
> _do_dispatch
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> result = func(ctxt, **new_args)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in  
> wrapped
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> function_name, call_dict, binary)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in  
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in  
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in  
> wrapped
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> return f(self, context, *args, **kw)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in  
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> "Error: %s", e, instance=instance)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in  
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in  
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in  
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> return function(self, context, *args, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in  
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> return function(self, context, *args, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in  
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> kwargs['instance'], e, sys.exc_info())
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in  
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in  
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in  
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> return function(self, context, *args, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in  
> resize_instance
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> self._revert_allocation(context, instance, migration)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in  
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in  
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in  
> resize_instance
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> instance_type, clean_shutdown)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in  
> _resize_instance
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> timeout, retry_interval)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in  
> migrate_disk_and_power_off
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> shared_storage)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in  
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in  
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in  
> migrate_disk_and_power_off
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> compression=compression)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in  
> copy_image
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> compression=compression)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in  
> copy_file
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> compression=compression)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in  
> copy_file
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> on_execute=on_execute, on_completion=on_completion)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py",  
> line 231, in execute
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> return processutils.execute(*cmd, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File  
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in  
> execute
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server      
> cmd=sanitized_cmd)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server  
> ProcessExecutionError: Unexpected error while running command.
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server  
> Command: scp -r  
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0  
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u''
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server  
> Stderr:  
> u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or  
> directory\n'
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache  
> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image  
> ca89475a-cf15-46d1-ab2d-c2b96c403248 at  
> (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07):  
> checking
> 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache  
> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base  
> files:  
> /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07
> 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-]  
> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped  
> (Lifecycle Event)
> 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager  
> [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance:  
> 5a6dae58-00d6-4317-b635-909fdf09ac49] During  
> _sync_instance_power_state the DB power_state (1) does not match the  
> vm_power_state from the hypervisor (4). Updating power_state in the  
> DB to match the hypervisor.
>
>
> The disk has its root FS n an RBD device (a volume created by  
> cinder) and it has its ephemeral disk on a local LVM volume created  
> by nova.
>
> It seems that the  
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other  
> side.
>
> Any ideas how to fix this?
>
> The nova config on the nodes looks like this:
>
> # Ansible managed
> [DEFAULT]
> allow_resize_to_same_host = True
> # Compute
> compute_driver = libvirt.LibvirtDriver
> # Scheduler
> cpu_allocation_ratio = 2.0
> # Logs / State
> debug = False
> # Hypervisor
> default_ephemeral_format = ext4
> disk_allocation_ratio = 1.0
> # Api's
> enabled_apis = osapi_compute,metadata
> executor_thread_pool_size = 64
> fatal_deprecations = False
> # Configdrive
> force_config_drive = False
> host = compute8.mgmt.lab.mydomain.com
> image_cache_manager_interval = 0
> instance_name_template = instance-%08x
> # Ceilometer notification configurations
> instance_usage_audit = True
> instance_usage_audit_period = hour
> instances_path = /var/lib/nova/instances
> ## Vif
> libvirt_vif_type = ethernet
> log_dir = /var/log/nova
> # Metadata
> metadata_workers = 16
> # Network
> my_ip = 192.168.56.47
> notify_on_state_change = vm_and_task_state
> osapi_compute_workers = 16
> ram_allocation_ratio = 1.0
> reserved_host_disk_mb = 0
> reserved_host_memory_mb = 2048
> resume_guests_state_on_host_boot = False
> rootwrap_config = /etc/nova/rootwrap.conf
> rpc_response_timeout = 60
> service_down_time = 120
> state_path = /var/lib/nova
> # Rpc all
> transport_url =  
> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e
> 8cf8bd7 at 192.168.56.179:5671//nova
> # Disable stderr logging
> use_stderr = False
> vif_plugging_is_fatal = True
> vif_plugging_timeout = 30
>
> [api]
> auth_strategy = keystone
> enable_instance_password = True
> use_forwarded_for = False
> vendordata_jsonfile_path = /etc/nova/vendor_data.json
>
> # Cache
> [cache]
> backend = oslo_cache.memcache_pool
> enabled = true
> memcache_servers =  
> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211
>
> # Cinder
> [cinder]
> cafile = /etc/ssl/certs/ca-certificates.crt
> catalog_info = volumev3:cinderv3:internalURL
> cross_az_attach = True
> os_region_name = A_Lab
>
> [conductor]
> workers = 16
>
> [filter_scheduler]
> available_filters = nova.scheduler.filters.all_filters
> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter,  
> AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter,  
> AggregateDiskFilter, AggregateNumInstancesFilter,  
> AggregateIoOpsFilter, ComputeCapabilitiesFilter,  
> ImagePropertiesFilter, ServerGroupAntiAffinityFilter,  
> ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter,  
> DifferentHostFilter
> host_subset_size = 10
> max_instances_per_host = 50
> max_io_ops_per_host = 10
> ram_weight_multiplier = 5.0
> tracks_instance_changes = True
> weight_classes = nova.scheduler.weights.all_weighers
>
> # Glance
> [glance]
> api_servers = https://vip.mgmt.lab.mydomain.com:9292
> cafile = /etc/ssl/certs/ca-certificates.crt
>
> [key_manager]
> fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> [keystone_authtoken]
> auth_type = password
> auth_uri = https://vip.mgmt.lab.mydomain.com:5000
> auth_url = https://vip.mgmt.lab.mydomain.com:35357
> cafile = /etc/ssl/certs/ca-certificates.crt
> insecure = False
> memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> # if your memcached server is shared, use these settings to avoid  
> cache poisoning
> memcache_security_strategy = ENCRYPT
> memcached_servers =  
> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211
> password = xxxxxxxxxxxxxxxxxxxxxxx
>
> project_domain_id = default
> project_name = service
> region_name = A_Lab
> token_cache_time = 300
> user_domain_id = default
> username = nova
>
> [libvirt]
> disk_cachemodes = network=writeback
> hw_disk_discard = unmap
> images_rbd_ceph_conf = /etc/ceph/ceph.conf
> images_rbd_pool = vms
> images_type = lvm
> images_volume_group = ssd_nova
> inject_key = False
> inject_partition = -2
> inject_password = False
> live_migration_tunnelled = True
> live_migration_uri =  
> "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa"
> rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> # ceph rbd support
> rbd_user = cinder
> remove_unused_resized_minimum_age_seconds = 3600
> use_virtio_for_bridges = True
> virt_type = kvm
>
> # Neutron
> [neutron]
> auth_type = password
> # Keystone client plugin authentication URL option
> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3
> cafile = /etc/ssl/certs/ca-certificates.crt
> default_floating_pool = public
> insecure = False
> metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> # Keystone client plugin password option
> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> project_domain_name = Default
> project_name = service
> region_name = A_Lab
> service_metadata_proxy = True
> url = https://vip.mgmt.lab.mydomain.com:9696
> user_domain_name = Default
> # Keystone client plugin username option
> username = neutron
>
> [oslo_concurrency]
> lock_path = /var/lock/nova
> # Notifications
> [oslo_messaging_notifications]
> driver = messagingv2
> notification_topics = notifications
> transport_url =  
> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova
>
> [oslo_messaging_rabbit]
> rpc_conn_pool_size = 30
> ssl = True
>
> # Placement
> [placement]
> auth_type = "password"
> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3
> cafile = /etc/ssl/certs/ca-certificates.crt
> insecure = False
> os_interface = internal
> os_region_name = A_Lab
> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> project_domain_name = Default
> project_name = service
> user_domain_name = Default
> username = placement
>
> [quota]
> cores = 20
> injected_file_content_bytes = 10240
> injected_file_path_length = 255
> injected_files = 5
> instances = 10
> key_pairs = 100
> max_age = 0
> metadata_items = 128
> ram = 32768
> server_group_members = 10
> server_groups = 10
>
> [scheduler]
> discover_hosts_in_cells_interval = 60
> host_manager = host_manager
> max_attempts = 5
> periodic_task_interval = 60
> scheduler_driver = filter_scheduler
>
> [spice]
> agent_enabled = True
> enabled = True
> # Console Url and binds
> html5proxy_base_url =  
> https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html
> server_listen = 192.168.56.47
> server_proxyclient_address = 192.168.56.47
>
> [upgrade_levels]
> compute = auto
>
> [vnc]
> enabled = False
>
> [wsgi]
> api_paste_config = /etc/nova/api-paste.ini
> secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO
>
>
> Thank you in advance for any suggestions.
>
> Kind regards,
> Laszlo