osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Nova] migration issue


Dear all, 

we are testing the cold migratoin of instances that are using local ephemeral storage and it fails with the following error:


2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae
58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after 3 seconds.
2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed successfully.
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-
00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command.
Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph
0
Exit code: 1
Stdout: u''
Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured syst
em and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *
\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No suc
h file or directory\n'
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call last):
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 75
18, in _error_out_instance_on_exception
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     yield
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 42
75, in _resize_instance
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     timeout, retry_interval)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin
e 8200, in migrate_disk_and_power_off
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     shared_storage)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220
, in __exit__
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     self.force_reraise()
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196
, in force_reraise
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     six.reraise(self.type_, self.value, self.tb)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin
e 8185, in migrate_disk_and_power_off
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line
 226, in copy_image
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs
.py", line 110, in copy_file
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     compression=compression)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs
.py", line 196, in copy_file
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     on_execute=on_execute, on_completion=on_completion)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in exec
ute
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     return processutils.execute(*cmd, **kwargs)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py"
, line 424, in execute
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]     cmd=sanitized_cmd)
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: Unexpected error while running command.
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u''
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n'
2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]
2019-07-24 13:47:14.380 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration da66c416-2636-421b-99cb-470bd07dad4c for instance
2019-07-24 13:47:14.979 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task state from resize_migrating on failure for instance.
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
Exit code: 1
Stdout: u''
Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n'
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in resize_instance
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self._revert_allocation(context, instance, migration)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in resize_instance
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     instance_type, clean_shutdown)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in _resize_instance
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     timeout, retry_interval)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in migrate_disk_and_power_off
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     shared_storage)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in migrate_disk_and_power_off
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     compression=compression)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     on_execute=on_execute, on_completion=on_completion)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in execute
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server   File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u''
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stderr: u'------------------------------------------------------------------------------\n* WARNING                                                                    *\n* You are accessing a secured system and your actions will be logged along   *\n* with identifying information. Disconnect immediately if you are not an     *\n* authorized user of this system.                                            *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n'
2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image ca89475a-cf15-46d1-ab2d-c2b96c403248 at (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): checking
2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base files: /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07
2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped (Lifecycle Event)
2019-07-24 13:47:27.918 58902 INFO nova.compute.manager [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.


The disk has its root FS n an RBD device (a volume created by cinder) and it has its ephemeral disk on a local LVM volume created by nova.

It seems that the /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other side.

Any ideas how to fix this?

The nova config on the nodes looks like this:

# Ansible managed
[DEFAULT]
allow_resize_to_same_host = True
# Compute
compute_driver = libvirt.LibvirtDriver
# Scheduler
cpu_allocation_ratio = 2.0
# Logs / State
debug = False
# Hypervisor
default_ephemeral_format = ext4
disk_allocation_ratio = 1.0
# Api's
enabled_apis = osapi_compute,metadata
executor_thread_pool_size = 64
fatal_deprecations = False
# Configdrive
force_config_drive = False
host = compute8.mgmt.lab.mydomain.com
image_cache_manager_interval = 0
instance_name_template = instance-%08x
# Ceilometer notification configurations
instance_usage_audit = True
instance_usage_audit_period = hour
instances_path = /var/lib/nova/instances
## Vif
libvirt_vif_type = ethernet
log_dir = /var/log/nova
# Metadata
metadata_workers = 16
# Network
my_ip = 192.168.56.47
notify_on_state_change = vm_and_task_state
osapi_compute_workers = 16
ram_allocation_ratio = 1.0
reserved_host_disk_mb = 0
reserved_host_memory_mb = 2048
resume_guests_state_on_host_boot = False
rootwrap_config = /etc/nova/rootwrap.conf
rpc_response_timeout = 60
service_down_time = 120
state_path = /var/lib/nova
# Rpc all
transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e
8cf8bd7 at 192.168.56.179:5671//nova
# Disable stderr logging
use_stderr = False
vif_plugging_is_fatal = True
vif_plugging_timeout = 30

[api]
auth_strategy = keystone
enable_instance_password = True
use_forwarded_for = False
vendordata_jsonfile_path = /etc/nova/vendor_data.json

# Cache
[cache]
backend = oslo_cache.memcache_pool
enabled = true
memcache_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211

# Cinder
[cinder]
cafile = /etc/ssl/certs/ca-certificates.crt
catalog_info = volumev3:cinderv3:internalURL
cross_az_attach = True
os_region_name = A_Lab

[conductor]
workers = 16

[filter_scheduler]
available_filters = nova.scheduler.filters.all_filters
enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, AggregateIoOpsFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, DifferentHostFilter
host_subset_size = 10
max_instances_per_host = 50
max_io_ops_per_host = 10
ram_weight_multiplier = 5.0
tracks_instance_changes = True
weight_classes = nova.scheduler.weights.all_weighers

# Glance
[glance]
api_servers = https://vip.mgmt.lab.mydomain.com:9292
cafile = /etc/ssl/certs/ca-certificates.crt

[key_manager]
fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[keystone_authtoken]
auth_type = password
auth_uri = https://vip.mgmt.lab.mydomain.com:5000
auth_url = https://vip.mgmt.lab.mydomain.com:35357
cafile = /etc/ssl/certs/ca-certificates.crt
insecure = False
memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# if your memcached server is shared, use these settings to avoid cache poisoning
memcache_security_strategy = ENCRYPT
memcached_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211
password = xxxxxxxxxxxxxxxxxxxxxxx

project_domain_id = default
project_name = service
region_name = A_Lab
token_cache_time = 300
user_domain_id = default
username = nova

[libvirt]
disk_cachemodes = network=writeback
hw_disk_discard = unmap
images_rbd_ceph_conf = /etc/ceph/ceph.conf
images_rbd_pool = vms
images_type = lvm
images_volume_group = ssd_nova
inject_key = False
inject_partition = -2
inject_password = False
live_migration_tunnelled = True
live_migration_uri = "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa"
rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# ceph rbd support
rbd_user = cinder
remove_unused_resized_minimum_age_seconds = 3600
use_virtio_for_bridges = True
virt_type = kvm

# Neutron
[neutron]
auth_type = password
# Keystone client plugin authentication URL option
auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3
cafile = /etc/ssl/certs/ca-certificates.crt
default_floating_pool = public
insecure = False
metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Keystone client plugin password option
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
project_domain_name = Default
project_name = service
region_name = A_Lab
service_metadata_proxy = True
url = https://vip.mgmt.lab.mydomain.com:9696
user_domain_name = Default
# Keystone client plugin username option
username = neutron

[oslo_concurrency]
lock_path = /var/lock/nova
# Notifications
[oslo_messaging_notifications]
driver = messagingv2
notification_topics = notifications
transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7 at 192.168.56.179:5671//nova

[oslo_messaging_rabbit]
rpc_conn_pool_size = 30
ssl = True

# Placement
[placement]
auth_type = "password"
auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3
cafile = /etc/ssl/certs/ca-certificates.crt
insecure = False
os_interface = internal
os_region_name = A_Lab
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
project_domain_name = Default
project_name = service
user_domain_name = Default
username = placement

[quota]
cores = 20
injected_file_content_bytes = 10240
injected_file_path_length = 255
injected_files = 5
instances = 10
key_pairs = 100
max_age = 0
metadata_items = 128
ram = 32768
server_group_members = 10
server_groups = 10

[scheduler]
discover_hosts_in_cells_interval = 60
host_manager = host_manager
max_attempts = 5
periodic_task_interval = 60
scheduler_driver = filter_scheduler

[spice]
agent_enabled = True
enabled = True
# Console Url and binds
html5proxy_base_url = https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html
server_listen = 192.168.56.47
server_proxyclient_address = 192.168.56.47

[upgrade_levels]
compute = auto

[vnc]
enabled = False

[wsgi]
api_paste_config = /etc/nova/api-paste.ini
secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO


Thank you in advance for any suggestions.

Kind regards,
Laszlo