osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Problem with octavia LBaaS and nova availability zones


Hi one and all!

Currently, we have an ussuri openstack installation via kolla-ansible.
The installation consists of three control nodes and two availability 
zones (az-1, az-2), each with three compute nodes:
- os-cpt-10[1-3] > az-1
- os-cpt-20[1-3] > az-2

az-1 is the default compute availability zone.

The compute nodes are hardwarewise equipped exactly the same.
As storage system a ceph cluster is used.

We wanted to also use Octavia LBaaS and got it up and running after some 
experimentation.
We also wanted to be able to choose the availibility zones when starting 
a loadbalancer. We added the az info with:

openstack --os-cloud service_octavia loadbalancer 
availabilityzoneprofile create --name az-1 --provider amphora 
--availability-zone-data '{"compute_zone": "az-1"}'
openstack --os-cloud service_octavia loadbalancer 
availabilityzoneprofile create --name az-2 --provider amphora 
--availability-zone-data '{"compute_zone": "az-2"}'
openstack --os-cloud service_octavia loadbalancer availabilityzone 
create --name az-1 --availabilityzoneprofile az-1
openstack --os-cloud service_octavia loadbalancer availabilityzone 
create --name az-2 --availabilityzoneprofile az-2

Creating a loadbalancer via cli:

openstack --os-cloud $LB_PROJECT loadbalancer create --flavor $LB_FLAVOR 
--name $LB_NAME --vip-subnet-id $(openstack --os-cloud $LB_PROJECT 
subnet list --name $LB_SUBNET -f value -c ID) --availability-zone $LB_AZ

works, as long as $LB_AZ == az-1.

If we want to start the loadbalancer in az-2, this fails with an error 
in octavia-worker.log:
...
2020-09-30 07:21:01.482 34 ERROR oslo_messaging.rpc.server 
octavia.common.exceptions.ComputeBuildException: Failed to build compute 
instance due to: {'code': 500, 'created': '2020-09-30T07:20:55Z', 
'message': 'No valid host was found. There are not enough hosts 
available.', 'details': 'Traceback (most recent call last):\n  File 
"/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, 
in schedule_and_build_instances\n    instance_uuids, 
return_alternates=True)\n  File 
"/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, 
in _schedule_instances\n    return_alternates=return_alternates)\n  File 
"/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 
42, in select_destinations\n    instance_uuids, return_objects, 
return_alternates)\n  File 
"/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, 
in select_destinations\n    return cctxt.call(ctxt, 
\'select_destinations\', **msg_args)\n  File 
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 
181, in call\n    transport_options=self.transport_options)\n  File 
"/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 
129, in _send\n    transport_options=transport_options)\n  File 
"/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 654, in send\n    transport_options=transport_options)\n  File 
"/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 644, in _send\n    raise 
result\nnova.exception_Remote.NoValidHost_Remote: No valid host was 
found. There are not enough hosts available.\nTraceback (most recent 
call last):\n\n  File 
"/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 
241, in inner\n    return func(*args, **kwargs)\n\n  File 
"/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 215, 
in select_destinations\n    allocation_request_version, 
return_alternates)\n\n  File 
"/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", 
line 96, in select_destinations\n    allocation_request_version, 
return_alternates)\n\n  File 
"/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", 
line 265, in _schedule\n    claimed_instance_uuids)\n\n  File 
"/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py", 
line 302, in _ensure_sufficient_hosts\n    raise 
exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No 
valid host was found. There are not enough hosts available.\n\n'}
...

We enabled debug logging for nova and found:

...
2020-10-13 13:50:49.193 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" acquired by 
"nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" 
:: waited 0.000s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.196 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "7b9846a9-f8be-4d60-92ad-0fb531e48e64" released by 
"nova.context.set_target_cell.<locals>.get_or_set_cached_cell_and_set_connections" 
:: held 0.003s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.217 33 DEBUG oslo_db.sqlalchemy.engines 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] MySQL server mode set to 
STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION 
_check_effective_sql_mode 
/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:304
2020-10-13 13:50:49.245 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "('os-cpt-103', 'os-cpt-103')" acquired by 
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update" 
:: waited 0.000s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.246 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state from compute node: 
ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", 
"model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, 
"sockets": 1, "cores": 24, "threads": 2}, "features": ["bmi1", "smep", 
"sha-ni", "fsgsbase", "adx", "cmov", "invtsc", "fpu", "mmx", "sep", 
"abm", "pni", "msr", "xsavec", "f16c", "fma", "nx", "pat", "sse4.1", 
"rdrand", "wbnoinvd", "vme", "lahf_lm", "cr8legacy", "xsave", "bmi2", 
"clzero", "mtrr", "arat", "amd-ssbd", "aes", "avx", "avx2", "cx8", 
"umip", "de", "ibpb", "misalignsse", "osvw", "perfctr_core", "pse36", 
"mce", "skinit", "syscall", "sse2", "apic", "fxsr_opt", "pse", "ht", 
"pdpe1gb", "rdtscp", "xsaves", "clflushopt", "pclmuldq", "sse4.2", 
"movbe", "smap", "ibs", "clwb", "xgetbv1", "sse4a", "cx16", "extapic", 
"wdt", "perfctr_nb", "tsc", "mca", "topoext", "pae", "fxsr", "lm", 
"cmp_legacy", "monitor", "3dnowprefetch", "ssse3", "pge", "popcnt", 
"rdseed", "mmxext", "tce", "clflush", "xsaveopt", "svm", 
"sse"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=94,free_disk_gb=99,free_ram_mb=206453,host='os-cpt-103',host_ip=172.20.1.13,hypervisor_hostname='os-cpt-103',hypervisor_type='QEMU',hypervisor_version=4002000,id=9,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=51200,metrics='[{"name": 
"cpu.user.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", 
"timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.iowait.percent", 
"timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.time", 
"timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 1373295770000000}, {"name": 
"cpu.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.frequency", 
"timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.user.time", 
"timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 386827280000000}, {"name": 
"cpu.idle.time", "timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 139696145240000000}, {"name": 
"cpu.idle.percent", "timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.iowait.time", 
"timestamp": "2020-10-13T13:50:07.846401", "source": 
"libvirt.LibvirtDriver", "value": 
3390650000000}]',numa_topology='{"nova_object.name": "NUMATopology", 
"nova_object.namespace": "nova", "nova_object.version": "1.2", 
"nova_object.data": {"cells": [{"nova_object.name": "NUMACell", 
"nova_object.namespace": "nova", "nova_object.version": "1.4", 
"nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 
46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 
257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], 
"siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 
32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 
22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], 
[7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, 
"total": 65959381, "used": 0, "reserved": 0}, "nova_object.changes": 
["used", "reserved", "size_kb", "total"]}, {"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, 
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", 
"reserved", "size_kb", "total"]}, {"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, 
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["used", 
"reserved", "size_kb", "total"]}], "network_metadata": 
{"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", 
"nova_object.version": "1.0", "nova_object.data": {"physnets": [], 
"tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, 
"nova_object.changes": ["mempages", "cpu_usage", "memory", 
"memory_usage", "id", "pinned_cpus", "pcpuset", "network_metadata", 
"siblings", "cpuset"]}]}, "nova_object.changes": 
["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=3,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='3',num_os_type_None='3',num_proj_e06101e290f24563a836e16909146ea0='3',num_task_None='3',num_vm_active='3'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:07Z,uuid=34993097-d180-49a0-ae3c-9111e6aa8968,vcpus=48,vcpus_used=7) 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
2020-10-13 13:50:49.250 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with aggregates: 
[Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with service dict: {'id': 27, 'uuid': 
'62ea00b1-e61d-4f84-8b35-c1d3d7a333e7', 'host': 'os-cpt-103', 'binary': 
'nova-compute', 'topic': 'compute', 'report_count': 294508, 'disabled': 
False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 
10, 13, 13, 50, 45, tzinfo=<iso8601.Utc>), 'forced_down': False, 
'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, 
tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 
50, 45, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
2020-10-13 13:50:49.251 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with instances: 
['c88b77f7-2c59-497b-9633-139e62ba2bb1', 
'635fbd95-dd64-4e0a-9767-a0e0a6958610', 
'a02369ab-9002-4bf8-b65c-03e93cbc7df5'] _locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
2020-10-13 13:50:49.252 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "('os-cpt-103', 'os-cpt-103')" released by 
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update" 
:: held 0.007s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.253 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "('os-cpt-101', 'os-cpt-101')" acquired by 
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update" 
:: waited 0.000s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.253 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state from compute node: 
ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", 
"model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, 
"sockets": 1, "cores": 24, "threads": 2}, "features": ["3dnowprefetch", 
"fpu", "sse4.1", "fma", "arat", "avx2", "bmi1", "sse2", "extapic", 
"adx", "aes", "sha-ni", "tce", "osvw", "invtsc", "sse", "xsave", "de", 
"movbe", "pse", "pdpe1gb", "clflush", "mmx", "wdt", "cmov", 
"perfctr_core", "skinit", "umip", "bmi2", "cx8", "amd-ssbd", 
"cmp_legacy", "perfctr_nb", "rdrand", "ibpb", "monitor", "mtrr", 
"clflushopt", "smap", "msr", "sep", "f16c", "pat", "avx", "xsavec", 
"mca", "apic", "pni", "xsaves", "cr8legacy", "popcnt", "svm", "clzero", 
"pae", "lm", "pclmuldq", "pge", "rdseed", "xgetbv1", "sse4.2", "ht", 
"rdtscp", "fxsr", "lahf_lm", "vme", "sse4a", "tsc", "misalignsse", 
"abm", "fxsr_opt", "mce", "syscall", "ssse3", "cx16", "ibs", "smep", 
"fsgsbase", "topoext", "wbnoinvd", "xsaveopt", "mmxext", "nx", "pse36", 
"clwb"]}',created_at=2020-09-08T09:22:00Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=64,free_disk_gb=79,free_ram_mb=207477,host='os-cpt-101',host_ip=172.20.1.11,hypervisor_hostname='os-cpt-101',hypervisor_type='QEMU',hypervisor_version=4002000,id=6,local_gb=99,local_gb_used=20,mapped=0,memory_mb=257653,memory_mb_used=50176,metrics='[{"name": 
"cpu.iowait.percent", "timestamp": "2020-10-13T13:50:21.645565", 
"source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": 
"cpu.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.idle.percent", 
"timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 0.98}, {"name": "cpu.frequency", 
"timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 1796}, {"name": "cpu.idle.time", 
"timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 139958595980000000}, {"name": 
"cpu.user.percent", "timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.kernel.percent", 
"timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 0.01}, {"name": "cpu.user.time", 
"timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 388866090000000}, {"name": 
"cpu.kernel.time", "timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 1118743070000000}, {"name": 
"cpu.iowait.time", "timestamp": "2020-10-13T13:50:21.645565", "source": 
"libvirt.LibvirtDriver", "value": 
3637460000000}]',numa_topology='{"nova_object.name": "NUMATopology", 
"nova_object.namespace": "nova", "nova_object.version": "1.2", 
"nova_object.data": {"cells": [{"nova_object.name": "NUMACell", 
"nova_object.namespace": "nova", "nova_object.version": "1.4", 
"nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 
46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 
257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], 
"siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 
32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 
22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], 
[7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, 
"total": 65959383, "used": 0, "reserved": 0}, "nova_object.changes": 
["reserved", "used", "total", "size_kb"]}, {"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, 
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": 
["reserved", "used", "total", "size_kb"]}, {"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, 
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": 
["reserved", "used", "total", "size_kb"]}], "network_metadata": 
{"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", 
"nova_object.version": "1.0", "nova_object.data": {"physnets": [], 
"tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, 
"nova_object.changes": ["id", "cpu_usage", "pcpuset", "memory", 
"cpuset", "siblings", "mempages", "network_metadata", "memory_usage", 
"pinned_cpus"]}]}, "nova_object.changes": 
["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=4,service_id=None,stats={failed_builds='0',io_workload='0',num_instances='4',num_os_type_None='4',num_proj_05af2e78169748f69938d01b5238fc8b='1',num_proj_882ea7b2a43a41819ef796f797cdbe82='1',num_proj_d52e2dccbc7e46d8b518120fa7c8753a='1',num_proj_e06101e290f24563a836e16909146ea0='1',num_task_None='4',num_vm_active='4'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:21Z,uuid=5ab97e91-6b03-48f3-a320-c8cbb032cd3a,vcpus=48,vcpus_used=7) 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with aggregates: 
[Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
2020-10-13 13:50:49.257 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with service dict: {'id': 21, 'uuid': 
'c178b502-9b8c-43f3-9053-e3d0e2405c89', 'host': 'os-cpt-101', 'binary': 
'nova-compute', 'topic': 'compute', 'report_count': 294504, 'disabled': 
False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 
10, 13, 13, 50, 43, tzinfo=<iso8601.Utc>), 'forced_down': False, 
'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 22, 
tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 
50, 43, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
2020-10-13 13:50:49.258 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with instances: 
['96f4c101-d75b-44b6-8af9-dd75068d207d', 
'121c5df5-2d8b-48e0-a270-9f6f79193dd6', 
'e235063d-f7a8-4f38-8294-d7a289167348', 
'425b53c3-f6f5-43e9-beeb-25c9b6ead1f9'] _locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
2020-10-13 13:50:49.258 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "('os-cpt-101', 'os-cpt-101')" released by 
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update" 
:: held 0.005s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.259 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "('os-cpt-102', 'os-cpt-102')" acquired by 
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update" 
:: waited 0.000s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:358
2020-10-13 13:50:49.260 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state from compute node: 
ComputeNode(cpu_allocation_ratio=4.0,cpu_info='{"arch": "x86_64", 
"model": "EPYC-IBPB", "vendor": "AMD", "topology": {"cells": 1, 
"sockets": 1, "cores": 24, "threads": 2}, "features": ["lahf_lm", 
"clflush", "osvw", "extapic", "vme", "wdt", "monitor", "msr", "adx", 
"pse36", "sse4a", "fma", "pat", "mce", "sse2", "nx", "f16c", "mca", 
"xsaveopt", "avx", "syscall", "rdrand", "clwb", "ssse3", "xsavec", 
"invtsc", "bmi2", "fpu", "movbe", "aes", "bmi1", "cr8legacy", "mmxext", 
"ibpb", "amd-ssbd", "skinit", "topoext", "umip", "cmp_legacy", "arat", 
"lm", "svm", "fxsr", "ibs", "pae", "misalignsse", "mtrr", "sep", "ht", 
"smap", "xgetbv1", "clzero", "pdpe1gb", "apic", "abm", "pge", "pni", 
"tsc", "xsaves", "wbnoinvd", "sse4.2", "cmov", "cx8", "pse", "rdtscp", 
"cx16", "sse", "sse4.1", "fxsr_opt", "popcnt", "sha-ni", "perfctr_core", 
"fsgsbase", "avx2", "mmx", "rdseed", "clflushopt", "pclmuldq", 
"perfctr_nb", "smep", "de", "3dnowprefetch", "xsave", 
"tce"]}',created_at=2020-09-08T09:21:58Z,current_workload=0,deleted=False,deleted_at=None,disk_allocation_ratio=1.0,disk_available_least=93,free_disk_gb=99,free_ram_mb=216693,host='os-cpt-102',host_ip=172.20.1.12,hypervisor_hostname='os-cpt-102',hypervisor_type='QEMU',hypervisor_version=4002000,id=3,local_gb=99,local_gb_used=0,mapped=0,memory_mb=257653,memory_mb_used=40960,metrics='[{"name": 
"cpu.user.time", "timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 161246250000000}, {"name": 
"cpu.kernel.percent", "timestamp": "2020-10-13T13:50:03.450583", 
"source": "libvirt.LibvirtDriver", "value": 0.0}, {"name": 
"cpu.user.percent", "timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.percent", 
"timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 0.99}, {"name": "cpu.percent", 
"timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 0.0}, {"name": "cpu.idle.time", 
"timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 140551238730000000}, {"name": 
"cpu.frequency", "timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 3343}, {"name": "cpu.kernel.time", 
"timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 366274340000000}, {"name": 
"cpu.iowait.time", "timestamp": "2020-10-13T13:50:03.450583", "source": 
"libvirt.LibvirtDriver", "value": 3829690000000}, {"name": 
"cpu.iowait.percent", "timestamp": "2020-10-13T13:50:03.450583", 
"source": "libvirt.LibvirtDriver", "value": 
0.0}]',numa_topology='{"nova_object.name": "NUMATopology", 
"nova_object.namespace": "nova", "nova_object.version": "1.2", 
"nova_object.data": {"cells": [{"nova_object.name": "NUMACell", 
"nova_object.namespace": "nova", "nova_object.version": "1.4", 
"nova_object.data": {"id": 0, "cpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 
46, 47], "pcpuset": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], "memory": 
257653, "cpu_usage": 0, "memory_usage": 0, "pinned_cpus": [], 
"siblings": [[12, 36], [10, 34], [2, 26], [1, 25], [9, 33], [0, 24], [8, 
32], [11, 35], [13, 37], [19, 43], [21, 45], [18, 42], [38, 14], [46, 
22], [4, 28], [20, 44], [15, 39], [17, 41], [23, 47], [16, 40], [6, 30], 
[7, 31], [3, 27], [5, 29]], "mempages": [{"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 4, 
"total": 65959382, "used": 0, "reserved": 0}, "nova_object.changes": 
["total", "used", "reserved", "size_kb"]}, {"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 2048, 
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", 
"used", "reserved", "size_kb"]}, {"nova_object.name": 
"NUMAPagesTopology", "nova_object.namespace": "nova", 
"nova_object.version": "1.1", "nova_object.data": {"size_kb": 1048576, 
"total": 0, "used": 0, "reserved": 0}, "nova_object.changes": ["total", 
"used", "reserved", "size_kb"]}], "network_metadata": 
{"nova_object.name": "NetworkMetadata", "nova_object.namespace": "nova", 
"nova_object.version": "1.0", "nova_object.data": {"physnets": [], 
"tunneled": false}, "nova_object.changes": ["physnets", "tunneled"]}}, 
"nova_object.changes": ["cpu_usage", "pinned_cpus", "mempages", 
"memory", "siblings", "network_metadata", "cpuset", "memory_usage", 
"id", "pcpuset"]}]}, "nova_object.changes": 
["cells"]}',pci_device_pools=PciDevicePoolList,ram_allocation_ratio=1.2,running_vms=0,service_id=None,stats={failed_builds='0'},supported_hv_specs=[HVSpec,HVSpec,HVSpec,HVSpec],updated_at=2020-10-13T13:50:03Z,uuid=81c2e888-56e4-4469-ab93-f204fa85a5c5,vcpus=48,vcpus_used=2) 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:172
2020-10-13 13:50:49.263 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with aggregates: 
[Aggregate(created_at=2020-09-08T14:01:05Z,deleted=False,deleted_at=None,hosts=['os-cpt-101','os-cpt-102','os-cpt-103'],id=3,metadata={availability_zone='az-1'},name='az-1',updated_at=None,uuid=8f37e9f3-506d-40da-8997-38915f1dfe67)] 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:175
2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with service dict: {'id': 18, 'uuid': 
'342a2d9b-4b32-45a2-ba16-b808343b3b8f', 'host': 'os-cpt-102', 'binary': 
'nova-compute', 'topic': 'compute', 'report_count': 294494, 'disabled': 
False, 'disabled_reason': None, 'last_seen_up': datetime.datetime(2020, 
10, 13, 13, 50, 49, tzinfo=<iso8601.Utc>), 'forced_down': False, 
'version': 51, 'created_at': datetime.datetime(2020, 9, 8, 9, 21, 58, 
tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 13, 13, 
50, 49, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'deleted': False} 
_locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:178
2020-10-13 13:50:49.264 33 DEBUG nova.scheduler.host_manager 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Update host state with instances: [] _locked_update 
/usr/lib/python3.6/site-packages/nova/scheduler/host_manager.py:182
2020-10-13 13:50:49.265 33 DEBUG oslo_concurrency.lockutils 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Lock "('os-cpt-102', 'os-cpt-102')" released by 
"nova.scheduler.host_manager.HostState.update.<locals>._locked_update" 
:: held 0.005s inner 
/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:370
2020-10-13 13:50:49.266 33 DEBUG nova.filters 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Starting with 3 host(s) get_filtered_objects 
/usr/lib/python3.6/site-packages/nova/filters.py:70
2020-10-13 13:50:49.266 33 DEBUG 
nova.scheduler.filters.availability_zone_filter 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Availability Zone 'az-2' requested. (os-cpt-101, 
os-cpt-101) ram: 207477MB disk: 65536MB io_ops: 0 instances: 4 has AZs: 
{'az-1'} host_passes 
/usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
2020-10-13 13:50:49.267 33 DEBUG 
nova.scheduler.filters.availability_zone_filter 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Availability Zone 'az-2' requested. (os-cpt-102, 
os-cpt-102) ram: 216693MB disk: 95232MB io_ops: 0 instances: 0 has AZs: 
{'az-1'} host_passes 
/usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
2020-10-13 13:50:49.267 33 DEBUG 
nova.scheduler.filters.availability_zone_filter 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Availability Zone 'az-2' requested. (os-cpt-103, 
os-cpt-103) ram: 206453MB disk: 96256MB io_ops: 0 instances: 3 has AZs: 
{'az-1'} host_passes 
/usr/lib/python3.6/site-packages/nova/scheduler/filters/availability_zone_filter.py:61
2020-10-13 13:50:49.268 33 INFO nova.filters 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Filter AvailabilityZoneFilter returned 0 hosts
2020-10-13 13:50:49.269 33 DEBUG nova.filters 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Filtering removed all hosts for the request with 
instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: 
[('AvailabilityZoneFilter', None)] get_filtered_objects 
/usr/lib/python3.6/site-packages/nova/filters.py:115
2020-10-13 13:50:49.269 33 INFO nova.filters 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Filtering removed all hosts for the request with 
instance ID 'd18bf174-9ccb-4721-b7d8-36cd31834f62'. Filter results: 
['AvailabilityZoneFilter: (start: 3, end: 0)']
2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] Filtered [] _get_sorted_hosts 
/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:443
2020-10-13 13:50:49.270 33 DEBUG nova.scheduler.filter_scheduler 
[req-390e67a0-5078-4b5d-a2b5-0111e8a1527b 
f181ae7dec9b493f9e9311f3c5cc60e2 882ea7b2a43a41819ef796f797cdbe82 - 
default default] There are 0 hosts available but 1 instances requested 
to build. _ensure_sufficient_hosts 
/usr/lib/python3.6/site-packages/nova/scheduler/filter_scheduler.py:300
...

So nova seems to only consider compute nodes in az-1.

The amphora image is available in both availability zones. Starting an 
instance from the image directly in az-2 works.

Does anybody have any idea on how to debug this any further? Any hints, 
tips or ideas would be greatly appreciated!

With kind regards,
Björn Puttmann