Database/Cloudstack management cluster issue?

Hello

 

When there is a DB node failure and the database briefly disconnects, the following entry keeps appearing in the cloudstack-management.log until the management service is restarted. All Cloudstack operations still function – starting and stopping an instance for example

 

 

Cloudstack 4.9.3.1

Centos 7

 

3 CS management servers

3 active/active mariadb galera cluster - loadbalanced 1 active 2 standby

 

2018-05-02 09:27:40,685 ERROR [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-64ee96a9) (logid:d15e998c) Unexpected exception in cluster heartbeat

java.lang.RuntimeException: No database selected for the transaction

        at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:148)

        at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

        at com.sun.proxy.$Proxy202.update(Unknown Source)

        at com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:554)

        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)

        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)

        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)

        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)

        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: com.cloud.utils.exception.CloudRuntimeException: No database selected for the transaction

        at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:580)

        at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:467)

        at com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:460)

        ... 26 more

 

 

Kind Regards,

Adam Witwicki | Hosted Systems Specialist

01380 710278 / awitwicki@xxxxxxxxxxxxx

Oakford Internet Services Office: 01380 888088 
10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT
www.oakfordis.com sales@xxxxxxxxxxxxx