OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: certificate issue second mgmt-server


Hi All,


So, i was working on my primary issue with the certificate on my second node...i've stopped and restart the first node and now the cloudstack-management is in status failed :'(


I got an "invalid specified classpath" and "Cannot parse command line arguments".

I modified the file cloudstack-management.service" to manually set up these varirable $JAR:$CLASSPATH.


After a deamon-reload and a restart of the service, it's  run around 30 sec and fail again, here the log from the status service:


mai 04 20:43:24 ASPRCSMGMT01 systemd[1]: Starting CloudStack Management Server...
mai 04 20:43:24 ASPRCSMGMT01 systemd[1]: Started CloudStack Management Server.
mai 04 20:44:00 ASPRCSMGMT01 sudo[5033]:    cloud : TTY=unknown ; PWD=/var/log/cloudstack/management ; USER=root ; COMMAND=/bin/mkdir -p /systemvm_mnt
mai 04 20:44:00 ASPRCSMGMT01 sudo[5036]:    cloud : TTY=unknown ; PWD=/var/log/cloudstack/management ; USER=root ; COMMAND=/bin/mount -o loop /usr/share/cloudstack-common/vms/systemvm.iso /systemvm_mnt
mai 04 20:44:00 ASPRCSMGMT01 sudo[5041]:    cloud : TTY=unknown ; PWD=/var/log/cloudstack/management ; USER=root ; COMMAND=/bin/umount /systemvm_mnt
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: cloudstack-management.service: main process exited, code=exited, status=1/FAILURE
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: cloudstack-management.service: control process exited, code=exited status=255
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: Unit cloudstack-management.service entered failed state.
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: cloudstack-management.service failed.

>From /var/log/messages :

May  4 20:43:28 ASPRCSMGMT01 systemd: Configuration file /usr/lib/systemd/system/cloudstack-management.service is marked executable. Please remove executable permission bits. Proceeding anyway.
May  4 20:44:15 ASPRCSMGMT01 systemd: cloudstack-management.service: main process exited, code=exited, status=1/FAILURE
May  4 20:44:15 ASPRCSMGMT01 systemd: cloudstack-management.service: control process exited, code=exited status=255
May  4 20:44:15 ASPRCSMGMT01 systemd: Unit cloudstack-management.service entered failed state.



This management was working fine for one month and i restarted it without any problem before...


If one of you have any idea, it would be appreciated,

Thanks upfront for any help,

Best regards,

Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d’Asnière – 92600 Asnière sur Seine
T/ +33 (0)6 28 98 53 40


________________________________
De : Nicolas Bouige <n.bouige@xxxxxxxx>
Envoyé : mercredi 2 mai 2018 10:54:36
À : users@xxxxxxxxxxxxxxxxxxxxx
Objet : RE: certificate issue second mgmt-server

Rohit,

Thanks for the details, i'll keep you update if its work.

Best regards,
N.B

-----Message d'origine-----
De : Rohit Yadav [mailto:rohit.yadav@xxxxxxxxxxxxx]
Envoyé : mercredi 2 mai 2018 10:39
À : users@xxxxxxxxxxxxxxxxxxxxx
Objet : Re: certificate issue second mgmt-server

Nicolas,

Yes, if you've existing systemvms and KVM hosts changing the ca private/public key could cause system-wide cert issue. You can retry shutting down both management server(s), start the primary mgmt server to come up first and then start/deploy other mgmt servers one by one.


- Rohit

<https://cloudstack.apache.org>



________________________________
From: Nicolas Bouige <n.bouige@xxxxxxxx>
Sent: Wednesday, May 2, 2018 1:57:07 PM
To: users@xxxxxxxxxxxxxxxxxxxxx
Subject: RE: certificate issue second mgmt-server

Hi Rohit,

Thanks for your answer, i can't remember if i added the second node before the end of the initialization maybe i was too impatient :/ I'll give a try this week with your workaround.
Your workaround will affect also KVM server and System-VM ?

So, i guess, it's not enought to delete the second node and redeploy it ?

Best regards,
N.B

rohit.yadav@xxxxxxxxxxxxx
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




-----Message d'origine-----
De : Rohit Yadav [mailto:rohit.yadav@xxxxxxxxxxxxx]
Envoyé : mardi 1 mai 2018 12:56
À : users@xxxxxxxxxxxxxxxxxxxxx
Objet : Re: certificate issue second mgmt-server

Hi Nicolas,


Did you deploy multiple managements at the same time? When you deploy multiple management server(s), wait for the first management server to initialize database where it sets up some default offerings, global settings and the root CA keypair and certificate. Only when you see the first management server's UI in browser, proceed with deployment of other management server(s).


For your environment, you can test this workaround and let me know if that works for you:


  1.  Shutdown all the management server(s).
  2.  Delete ca keypair and cert:
 delete from configuration where name like "ca.plugin.root.private.key";  delete from configuration where name like "ca.plugin.root.public.key";  delete from configuration where name="ca.plugin.root.ca.certificate";
  3.  Start one management server and wait for it to complete internal setup, until you see the UI.
  4.  Start all the other management server(s).



- Rohit

<https://cloudstack.apache.org>



________________________________
From: Nicolas Bouige <n.bouige@xxxxxxxx>
Sent: Monday, April 30, 2018 2:59:29 PM
To: users@xxxxxxxxxxxxxxxxxxxxx
Subject: certificate issue second mgmt-server

Hello All,


I have an issue with one of my Cloudstack mgmt-server (4.11)

The second node has been deployed with the command  "cloudstack-setup-databases cloud:dbpassword@dbhost"


i didnt have any problem during few days and now sometimes i got an error on web GUI when i perfom some basic task, the error is "Resource [Host:1] is unreachable: Host 1: Unable to reach the peer that the agent is connected"


After a quick investigation, i had to stop cloudstack-management service from second mgmt-server and i noticed a lot of messages related with ca-certificate used by cloudstack :


2018-04-27 11:18:24,076 ERROR [c.c.u.n.Link] (StatsCollector-1:ctx-82335701) (logid:95fda6d7) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/172.16.22.61:60128, remote address=/172.16.22.60:8250. The client may have invalid ca-certificates.
2018-04-27 11:18:24,076 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:ctx-82335701) (logid:95fda6d7) Unable to connect to peer management server: 130719784044197, ip: 172.16.22.60 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '130719784044197' on 172.16.22.60:8250
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '130719784044197' on 172.16.22.60:8250
        at com.cloud.agent.manager.ClusteredAgentManagerImpl.connectToPeer(ClusteredAgentManagerImpl.java:529)
        at com.cloud.agent.manager.ClusteredAgentAttache.send(ClusteredAgentAttache.java:177)
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:456)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
        at com.cloud.agent.manager.AgentManagerImpl.easySend(AgentManagerImpl.java:954)
        at com.cloud.resource.ResourceManagerImpl.getHostStatistics(ResourceManagerImpl.java:2645)
        at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy178.getHostStatistics(Unknown Source)
        at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:438)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-04-27 11:18:24,077 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-82335701) (logid:95fda6d7) Seq 9-9075597674081682614: Unable to forward null
2018-04-27 11:18:24,177 ERROR [c.c.u.n.Link] (StatsCollector-1:ctx-82335701) (logid:95fda6d7) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/172.16.22.61:60130, remote address=/172.16.22.60:8250. The client may have invalid ca-certificates.
2018-04-27 11:18:24,177 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:ctx-82335701) (logid:95fda6d7) Unable to connect to peer management server: 130719784044197, ip: 172.16.22.60 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '130719784044197' on 172.16.22.60:8250

Im not familiar with the using of self-signed certificate in cloudstack, do you know where i can find out more information to investigate deeper ? or if you have any idea ?
I tried to check keystore on both mgmt-server but i need a password i havnt...


Thanks upfront,
Have a nice day,

Best regards,

Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d'Asnière - 92600 Asnière sur Seine

T/ +33 (0)6 28 98 53 40


rohit.yadav@xxxxxxxxxxxxx
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue