osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cloudtack manager UI returns Error 404


Hi Natalia,

Thanks for the update. We missed that context for root causing the issue.

-Suresh

On Tue, May 22, 2018 at 4:38 PM, Natalia Costas Lago <natalia@xxxxxxxx>
wrote:

>
> Hi,
>
> We finally were able to recover the manager :/
>
> The problem was due to a database modification made by hand a couple of
> months ago in order to eliminate one of the zombie virtual machines
> (machines that cannot be expunged because they are dependent on a network
> that was deleted before).
>
> The state of one of the machines was modified and Cloudstack did not like
> that change (setting a VM state as Destroyed), this eliminated the VM for
> the web interface but we did not detect the problem as we did not restart
> the management server. All backups since then were wrong and for some
> reason the management server crashed last week.
>
> Now it is fully operational :)
>
> Thank you,
> Kind regards,
>
>
> El 21/05/2018 a las 12:19, Natalia Costas Lago escribió:
>
>>
>> This is what I can see in localhost log:
>>
>> cat /var/log/cloudstack/management/localhost.2018-05-21.log
>>
>> [...]
>>
>> SEVERE: Exception sending context initialized event to listener instance
>> of class org.apache.cloudstack.spring.module.web.CloudStackContextLoa
>> derListener
>> java.lang.NullPointerException
>>         at com.cloud.vm.UserVmManagerImpl.loadVmDetailsInMapForExternal
>> DhcpIp(UserVmManagerImpl.java:2019)
>>         at com.cloud.vm.UserVmManagerImpl.start(UserVmManagerImpl.java:
>> 1999)
>>         at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycle$1.with(CloudStackExtendedLifeCycle.java:75)
>>         at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycle.with(CloudStackExtendedLifeCycle.java:154)
>>         at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycle.startBeans(CloudStackExtendedLifeCycle.java:72)
>>         at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif
>> eCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet$1.with(DefaultModuleDefinitionSet.java:105)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.startContexts(DefaultModuleDefinitionSet.java:97)
>>         at org.apache.cloudstack.spring.module.model.impl.DefaultModule
>> DefinitionSet.load(DefaultModuleDefinitionSet.java:80)
>>         at org.apache.cloudstack.spring.module.factory.ModuleBasedConte
>> xtFactory.loadModules(ModuleBasedContextFactory.java:37)
>>         at org.apache.cloudstack.spring.module.factory.CloudStackSpring
>> Context.init(CloudStackSpringContext.java:71)
>>         at org.apache.cloudstack.spring.module.factory.CloudStackSpring
>> Context.<init>(CloudStackSpringContext.java:58)
>>         at org.apache.cloudstack.spring.module.factory.CloudStackSpring
>> Context.<init>(CloudStackSpringContext.java:62)
>>         at org.apache.cloudstack.spring.module.web.CloudStackContextLoa
>> derListener.contextInitialized(CloudStackContextLoaderListener.java:52)
>>         at org.apache.catalina.core.StandardContext.listenerStart(Stand
>> ardContext.java:5118)
>>         at org.apache.catalina.core.StandardContext.startInternal(Stand
>> ardContext.java:5634)
>>         at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.
>> java:145)
>>         at org.apache.catalina.core.ContainerBase.addChildInternal(Cont
>> ainerBase.java:899)
>>         at org.apache.catalina.core.ContainerBase.addChild(ContainerBas
>> e.java:875)
>>         at org.apache.catalina.core.StandardHost.addChild(StandardHost.
>> java:652)
>>         at org.apache.catalina.startup.HostConfig.deployDirectory(HostC
>> onfig.java:1260)
>>         at org.apache.catalina.startup.HostConfig$DeployDirectory.run(
>> HostConfig.java:2002)
>>         at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:473)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1152)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:622)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> El 21/05/2018 a las 9:21, Natalia Costas Lago escribió:
>>
>>>
>>> It has been working with 1.8 for long...
>>>
>>> I switched to java 1.7, but java errors persist.
>>>
>>> We recovered backups from a couple of days prior to the disaster, but we
>>> are still unlucky, it seems the problem is also in the backups copy.
>>>
>>> I installed a fresh cloudstack and copied a recent database and the
>>> manager still gives and Exception but to a later point in the execution:
>>>
>>>
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Configuring CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Done Configuring CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
>>> (logid:) Loaded module context [ipmitool] in 171 ms
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
>>> (logid:) Starting module [system]
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Starting CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Done Starting CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null)
>>> (logid:) Starting module [core]
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
>>> (logid:) Starting CloudStack Components
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:) Cleanup
>>> mounted mount points used in previous session
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
>>> Grabbing lock to check for database integrity.
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:)
>>> Performing database integrity check
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Starting
>>> Cluster manager, msid : 2199193518866
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:)
>>> Cluster service servlet container listening on port 9090
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Management
>>> server 2199193518866, runId 1526887063086 is being started
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Management
>>> server (host id : 2) is being started at 10.38.3.18:9090
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Cluster
>>> manager was started successfully
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:)
>>> Starting work
>>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
>>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:)
>>> Starting work
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No appenders
>>> could be found for logger (com.cloud.utils.db.ConnectionConcierge).
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please
>>> initialize the log4j system properly.
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See
>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread
>>> "Thread-10" java.lang.NoClassDefFoundError:
>>> org/apache/log4j/spi/ThrowableInformation
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.log4j.Category.forcedLog(Category.java:391)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.log4j.Category.error(Category.java:322)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> com.cloud.cluster.ClusterServiceServletContainer$
>>> ListenerThread.run(ClusterServiceServletContainer.java:168)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by:
>>> java.lang.ClassNotFoundException: org.apache.log4j.spi.Throwable
>>> Information
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(W
>>> ebappClassLoaderBase.java:1892)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: at
>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(W
>>> ebappClassLoaderBase.java:1735)
>>> May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more
>>>
>>>
>>>
>>> El 17/05/2018 a las 7:39, ilya musayev escribió:
>>>
>>>> You must be running java 1.7 - I assume Suresh was alluding to this.
>>>> We’ve seen this error with 1.8...
>>>>
>>>> On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
>>>> sureshkumar.anaparti@xxxxxxxxx> wrote:
>>>>
>>>> Hi Natalia,
>>>>>
>>>>> Any changes in the java version or system configuration?
>>>>>
>>>>> -Suresh
>>>>>
>>>>> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo <
>>>>> Dag.Sonstebo@xxxxxxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>> Hi Natalia,
>>>>>>
>>>>>> My guess is you have some sort of corruption on your original host.
>>>>>> Since
>>>>>> the management service itself is stateless it’s generally much quicker
>>>>>>
>>>>> just
>>>>>
>>>>>> building a new one than trying to recover the old one.
>>>>>>
>>>>>> So my suggestions would be:
>>>>>> - Recover your DB to a dedicated DB host if you haven’t done so
>>>>>> already.
>>>>>> - Build a new management server and just point this to the original
>>>>>> DB as
>>>>>> described in http://docs.cloudstack.apache.org/projects/cloudstack-
>>>>>> installation/en/4.9/management-server/index.html#
>>>>>> additional-management-servers
>>>>>> NOTE: do not use the “–deploy-as” option as this will write a new
>>>>>> blank
>>>>>> database. If you used encryption keys during the original setup you
>>>>>> also
>>>>>> need to specify these.
>>>>>>
>>>>>> Once you have these two speaking you should in theory be back up and
>>>>>> running again.
>>>>>>
>>>>>> Regards,
>>>>>> Dag Sonstebo
>>>>>> Cloud Architect
>>>>>> ShapeBlue
>>>>>>
>>>>>>
>>>>>> Dag.Sonstebo@xxxxxxxxxxxxx
>>>>>> www.shapeblue.com
>>>>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>>>>> @shapeblue
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 16/05/2018, 16:44, "Natalia Costas Lago" <natalia@xxxxxxxx> wrote:
>>>>>>
>>>>>>
>>>>>>      Dear all,
>>>>>>
>>>>>>      For some reason our cloudstack manager died and now we are not
>>>>>> able
>>>>>>
>>>>> to
>>>>>
>>>>>>      make it work. Our cloudstack version is 4.9.0.
>>>>>>
>>>>>>      We tried to do a clean install of the manager and recover the
>>>>>>
>>>>> database
>>>>>
>>>>>>      from the last backup, but we were not able to recover the
>>>>>> service.
>>>>>>
>>>>>>      At present we can see in the logs that it almost boot, but at
>>>>>> the end
>>>>>> we
>>>>>>      get this error:
>>>>>>
>>>>>>      May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>>>      [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null)
>>>>>> (logid:)
>>>>>>      Network Manager will run the NetworkGarbageCollector every '600'
>>>>>> seconds.
>>>>>>      May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>>> [c.c.a.ApiServer]
>>>>>>      (Thread-12:null) (logid:) ApiServer listening on port 8096
>>>>>>      May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
>>>>>>      [o.a.c.s.SecondaryStorageManagerImpl]
>>>>>> (localhost-startStop-1:null)
>>>>>>      (logid:) Start secondary storage vm manager
>>>>>>      May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
>>>>>> appenders
>>>>>>      could be found for logger (com.cloud.utils.db.Connection
>>>>>> Concierge).
>>>>>>      May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN
>>>>>> Please
>>>>>>      initialize the log4j system properly.
>>>>>>      May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
>>>>>>      http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
>>>>>> info.
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in
>>>>>> thread
>>>>>>      "CapacityChecker" java.lang.NoClassDefFoundError:
>>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>>>> ManagedContextTimerTask.java:27)
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>>      java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>>      java.util.TimerThread.run(Timer.java:505)
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
>>>>>>      java.lang.ClassNotFoundException:
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>>>> WebappClassLoaderBase.java:1892)
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
>>>>>> WebappClassLoaderBase.java:1735)
>>>>>>      May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
>>>>>>      May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in
>>>>>> thread
>>>>>>      "Timer-1" java.lang.NoClassDefFoundError:
>>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
>>>>>>      May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
>>>>>> ManagedContextTimerTask.java:27)
>>>>>>      May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>>>      java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>>      May 16 17:34:24 cldmanager.srv.cesga.es server: at
>>>>>>      java.util.TimerThread.run(Timer.java:505)
>>>>>>
>>>>>>      Any help is appreciated..
>>>>>>
>>>>>>      Kind regards,
>>>>>>
>>>>>>      NOTA: Para cualquier incidencia/consulta/petición que requiera
>>>>>>      seguimiento por favor enviar a helpdesk_comunicaciones@xxxxxxxx
>>>>>>
>>>>>>      --
>>>>>>      ====================================================
>>>>>>      Natalia Costas Lago
>>>>>>      Senior Communications Technician
>>>>>>      Galicia Supercomputing Centre (CESGA)
>>>>>>      (CESGA on Twitter  |  CESGA on Facebook)
>>>>>>
>>>>>>      Avenida de Vigo, s/n (Campus Vida)
>>>>>>      15705 Santiago de Compostela - SPAIN
>>>>>>
>>>>>>      E-mail: natalia@xxxxxxxx
>>>>>>      Cell: +34 981 56 98 10 (ext. 237)
>>>>>>      Fax: +34 981 59 46 16
>>>>>>      Web: https://www.cesga.es/
>>>>>>      ====================================================
>>>>>>      [IMPORTANTE] La información contenida en este mensaje y
>>>>>>      sus posibles documentos adjuntos es privada y confidencial
>>>>>>      y está dirigida únicamente a su destinatario/a. Si usted no
>>>>>>      es el/la destinatario/a original de este mensaje, por favor
>>>>>>      elimínelo. La distribución o copia de este mensaje no está
>>>>>>      autorizada.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>>> seguimiento por favor enviar a helpdesk_comunicaciones@xxxxxxxx
>>>
>>>
>> NOTA: Para cualquier incidencia/consulta/petición que requiera
>> seguimiento por favor enviar a helpdesk_comunicaciones@xxxxxxxx
>>
>>
> NOTA: Para cualquier incidencia/consulta/petición que requiera
> seguimiento por favor enviar a helpdesk_comunicaciones@xxxxxxxx
>
> --
> ====================================================
> Natalia Costas Lago
> Senior Communications Technician
> Galicia Supercomputing Centre (CESGA)
> (CESGA on Twitter  |  CESGA on Facebook)
>
> Avenida de Vigo, s/n (Campus Vida)
> 15705 Santiago de Compostela - SPAIN
>
> E-mail: natalia@xxxxxxxx
> Cell: +34 981 56 98 10 (ext. 237)
> Fax: +34 981 59 46 16
> Web: https://www.cesga.es/
> ====================================================
> [IMPORTANTE] La información contenida en este mensaje y
> sus posibles documentos adjuntos es privada y confidencial
> y está dirigida únicamente a su destinatario/a. Si usted no
> es el/la destinatario/a original de este mensaje, por favor
> elimínelo. La distribución o copia de este mensaje no está
> autorizada.
>
>