OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SSVM's not starting, timeout for libvirt python script in agent.log


The routers managed to start after a while. Looks like most things are OK
now, however I hope some of you could answer some questions for me.

- I have enabled remote VPN in a VPC but I cannot seem to add users
anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
users are now accessed by changing views at the networks tab". However, I
cannot find out where such view would be? The documentation doesn't state
anything about this.
- When I create a new project, choose the view from that project and go to
"Network", I always get this message: " Account and projectId can't be
specified together". What is it exactly and how can I get rid of it?
- I don't seem to be able to do port-forwarding with the first configured
IP for the VPC. However when adding another IP, I can do it on that
specific IP. Isn't it possible to do forwarding on the first assigned IP?
- Is it normal that it takes minutes, sometimes hours to create the VPC? It
really takes a while for the system to get going once I have created a VPC.

Perhaps this is for another email-thread, please let me know if so.

Apart from that it seems like the cloudstack instance is finally working,
first VM booted up!

Thank you!
Chris

On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n.bouige@xxxxxxxx> wrote:

> Hello All,
>
> Maybe is not related to your case but in case of....as we don't use the
> same OS
>
> I was facing the exact same error during a new creation of virtual router.
> The only change i made was to update my qemu version on our KVM (centOS7 +
> CS 4.11) from 1.5 to 2.3
> Before that VR creation was working fine.
>
> Apperently some users got the same issue but with an old version of CS
> (4.9)
> http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
> patchviasocket-seems-to-be-broken-with-qemu-2-3
>
>
>
> -----Message d'origine-----
> De : Christoffer Pedersen [mailto:vrod@xxxxxxx]
> Envoyé : mardi 12 juin 2018 19:17
> À : users@xxxxxxxxxxxxxxxxxxxxx
> Objet : Re: SSVM's not starting, timeout for libvirt python script in
> agent.log
>
> Hi James,
>
> I looked at the value and it's already on 50GB. I looked a little more and
> it seems like after restarting the cloudstack-management service I could no
> longer reach the console or storage system vm.... looking at the host, now
> the cloud0 interface is gone. I am using OVS, but according to a Shapeblue
> article, no extra config is needed for the cloud0 interface.
>
>
>
> On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
> mcclunej@xxxxxxxxxxxxxxxxxxx> wrote:
>
> > Hi Christoffer,
> >
> > Did you try setting the max.template.iso.size to a higher value (e.g.
> > 50GB)?
> >
> > I think the default is set pretty low.
> >
> > Best,
> > James
> >
> > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vrod@xxxxxxx>
> > wrote:
> >
> > > Just want to update that I still get the ISO error:
> > >
> > > 2018-06-12 17:09:42,194 INFO  [c.c.t.HypervisorTemplateAdapter]
> > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image
> > > store doesn't have enough capacity. Skip downloading template to
> > > this image
> > store
> > > 1
> > >
> > > Thanks!
> > >
> > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vrod@xxxxxxx>
> > > wrote:
> > >
> > >> Hi Dag,
> > >>
> > >> Yes, I altered the IP addresses as I do not fancy throwing them out
> > >> on the public net. If you think they have value to the
> > >> troubleshooting, I
> > can
> > >> send you the original logdata directly. I configured the networking
> > >> with OVS and as follows:
> > >>
> > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts
> > >> the system VMs I guess. In the Zone setup, I labelled the
> > >> Management
> > network as
> > >> MGMT0, I guess that's OK? It's a Internal network where other
> > >> servers
> > are
> > >> also connected. I can ping the VMs here, also over a S2S connection
> > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I
> > >> use real public IP-addresses from a scope and I am able to ping the
> > addresses
> > >> of both the CPVM and SSVM from my home.
> > >>
> > >> I have to update the situation though...
> > >>
> > >> Somehow at 13:02, I had the last error from the agent. Now, I do
> > >> not
> > have
> > >> any other errors and both VMs are now showing as "running" as well
> > >> as
> > agent
> > >> being "up" in the UI...  had these errors since yesterday and did
> > >> not change anything after 9:00 this morning. Maybe I'm impatient
> > >> but 4 hours seems a bit long for them to get to work. :)
> > >>
> > >> Console VM works though at least:
> > >>
> > >>
> > >>
> > >> My impression is that cloudstack needs a while to get hold of
> things...
> > >> or am I just experiencing unusual things? I have a question though:
> > >> When adding an ISO this morning, I had an error about that there
> > >> was no space left (though the storage is 20TB). Was this because
> > >> the SSVM was not running at the time?
> > >>
> > >> Thank you!
> > >> Chris
> > >>
> > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> > Dag.Sonstebo@xxxxxxxxxxxxx
> > >> > wrote:
> > >>
> > >>> Chris,
> > >>>
> > >>> Going off in a slightly different direction to previous answers. I
> > >>> suspect your problem is with networking - how have you configured
> this?
> > >>> When you say you can ping the SSVM on the private interface which
> > >>> IP address do you use and where do you successfully ping from?
> > >>>
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> > patchviasocket.py
> > >>> -n
> > >>> v-1-VM -p
> > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > >>> %dns2=8.8.4.4
> > >>>
> > >>> It could be you have edited the above IP addresses to mask your
> > >>> real addresses – if so ignore this.
> > >>>
> > >>> If not then the above points to:
> > >>> - Management host is on 1.1.1.1
> > >>> - Eth2 which for a console proxy is public traffic is on
> > >>> 9.9.9.9/24
> > >>> - Eth0 which is the link local management interface is on
> > >>> 169.254.3.159/16 (system generated)
> > >>> - Eth1 is the main management interface on 6.6.6.6/24
> > >>> - You have a gateway address of 1.2.3.4
> > >>>
> > >>> So in this case – the CPVM can not check in to the management host
> > >>> on
> > >>> 1.1.1.1 -  It’s got no interface on that subnet and it also has a
> > gateway
> > >>> it’s not able to reach.
> > >>>
> > >>> Regards,
> > >>> Dag Sonstebo
> > >>> Cloud Architect
> > >>> ShapeBlue
> > >>>
> > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n.bouige@xxxxxxxx> wrote:
> > >>>
> > >>>     Hi Ivan,
> > >>>
> > >>>
> > >>>     Are you talking about this global parameters :
> > >>>
> > >>>     router.aggregation.command.each.timeout
> > >>>
> > >>>
> > >>>
> > >>>     Best regards,
> > >>>
> > >>>     Nicolas Bouige
> > >>>     DIMSI
> > >>>     cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > >>>     4, avenue Laurent Cely
> > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > gmail&source=g>
> > >>>     Tour d’Asnière – 92600 Asnière sur Seine
> > >>>     T/ +33 (0)6 28 98 53 40
> > >>>
> > >>>
> > >>>     ________________________________
> > >>>     De : Ivan Kudryavtsev <kudryavtsev_ia@xxxxxxxxx>
> > >>>     Envoyé : mardi 12 juin 2018 13:59:39
> > >>>     À : users
> > >>>     Objet : Re: SSVM's not starting, timeout for libvirt python
> > >>> script in agent.log
> > >>>
> > >>>     Increasing command timeouts in global parameters can work
> > >>> here. At least I
> > >>>     met similar behaviour with VR.
> > >>>
> > >>>     вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vrod@xxxxxxx>:
> > >>>
> > >>>     > Hi Nicolas,
> > >>>     >
> > >>>     > I did a apt show qemu and it gave me this version:
> > >>>     >
> > >>>     > Version: 1:2.5+dfsg-5ubuntu10.29
> > >>>     >
> > >>>     > So I guess tha would be version 2.5?
> > >>>     >
> > >>>
> > >>> Dag.Sonstebo@xxxxxxxxxxxxx
> > >>> www.shapeblue.com
> > >>> 53 Chandos Place, Covent Garden, London
> > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> > Garden,+London&entry=gmail&source=g>
> > >>> WC2N 4HSUK
> > >>> @shapeblue
> > >>>
> > >>>
> > >>>
> > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
> > >>> > <n.bouige@xxxxxxxx>
> > >>> wrote:
> > >>>     >
> > >>>     > > Hello Christoffer,
> > >>>     > >
> > >>>     > >
> > >>>     > > Could you tell us wich qemu version are you using ?
> > >>>     > >
> > >>>     > > Nicolas Bouige
> > >>>     > > DIMSI
> > >>>     > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > >>>     > > 4, avenue Laurent Cely
> > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > gmail&source=g>
> > >>>     > > Tour d’Asnière – 92600 Asnière sur Seine
> > >>>     > > T/ +33 (0)6 28 98 53 40
> > >>>     > >
> > >>>     > >
> > >>>     > > ________________________________
> > >>>     > > De : Christoffer Pedersen <vrod@xxxxxxx>
> > >>>     > > Envoyé : mardi 12 juin 2018 12:30:48
> > >>>     > > À : users@xxxxxxxxxxxxxxxxxxxxx
> > >>>     > > Objet : SSVM's not starting, timeout for libvirt python
> > >>> script
> > in
> > >>>     > agent.log
> > >>>     > >
> > >>>     > > Hi all,
> > >>>     > >
> > >>>     > > I have an issue regarding the system VMs. After deploying
> > >>> an advanced
> > >>>     > zone,
> > >>>     > > the system VMs are trying to be created but gets stuck in
> > >>> a "Starting"
> > >>>     > > state, however the Agent state is "Up". I have these logs
> > >>> in
> > the
> > >>>     > agent.log
> > >>>     > > (sorry for the formatting)
> > >>>     > >
> > >>>     > > 2018-06-12 12:22:06,354 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>>     > > (Script-8:null) (logid:) Interrupting script.
> > >>>     > > 2018-06-12 12:22:06,355 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > >>>     > >
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > >>> viasocket.py
> > >>>     > > -n
> > >>>     > > v-1-VM -p
> > >>>     > > %template=domP%type=consoleproxy%host=1.1.1.1%
> > >>>     > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > >>>     > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > >>>     > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > >>>     > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >>>     > >
> > >>>     > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > >>> %dns2=8.8.4.4
> > >>>     > > .  Output is:
> > >>>     > > 2018-06-12 12:22:06,355 ERROR
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> > >>> failed:timeout
> > >>>     > > 2018-06-12 12:22:08,914 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>>     > > (Script-4:null) (logid:) Interrupting script.
> > >>>     > > 2018-06-12 12:22:08,915 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > >>>     > >
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > >>> viasocket.py
> > >>>     > > -n
> > >>>     > > s-2-VM -p
> > >>>     > > %template=domP%type=secstorage%host=1.1.1.1%port=
> > >>>     > >
> > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> > >>> =org.apache.
> > >>>     > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > >>>     > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > >>>     > >
> > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> > >>> .1%public.
> > >>>     > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > >>>     > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > >>>     > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > >>>     > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=
> null
> > >>>     > > .  Output is:
> > >>>     > > 2018-06-12 12:22:08,915 ERROR
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> > >>> failed:timeout
> > >>>     > >
> > >>>     > > I have seen this error around but did not really find a
> > solution
> > >>> to it. I
> > >>>     > > am not exactly sure whats "timing" out? I can ping both
> > >>> SSVM's on their
> > >>>     > > private and public interface.
> > >>>     > >
> > >>>     > > I hope someone can help me out here. :)
> > >>>     > >
> > >>>     > > --
> > >>>     > > Thanks,
> > >>>     > > Chris pedersen
> > >>>     > >
> > >>>     >
> > >>>     >
> > >>>     >
> > >>>     > --
> > >>>     > Thanks,
> > >>>     > Chris pedersen
> > >>>     >
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Chris pedersen
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Chris pedersen
> > >
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>



-- 
Thanks,
Chris pedersen