OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SSVM's not starting, timeout for libvirt python script in agent.log


Hi Dag,


- Network menu > select view pulldown box > VPN users
=>>> You mean under "Select view:" (which defaults to Guest Network),
right? There I am only able to see "Guest Network", "VPC" and "VPN Customer
Gateway"

- I have just tested creating a new project as a normal user, change this
on the pulldown project menu and clicked on the “network” menu item, no
error in RC2 – is this where you found it?
=>>> I just tried again with a user i created, it's an admin though. I
still get the message.

- This is by design, for VPCs you have to add additional IPs to configure
services. The only service available on the primary VPC IP is VPN.
=>>> Ok thanks, good to know!

- Minutes yes, hours no. Going by your previous posts I would guess you
still have some tuning up to do and possibly run on low spec hardware
though?
=>>> The 2 hosts I have run with 2x E5-2680's and 64GB of ram. The
management server itself is a VMware hosted VM with 16gb of memory and
4vCPU's (E5-2650). It could be the tuning I guess, I am currently at a loss
here. The storage is a FreeNAS based NFS share but it's pretty fast.


On Wed, Jun 13, 2018 at 5:54 PM, Dag Sonstebo <Dag.Sonstebo@xxxxxxxxxxxxx>
wrote:

> Hi Chris,
>
> - I have enabled remote VPN in a VPC but I cannot seem to add users
> anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
> users are now accessed by changing views at the networks tab". However, I
> cannot find out where such view would be? The documentation doesn't state
> anything about this.
>
> =>>> Network menu > select view pulldown box > VPN users
>
> - When I create a new project, choose the view from that project and go to
> "Network", I always get this message: " Account and projectId can't be
> specified together". What is it exactly and how can I get rid of it?
>
> =>>> I have just tested creating a new project as a normal user, change
> this on the pulldown project menu and clicked on the “network” menu item,
> no error in RC2 – is this where you found it?
>
> - I don't seem to be able to do port-forwarding with the first configured
> IP for the VPC. However when adding another IP, I can do it on that
> specific IP. Isn't it possible to do forwarding on the first assigned IP?
>
> =>>> This is by design, for VPCs you have to add additional IPs to
> configure services. The only service available on the primary VPC IP is VPN.
>
> - Is it normal that it takes minutes, sometimes hours to create the VPC? It
> really takes a while for the system to get going once I have created a VPC.
>
> =>>> Minutes yes, hours no. Going by your previous posts I would guess you
> still have some tuning up to do and possibly run on low spec hardware
> though?
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 13/06/2018, 16:25, "Christoffer Pedersen" <vrod@xxxxxxx> wrote:
>
>     The routers managed to start after a while. Looks like most things are
> OK
>     now, however I hope some of you could answer some questions for me.
>
>     - I have enabled remote VPN in a VPC but I cannot seem to add users
>     anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
>     users are now accessed by changing views at the networks tab".
> However, I
>     cannot find out where such view would be? The documentation doesn't
> state
>     anything about this.
>     - When I create a new project, choose the view from that project and
> go to
>     "Network", I always get this message: " Account and projectId can't be
>     specified together". What is it exactly and how can I get rid of it?
>     - I don't seem to be able to do port-forwarding with the first
> configured
>     IP for the VPC. However when adding another IP, I can do it on that
>     specific IP. Isn't it possible to do forwarding on the first assigned
> IP?
>     - Is it normal that it takes minutes, sometimes hours to create the
> VPC? It
>     really takes a while for the system to get going once I have created a
> VPC.
>
>     Perhaps this is for another email-thread, please let me know if so.
>
>     Apart from that it seems like the cloudstack instance is finally
> working,
>     first VM booted up!
>
>     Thank you!
>     Chris
>
>
> Dag.Sonstebo@xxxxxxxxxxxxx
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
> On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n.bouige@xxxxxxxx> wrote:
>
>     > Hello All,
>     >
>     > Maybe is not related to your case but in case of....as we don't use
> the
>     > same OS
>     >
>     > I was facing the exact same error during a new creation of virtual
> router.
>     > The only change i made was to update my qemu version on our KVM
> (centOS7 +
>     > CS 4.11) from 1.5 to 2.3
>     > Before that VR creation was working fine.
>     >
>     > Apperently some users got the same issue but with an old version of
> CS
>     > (4.9)
>     > http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
>     > patchviasocket-seems-to-be-broken-with-qemu-2-3
>     >
>     >
>     >
>     > -----Message d'origine-----
>     > De : Christoffer Pedersen [mailto:vrod@xxxxxxx]
>     > Envoyé : mardi 12 juin 2018 19:17
>     > À : users@xxxxxxxxxxxxxxxxxxxxx
>     > Objet : Re: SSVM's not starting, timeout for libvirt python script in
>     > agent.log
>     >
>     > Hi James,
>     >
>     > I looked at the value and it's already on 50GB. I looked a little
> more and
>     > it seems like after restarting the cloudstack-management service I
> could no
>     > longer reach the console or storage system vm.... looking at the
> host, now
>     > the cloud0 interface is gone. I am using OVS, but according to a
> Shapeblue
>     > article, no extra config is needed for the cloud0 interface.
>     >
>     >
>     >
>     > On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
>     > mcclunej@xxxxxxxxxxxxxxxxxxx> wrote:
>     >
>     > > Hi Christoffer,
>     > >
>     > > Did you try setting the max.template.iso.size to a higher value
> (e.g.
>     > > 50GB)?
>     > >
>     > > I think the default is set pretty low.
>     > >
>     > > Best,
>     > > James
>     > >
>     > > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <
> vrod@xxxxxxx>
>     > > wrote:
>     > >
>     > > > Just want to update that I still get the ISO error:
>     > > >
>     > > > 2018-06-12 17:09:42,194 INFO  [c.c.t.HypervisorTemplateAdapter]
>     > > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7)
> Image
>     > > > store doesn't have enough capacity. Skip downloading template to
>     > > > this image
>     > > store
>     > > > 1
>     > > >
>     > > > Thanks!
>     > > >
>     > > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <
> vrod@xxxxxxx>
>     > > > wrote:
>     > > >
>     > > >> Hi Dag,
>     > > >>
>     > > >> Yes, I altered the IP addresses as I do not fancy throwing them
> out
>     > > >> on the public net. If you think they have value to the
>     > > >> troubleshooting, I
>     > > can
>     > > >> send you the original logdata directly. I configured the
> networking
>     > > >> with OVS and as follows:
>     > > >>
>     > > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also
> hosts
>     > > >> the system VMs I guess. In the Zone setup, I labelled the
>     > > >> Management
>     > > network as
>     > > >> MGMT0, I guess that's OK? It's a Internal network where other
>     > > >> servers
>     > > are
>     > > >> also connected. I can ping the VMs here, also over a S2S
> connection
>     > > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999).
> I
>     > > >> use real public IP-addresses from a scope and I am able to ping
> the
>     > > addresses
>     > > >> of both the CPVM and SSVM from my home.
>     > > >>
>     > > >> I have to update the situation though...
>     > > >>
>     > > >> Somehow at 13:02, I had the last error from the agent. Now, I do
>     > > >> not
>     > > have
>     > > >> any other errors and both VMs are now showing as "running" as
> well
>     > > >> as
>     > > agent
>     > > >> being "up" in the UI...  had these errors since yesterday and
> did
>     > > >> not change anything after 9:00 this morning. Maybe I'm impatient
>     > > >> but 4 hours seems a bit long for them to get to work. :)
>     > > >>
>     > > >> Console VM works though at least:
>     > > >>
>     > > >>
>     > > >>
>     > > >> My impression is that cloudstack needs a while to get hold of
>     > things...
>     > > >> or am I just experiencing unusual things? I have a question
> though:
>     > > >> When adding an ISO this morning, I had an error about that there
>     > > >> was no space left (though the storage is 20TB). Was this because
>     > > >> the SSVM was not running at the time?
>     > > >>
>     > > >> Thank you!
>     > > >> Chris
>     > > >>
>     > > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
>     > > Dag.Sonstebo@xxxxxxxxxxxxx
>     > > >> > wrote:
>     > > >>
>     > > >>> Chris,
>     > > >>>
>     > > >>> Going off in a slightly different direction to previous
> answers. I
>     > > >>> suspect your problem is with networking - how have you
> configured
>     > this?
>     > > >>> When you say you can ping the SSVM on the private interface
> which
>     > > >>> IP address do you use and where do you successfully ping from?
>     > > >>>
>     > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
>     > > patchviasocket.py
>     > > >>> -n
>     > > >>> v-1-VM -p
>     > > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
>     > > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
>     > > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
>     > > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
>     > > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
>     > > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
>     > > >>> %dns2=8.8.4.4
>     > > >>>
>     > > >>> It could be you have edited the above IP addresses to mask your
>     > > >>> real addresses – if so ignore this.
>     > > >>>
>     > > >>> If not then the above points to:
>     > > >>> - Management host is on 1.1.1.1
>     > > >>> - Eth2 which for a console proxy is public traffic is on
>     > > >>> 9.9.9.9/24
>     > > >>> - Eth0 which is the link local management interface is on
>     > > >>> 169.254.3.159/16 (system generated)
>     > > >>> - Eth1 is the main management interface on 6.6.6.6/24
>     > > >>> - You have a gateway address of 1.2.3.4
>     > > >>>
>     > > >>> So in this case – the CPVM can not check in to the management
> host
>     > > >>> on
>     > > >>> 1.1.1.1 -  It’s got no interface on that subnet and it also
> has a
>     > > gateway
>     > > >>> it’s not able to reach.
>     > > >>>
>     > > >>> Regards,
>     > > >>> Dag Sonstebo
>     > > >>> Cloud Architect
>     > > >>> ShapeBlue
>     > > >>>
>     > > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n.bouige@xxxxxxxx>
> wrote:
>     > > >>>
>     > > >>>     Hi Ivan,
>     > > >>>
>     > > >>>
>     > > >>>     Are you talking about this global parameters :
>     > > >>>
>     > > >>>     router.aggregation.command.each.timeout
>     > > >>>
>     > > >>>
>     > > >>>
>     > > >>>     Best regards,
>     > > >>>
>     > > >>>     Nicolas Bouige
>     > > >>>     DIMSI
>     > > >>>     cloud.dimsi.fr<http://www.cloud.dimsi.fr>
>     > > >>>     4, avenue Laurent Cely
>     > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
>     > > gmail&source=g>
>     > > >>>     Tour d’Asnière – 92600 Asnière sur Seine
>     > > >>>     T/ +33 (0)6 28 98 53 40
>     > > >>>
>     > > >>>
>     > > >>>     ________________________________
>     > > >>>     De : Ivan Kudryavtsev <kudryavtsev_ia@xxxxxxxxx>
>     > > >>>     Envoyé : mardi 12 juin 2018 13:59:39
>     > > >>>     À : users
>     > > >>>     Objet : Re: SSVM's not starting, timeout for libvirt python
>     > > >>> script in agent.log
>     > > >>>
>     > > >>>     Increasing command timeouts in global parameters can work
>     > > >>> here. At least I
>     > > >>>     met similar behaviour with VR.
>     > > >>>
>     > > >>>     вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <
> vrod@xxxxxxx>:
>     > > >>>
>     > > >>>     > Hi Nicolas,
>     > > >>>     >
>     > > >>>     > I did a apt show qemu and it gave me this version:
>     > > >>>     >
>     > > >>>     > Version: 1:2.5+dfsg-5ubuntu10.29
>     > > >>>     >
>     > > >>>     > So I guess tha would be version 2.5?
>     > > >>>     >
>     > > >>>
>     > > >>> Dag.Sonstebo@xxxxxxxxxxxxx
>     > > >>> www.shapeblue.com
>     > > >>> 53 Chandos Place, Covent Garden, London
>     > > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
>     > > Garden,+London&entry=gmail&source=g>
>     > > >>> WC2N 4HSUK
>     > > >>> @shapeblue
>     > > >>>
>     > > >>>
>     > > >>>
>     > > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
>     > > >>> > <n.bouige@xxxxxxxx>
>     > > >>> wrote:
>     > > >>>     >
>     > > >>>     > > Hello Christoffer,
>     > > >>>     > >
>     > > >>>     > >
>     > > >>>     > > Could you tell us wich qemu version are you using ?
>     > > >>>     > >
>     > > >>>     > > Nicolas Bouige
>     > > >>>     > > DIMSI
>     > > >>>     > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
>     > > >>>     > > 4, avenue Laurent Cely
>     > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
>     > > gmail&source=g>
>     > > >>>     > > Tour d’Asnière – 92600 Asnière sur Seine
>     > > >>>     > > T/ +33 (0)6 28 98 53 40
>     > > >>>     > >
>     > > >>>     > >
>     > > >>>     > > ________________________________
>     > > >>>     > > De : Christoffer Pedersen <vrod@xxxxxxx>
>     > > >>>     > > Envoyé : mardi 12 juin 2018 12:30:48
>     > > >>>     > > À : users@xxxxxxxxxxxxxxxxxxxxx
>     > > >>>     > > Objet : SSVM's not starting, timeout for libvirt python
>     > > >>> script
>     > > in
>     > > >>>     > agent.log
>     > > >>>     > >
>     > > >>>     > > Hi all,
>     > > >>>     > >
>     > > >>>     > > I have an issue regarding the system VMs. After
> deploying
>     > > >>> an advanced
>     > > >>>     > zone,
>     > > >>>     > > the system VMs are trying to be created but gets stuck
> in
>     > > >>> a "Starting"
>     > > >>>     > > state, however the Agent state is "Up". I have these
> logs
>     > > >>> in
>     > > the
>     > > >>>     > agent.log
>     > > >>>     > > (sorry for the formatting)
>     > > >>>     > >
>     > > >>>     > > 2018-06-12 12:22:06,354 WARN
>     > > >>> [kvm.resource.LibvirtComputing Resource]
>     > > >>>     > > (Script-8:null) (logid:) Interrupting script.
>     > > >>>     > > 2018-06-12 12:22:06,355 WARN
>     > > >>> [kvm.resource.LibvirtComputing Resource]
>     > > >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed
> out:
>     > > >>>     > >
>     > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
>     > > >>> viasocket.py
>     > > >>>     > > -n
>     > > >>>     > > v-1-VM -p
>     > > >>>     > > %template=domP%type=consoleproxy%host=1.1.1.1%
>     > > >>>     > > port=8250%name=v-1-VM%zone=1%
> pod=1%guid=Proxy.1%proxy_vm=1%
>     > > >>>     > > disable_rp_filter=true%eth2ip=
> 9.9.9.9%eth2mask=255.255.255.
>     > > >>>     > > 0%gateway=9.9.9.1%eth0ip=169.
> 254.3.159%eth0mask=255.255.0.
>     > > >>>     > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
>     > > >>>     > >
>     > > >>>     > 9.9.9.0/24%localgw=1.2.3.4%
> internaldns1=1.2.3.4%dns1=8.8.8.8
>     > > >>> %dns2=8.8.4.4
>     > > >>>     > > .  Output is:
>     > > >>>     > > 2018-06-12 12:22:06,355 ERROR
>     > > >>> [kvm.resource.LibvirtComputing Resource]
>     > > >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
>     > > >>> failed:timeout
>     > > >>>     > > 2018-06-12 12:22:08,914 WARN
>     > > >>> [kvm.resource.LibvirtComputing Resource]
>     > > >>>     > > (Script-4:null) (logid:) Interrupting script.
>     > > >>>     > > 2018-06-12 12:22:08,915 WARN
>     > > >>> [kvm.resource.LibvirtComputing Resource]
>     > > >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed
> out:
>     > > >>>     > >
>     > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
>     > > >>> viasocket.py
>     > > >>>     > > -n
>     > > >>>     > > s-2-VM -p
>     > > >>>     > > %template=domP%type=secstorage%host=1.1.1.1%port=
>     > > >>>     > >
>     > > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
>     > > >>> =org.apache.
>     > > >>>     > > cloudstack.storage.resource.
> NfsSecondaryStorageResource%
>     > > >>>     > > instance=SecStorage%sslcopy=
> false%role=templateProcessor%
>     > > >>>     > >
>     > > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
>     > > >>> .1%public.
>     > > >>>     > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
>     > > >>>     > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%
> mgmtcidr=
>     > > >>>     > > 9.9.9.0/24%localgw=1.2.3.4%
> private.network.device=eth1%
>     > > >>>     > > internaldns1=1.2.3.4%dns1=8.8.
> 8.8%dns2=8.8.4.4%nfsVersion=
>     > null
>     > > >>>     > > .  Output is:
>     > > >>>     > > 2018-06-12 12:22:08,915 ERROR
>     > > >>> [kvm.resource.LibvirtComputing Resource]
>     > > >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
>     > > >>> failed:timeout
>     > > >>>     > >
>     > > >>>     > > I have seen this error around but did not really find a
>     > > solution
>     > > >>> to it. I
>     > > >>>     > > am not exactly sure whats "timing" out? I can ping both
>     > > >>> SSVM's on their
>     > > >>>     > > private and public interface.
>     > > >>>     > >
>     > > >>>     > > I hope someone can help me out here. :)
>     > > >>>     > >
>     > > >>>     > > --
>     > > >>>     > > Thanks,
>     > > >>>     > > Chris pedersen
>     > > >>>     > >
>     > > >>>     >
>     > > >>>     >
>     > > >>>     >
>     > > >>>     > --
>     > > >>>     > Thanks,
>     > > >>>     > Chris pedersen
>     > > >>>     >
>     > > >>>
>     > > >>>
>     > > >>>
>     > > >>
>     > > >>
>     > > >> --
>     > > >> Thanks,
>     > > >> Chris pedersen
>     > > >>
>     > > >
>     > > >
>     > > >
>     > > > --
>     > > > Thanks,
>     > > > Chris pedersen
>     > > >
>     > >
>     >
>     >
>     >
>     > --
>     > Thanks,
>     > Chris pedersen
>     >
>
>
>
>     --
>     Thanks,
>     Chris pedersen
>
>
>


-- 
Thanks,
Chris pedersen