osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[kuryr] [kuryr-kubernetes] Unable to use kubernetes-related things after rebooting host


My server that I deployed devstack was halted suddenly. I don't know what
exact reason is but, that made rebooting devstack.

Thanks!

================================================
*Jaewook Oh* (���)
IISTRC - Internet Infra System Technology Research Center
369 Sangdo-ro, Dongjak-gu,
06978, Seoul, Republic of Korea
Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618
E-mail : jwoh95 at dcn.ssu.ac.kr
================================================


2019ë?? 4ì?? 15ì?¼ (ì??) ì?¤í?? 5:23, Daniel Mellado Area <dmellado at redhat.com>ë??ì?´ ì??ì?±:

> Overall and besides of what Michal said, yeah, openstack hasn't been
> designed to be rebootable.
> A looong time ago and before the migration to systemd services there was a
> script that restarted the screen instances [1]
>
> That said, not there since 2016 so please restack whenever you reboot. In
> any case, was there any reason for rebooting?
> I'd check if just restarting service would be enough for you.
>
> Thanks!
>
> [1]
> https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=9ba49cd82631ada84b8458b370ac535985d57725
>
> On Mon, Apr 15, 2019 at 9:58 AM MichaÅ? Dulko <mdulko at redhat.com> wrote:
>
>> On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
>> > Hello kuryr-kubernetes team,
>> >
>> > currently, I deployed openstack with devstack, and I added
>> "kuryr-kubernetes" to use kubernetes vim.
>> > At first it worked fine and no error was occurred,
>> > so I made some c-vnfs with kuryr-kubernetes.
>> > But after rebooting the host, the systems that worked well before are
>> not working normally at all.
>> >
>> > First, I deployed an example one ('redis') with kubectl command, (Not
>> using Tacker, but only using kubectl cli)
>> > it showed below error,
>> >
>> > Events:
>>
>> â??obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z"
>> >   Type     Reason                  Age    From               Message
>>
>>  â??,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f
>> >   ----     ------                  ----   ----               -------
>>
>>  â??alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]}
>> >   Normal   Scheduled               2m25s  default-scheduler
>> Successfully assigned default/redis-master-6fbbc44567-ms8nâ??Apr 15 11:36:52
>> master tacker-server[1881]:  from (pid=1881) request
>> /usr/local/lib/python2.7/dist-packages/kubernetes
>> > k to master
>>                                                 â??/client/rest.py:219
>> >   Warning  MissingClusterDNS       2m25s  kubelet, master    pod:
>> "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-â??Apr 15 11:36:52
>> master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG
>> tacker.vnfm.infra_drivers.kubernetes.kuberne
>> > 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP
>> configured and cannot create Pod using "ClusterFirst" polâ??tes_driver [-]
>> status: Pending from (pid=1881) create_wait
>> /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube
>> > icy. Falling back to "Default" policy.
>>                                                â??rnetes_driver.py:137
>> >   Warning  FailedCreatePodSandBox  19s    kubelet, master    Failed
>> create pod sandbox: rpc error: code = Unknown desc â??Apr 15 11:36:52 master
>> tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG
>> tacker.vnfm.infra_drivers.kubernetes.kuberne
>> > = failed to set up sandbox container
>> "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network
>> for podâ??tes_driver [-] VNF initializing status: ['default',
>> 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/
>> >  "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up
>> pod "redis-master-6fbbc44567-ms8nk_default"
>> networâ??tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141
>> > k: Got invalid status code from CNI daemon.; Traceback (most recent
>> call last):                                        â??Apr 15 11:36:52 master
>> tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING
>> tacker.vnfm.infra_drivers.kubernetes.kuber
>> >   File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line
>> 80, in run                                      â??netes_driver [-] VNF
>> Creation failed: Resource creation is not completed within 500 seconds as
>> creation of stack defau
>> >     vif = self._add(params)
>>
>> "Got invalid status code from CNI daemon" means that you should take a
>> look at kuryr-cni pods logs.
>>
>> >
>> > Are there any reason for this error? And how can I resolve this problem?
>>
>> In general I'm pretty sure DevStack is not designed to be rebootable,
>> so it's probably some ip route being gone or some service that haven't
>> restarted. If you really want to debug this instead of firing a new
>> DevStack, start from checking all "devstack@*" services in systemd.
>>
>> > Best Regards,
>> > Jaewook.
>> > ================================================
>> > Jaewook Oh (���)
>> > IISTRC - Internet Infra System Technology Research Center
>> > 369 Sangdo-ro, Dongjak-gu,
>> > 06978, Seoul, Republic of Korea
>> > Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618
>> > E-mail : jwoh95 at dcn.ssu.ac.kr
>> > ================================================
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190416/97650c97/attachment-0001.html>