[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Secondary storage issue with XenServer


Please excuse my post.
I think I found the issue. I have deployed cloudstack 4.9 instead of 4.11.

Best regards,
Jordan

From: Yordan Kostov
Sent: Tuesday, August 21, 2018 12:04 PM
To: users@xxxxxxxxxxxxxxxxxxxxx
Subject: Secondary storage issue with XenServer

Hello everyone,

                I recently found about Cloudstack and decided to give it a try and I really like it so far but I bumped into a very weird config issue.
                Here is my test environment.

                10.10.10.200 - XenServer 7.1 host (all free patches installed, ntp synced) eth0 is the management and also the only interface connected to the network. There are no bonds.

                10.10.10.201 - Centos 7 NFS server:
                                - firewalld disabled
                                - /etc/exports is:
                                                                /cloud_storage/isos     *(rw,async,no_root_squash,no_subtree_check)
/cloud_storage/vms     *(rw,async,no_root_squash,no_subtree_check)

                10.10.10.210 - Cloudstack management server (OS is centos 7, ntp synced)
                                Management network range - 10.10.10.202-209
                                Guest network range - 10.10.10.211-219
                                Storage network range - no ips set.
                                Primary storage is set to use the XenServer host local storage.

                10.10.10.150 - my PC address in the test network (10.10.10.0/24, gate 10.10.10.1)

Zone, pod and cluster and host are created successfully.
But when I try to add secondary NFS server in Cloustack the operation fails.
Going through logs I see the following highlights (full log snipets are attached to this mail):

Cloudstack-server.log  -

                Line 193: 2018-08-21 11:04:08,865 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-413:ctx-37d82b72) (logid:fac8aabf) createFileSR failed! due to the following: SR_BACKEND_FAILURE_61File SR creation error [opterr=fail to mount FileSR. Errno is 32]
                Line 221: 2018-08-21 11:04:08,865 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-413:ctx-37d82b72) (logid:fac8aabf) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template due to com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to the following: SR_BACKEND_FAILURE_61File SR creation error [opterr=fail to mount FileSR. Errno is 32]

                XenServer SMlog -
Aug 21 11:04:08 cloudstack-office SM: [20645] ['mount', '--bind', '/var/cloud_mount/12fb334c-64e6-3206-9373-7ae0627f83d2/template/tmpl/1/1', '/var/run/sr-mount/f517ce41-148e-4f88-9bdc-15d7e2d0eaa6']
Aug 21 11:04:08 cloudstack-office SM: [20645] FAILED in util.pread: (rc 32) stdout: '', stderr: 'mount: mount /var/cloud_mount/12fb334c-64e6-3206-9373-7ae0627f83d2/template/tmpl/1/1 on /run/sr-mount/f517ce41-148e-4f88-9bdc-15d7e2d0eaa6 failed: Stale file handle
Aug 21 11:04:08 cloudstack-office SM: [20645] '
Aug 21 11:04:08 cloudstack-office SM: [20645] Raising exception [61, File SR creation error [opterr=fail to mount FileSR. Errno is 32]]
Aug 21 11:04:08 cloudstack-office SM: [20645] lock: released /var/lock/sm/f517ce41-148e-4f88-9bdc-15d7e2d0eaa6/sr
Aug 21 11:04:08 cloudstack-office SM: [20645] ***** generic exception: sr_attach: EXCEPTION <class 'SR.SROSError'>, File SR creation error [opterr=fail to mount FileSR. Errno is 32]

                XenSever -XenSource.log -
Aug 21 11:05:38 cloudstack-office xapi: [debug|cloudstack-office|109042 UNIX /var/lib/xcp/xapi|dispatch:SR.add_to_other_config D:e87a119cfb36|api_effect] SR.add_to_other_config
Aug 21 11:05:38 cloudstack-office xapi: [ info|cloudstack-office|109033 |sm_exec D:143bd363be92|xapi] Session.destroy trackid=a5d69dd0a4351c2751078822b0a8844b
Aug 21 11:05:38 cloudstack-office xapi: [error|cloudstack-office|109033 |SR.attach D:070d838f0344|backtrace] sm_exec D:143bd363be92 failed with exception Storage_interface.Backend_error(_)
Aug 21 11:05:38 cloudstack-office xapi: [error|cloudstack-office|109033 |SR.attach D:070d838f0344|backtrace] Raised Storage_interface.Backend_error(_)
Aug 21 11:05:38 cloudstack-office xapi: [error|cloudstack-office|109033 |SR.attach D:070d838f0344|backtrace] 1/8 xapi @ cloudstack-office Raised at file sm_exec.ml, line 216


What is weird is that I am able to mount the NFS through XenServer -> new storage -> NFS wizard (using both NFSv3 and NFSv4 protocols). When added I am able to create disks (a.k.a. the host has write access).

Some NFS connection/accessibility tests were run from the XenServer hosts with the following results:

-          RPC test
                [root@cloudstack-office tmp]# rpcinfo -t 10.10.10.201 nfs
program 100003 version 3 ready and waiting
program 100003 version 4 ready and waiting


-          Mount scan

[root@cloudstack-office /]# showmount -e 10.10.10.201

Export list for 10.10.10.201:

/cloud_storage/vms  *

/cloud_storage/isos *



Note: both of those work manually but not through cloudstack



-          NFS sr-proble returns this for both shares:
[root@cloudstack-office tmp]# xe sr-probe type=nfs device-config:server=10.10.10.201 device-config:serverpath=/cloud_storage/vms
<?xml version="1.0" ?>
<SRlist/>

                Any idea why the NFS cannot be added as secondary storage :(?
I am so out of ideas. Sorry for the long mail.


Best regards,
Jordan Kostov