OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CloudStack - KVM / Ceph Performance


Rbd cache configured and active on KVM side ? What guest OS you test within
- you get these last results from inside VM as I understand?

Do you have ceph (client side on kVM hosts) version difference between
centos and ubuntu host? Kernel upgrade on centos (to 4.x) makes any
difference ? (Very easy to test i.e via elrepo kernel for centos)

Best
Andrija

On Tue, Jun 26, 2018, 11:57 Adam Witwicki <awitwicki@xxxxxxxxxxxxx> wrote:

> We are days away from completing our production CloudStack / Ceph
> deployment but have run into a small challenge in regards to Ceph
> performance. As you run a setup similar to ours I was wondering if you’ve
> seen a similar issue?
>
>
>
> Our Ceph cluster consists of 60 X 7.2K SAS drive, spread over 5 nodes each
> with a 960GB Samsung NVMe. Our KVM hypervisors are connected to this nodes
> via 2 x 10Gbps links, per node.
>
>
>
> 3 centos Cloudstack management servers 4.11.0
>
> 4 centos KVM hosts
>
>
>
> The RAW Ceph performance looks great!
>
>
>
> CEPH RAW
>
> *Average IOPS*
>
> *Max IOPS*
>
> *MB/sec*
>
> *Write 4K*
>
> 5,885
>
> 6,190
>
> 22.9907
>
> *Read 4K*
>
> 28,985
>
> 35,025
>
> 113
>
> *Write 4MB*
>
> 204
>
> 219
>
> 816
>
> *Read 4MB*
>
> 361
>
> 399
>
> 1447.51
>
>
>
> However when we run the tests from within KVM we get significantly reduced
> performance, particular in writes:
>
>
>
> VM Running on KVM Hypervisor
>
> *Average IOPS*
>
> *Max IOPS*
>
> *MB/sec*
>
> *Write 4K*
>
> 946.07
>
> 1390
>
> 3.69
>
> *Write 4MB*
>
> 472
>
> 614
>
> 116
>
>
>
> We believe the libvirt/ceph library could be causing the problem with in
> centos as we don’t see this on a Ubuntu KVM HOST
>
>
>
> Kind Regards,
>
> [image: Oakford Internet Services] <http://www.oakfordis.com/>
>
> *Adam Witwicki* | Hosted Systems Specialist
>
> 01380 710278 / awitwicki@xxxxxxxxxxxxx
>
> *Oakford Internet Services* Office: 01380 888088
> 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT
> www.oakfordis.com sales@xxxxxxxxxxxxx
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>