osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CloudStack - KVM / Ceph Performance


Hello, Adam. Try to attach md raid0 of two Ceph vols inside VM. I have
heard that people who run ceph use that practice to improve results. I
don't know what you are expecting, but read somewhere that all ops to a
single RBD volume are queued in a single IO queue, thus RAID0 helps here,
increasing the speed almost twice. I mean you don't see a ceph cluster
overall performance, you see a librbd+qemu performance in your tests.

вт, 26 июн. 2018 г., 15:57 Adam Witwicki <awitwicki@xxxxxxxxxxxxx>:

> We are days away from completing our production CloudStack / Ceph
> deployment but have run into a small challenge in regards to Ceph
> performance. As you run a setup similar to ours I was wondering if you’ve
> seen a similar issue?
>
>
>
> Our Ceph cluster consists of 60 X 7.2K SAS drive, spread over 5 nodes each
> with a 960GB Samsung NVMe. Our KVM hypervisors are connected to this nodes
> via 2 x 10Gbps links, per node.
>
>
>
> 3 centos Cloudstack management servers 4.11.0
>
> 4 centos KVM hosts
>
>
>
> The RAW Ceph performance looks great!
>
>
>
> CEPH RAW
>
> *Average IOPS*
>
> *Max IOPS*
>
> *MB/sec*
>
> *Write 4K*
>
> 5,885
>
> 6,190
>
> 22.9907
>
> *Read 4K*
>
> 28,985
>
> 35,025
>
> 113
>
> *Write 4MB*
>
> 204
>
> 219
>
> 816
>
> *Read 4MB*
>
> 361
>
> 399
>
> 1447.51
>
>
>
> However when we run the tests from within KVM we get significantly reduced
> performance, particular in writes:
>
>
>
> VM Running on KVM Hypervisor
>
> *Average IOPS*
>
> *Max IOPS*
>
> *MB/sec*
>
> *Write 4K*
>
> 946.07
>
> 1390
>
> 3.69
>
> *Write 4MB*
>
> 472
>
> 614
>
> 116
>
>
>
> We believe the libvirt/ceph library could be causing the problem with in
> centos as we don’t see this on a Ubuntu KVM HOST
>
>
>
> Kind Regards,
>
> [image: Oakford Internet Services] <http://www.oakfordis.com/>
>
> *Adam Witwicki* | Hosted Systems Specialist
>
> 01380 710278 / awitwicki@xxxxxxxxxxxxx
>
> *Oakford Internet Services* Office: 01380 888088
> 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT
> www.oakfordis.com sales@xxxxxxxxxxxxx
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>