[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CloudStack - KVM / Ceph Performance


https://ceph.com/geen-categorie/ceph-validate-that-the-rbd-cache-is-active/

On Tue, Jun 26, 2018, 17:16 Adam Witwicki <awitwicki@xxxxxxxxxxxxx> wrote:

> The speed difference between Centos and Ubuntu KVM hosts is 3 fold.
>
> How do I check the rbd cache configuration on KVM hosts?
>
> Thanks
>
> Adam
>
> -----Original Message-----
> From: Andrija Panic <andrija.panic@xxxxxxxxx>
> Sent: 26 June 2018 15:00
> To: users <users@xxxxxxxxxxxxxxxxxxxxx>
> Subject: Re: CloudStack - KVM / Ceph Performance
>
> ** This mail originated from OUTSIDE the Oakford corporate network. Treat
> hyperlinks and attachments in this email with caution. **
>
> You are obviously hitting some issues on Centos vs Ubuntu as KVM host, and
> can be due to difference in kernel, ceph (librbd), libvirt/qemu...
>
> But we have been using ceph (older, hammer release) with Ubuntu 14.04 and
> dont expect any miracle on single volume. BUT if you i.e. start Bitlocker
> drive encryption on multiple VMs at same time, you will be able to nicely
> saturate cluster to its full performance capacity.
>
> It's known feature that CEPH "excellent on parallel streams /io" i.e.
> single volume can't really reach nowhere near full performance of the
> cluster.
>
> Again questions about rbd cache configuration on KVM hosts?
>
> If you will eventually put more serious workloads on this specific
> cluster, it ain't gonna perform satisfactory for any customer - that is
> reality - been there, done that... moved away...
> Sorry for being too honest...
> (We had same size cluster, just used  Intel DC ssds for journals,
> collocated few journals on single SSD - not very optimal thought)
>
> Cheers
>
> On Jun 26, 2018 15:40, "Adam Witwicki" <awitwicki@xxxxxxxxxxxxx> wrote:
>
> Used the same guest VM, with a Ubuntu and Centos KVM host
>
> The Ubuntu host performed as expected, the guest on Centos KVM host is
> very slow
>
> I have been testing with KVM virtio drivers packaged in centos7 and
> downloaded them for windows guests
>
> Thanks
>
> Adam
>
>
> -----Original Message-----
> From: Simon Weller <sweller@xxxxxxx.INVALID>
> Sent: 26 June 2018 13:35
> To: users <users@xxxxxxxxxxxxxxxxxxxxx>
> Subject: Re: CloudStack - KVM / Ceph Performance
>
> ** This mail originated from OUTSIDE the Oakford corporate network. Treat
> hyperlinks and attachments in this email with caution. **
>
>
> I assume you've checked to make sure you're using the KVM virtio drivers,
> right?
>
>
> - Si
>
>
> ________________________________
> From: Ivan Kudryavtsev <kudryavtsev_ia@xxxxxxxxx>
> Sent: Tuesday, June 26, 2018 6:35 AM
> To: users
> Subject: Re: CloudStack - KVM / Ceph Performance
>
> Hello, Adam. Try to attach md raid0 of two Ceph vols inside VM. I have
> heard that people who run ceph use that practice to improve results. I
> don't know what you are expecting, but read somewhere that all ops to a
> single RBD volume are queued in a single IO queue, thus RAID0 helps here,
> increasing the speed almost twice. I mean you don't see a ceph cluster
> overall performance, you see a librbd+qemu performance in your tests.
>
> вт, 26 июн. 2018 г., 15:57 Adam Witwicki <awitwicki@xxxxxxxxxxxxx>:
>
> > We are days away from completing our production CloudStack / Ceph
> > deployment but have run into a small challenge in regards to Ceph
> > performance. As you run a setup similar to ours I was wondering if
> > you've seen a similar issue?
> >
> >
> >
> > Our Ceph cluster consists of 60 X 7.2K SAS drive, spread over 5 nodes
> > each with a 960GB Samsung NVMe. Our KVM hypervisors are connected to
> > this nodes via 2 x 10Gbps links, per node.
> >
> >
> >
> > 3 centos Cloudstack management servers 4.11.0
> >
> > 4 centos KVM hosts
> >
> >
> >
> > The RAW Ceph performance looks great!
> >
> >
> >
> > CEPH RAW
> >
> > *Average IOPS*
> >
> > *Max IOPS*
> >
> > *MB/sec*
> >
> > *Write 4K*
> >
> > 5,885
> >
> > 6,190
> >
> > 22.9907
> >
> > *Read 4K*
> >
> > 28,985
> >
> > 35,025
> >
> > 113
> >
> > *Write 4MB*
> >
> > 204
> >
> > 219
> >
> > 816
> >
> > *Read 4MB*
> >
> > 361
> >
> > 399
> >
> > 1447.51
> >
> >
> >
> > However when we run the tests from within KVM we get significantly
> > reduced performance, particular in writes:
> >
> >
> >
> > VM Running on KVM Hypervisor
> >
> > *Average IOPS*
> >
> > *Max IOPS*
> >
> > *MB/sec*
> >
> > *Write 4K*
> >
> > 946.07
> >
> > 1390
> >
> > 3.69
> >
> > *Write 4MB*
> >
> > 472
> >
> > 614
> >
> > 116
> >
> >
> >
> > We believe the libvirt/ceph library could be causing the problem with
> > in centos as we don't see this on a Ubuntu KVM HOST
> >
> >
> >
> > Kind Regards,
> >
> > [image: Oakford Internet Services] <http://www.oakfordis.com/>
> >
> > *Adam Witwicki* | Hosted Systems Specialist
> >
> > 01380 710278 / awitwicki@xxxxxxxxxxxxx
> >
> > *Oakford Internet Services* Office: 01380 888088
> > 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT
> > www.oakfordis.com<http://www.oakfordis.com> sales@xxxxxxxxxxxxx
> >
> >
> >
> > Disclaimer Notice:
> > This email has been sent by Oakford Technology Limited, while we have
> > checked this e-mail and any attachments for viruses, we can not
> > guarantee that they are virus-free. You must therefore take full
> > responsibility for virus checking.
> > This message and any attachments are confidential and should only be
> > read by those to whom they are addressed. If you are not the intended
> > recipient, please contact us, delete the message from your computer
> > and destroy any copies. Any distribution or copying without our prior
> > permission is prohibited.
> > Internet communications are not always secure and therefore Oakford
> > Technology Limited does not accept legal responsibility for this message.
> > The recipient is responsible for verifying its authenticity before
> > acting on the contents. Any views or opinions presented are solely
> > those of the author and do not necessarily represent those of Oakford
> Technology Limited.
> > Registered address: Oakford Technology Limited, 10 Prince Maurice
> > Court, Devizes, Wiltshire. SN10 2RT.
> > Registered in England and Wales No. 5971519
> >
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>