[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

qemu2 images are being corrupted

Dear colleagues,

I'm posting as an anonymous user, because there's a thing that concerns me a little and I'd like to share my experience with you, so maybe some people could relate to the same. ACS is amazing, it solves my tasks for 6 years, I'm running a few ACS-backed clouds that contain hundreds and hundreds of VMs. I'm enjoying ACS really much, but there's a thing that scares me sometimes.

It happens pretty seldom, but the more VMs you have is the more chances you run into this glitch. It usually happens on the sly and you don't get any error messages in log-files of your cloudstack-management server or a cloudstack-agent, so you don't even know that something had happened until you see that a virtual machine is having major problems. If you're lucky, you see it on the same day when it happens, but if you aren't - you won't suspect anything unusual for a week, but at some moment you realize that the filesystem had become a mess and you can't do anything to restore it. You're trying to restore it from a snapshot, but if you don't have a snapshot that would be created before the incident, your snapshots won't help. :-(

I experienced it for about 5-7 times during the last 5-6 years and there are a few conditions that always present:
 * it happens on KVM-based hosts (I experienced itt with CentOS 6 and CentOS 7) with qcow2-images (either 0.10 and 1.1 versions);
 * it happens on primary storages running different filesystems (I experiences it with local XFS and network-based GFS2 and NFS);
 * it happens when a volume snapshot is being made, according to the log-files inside of a VM (guest's operating system's kernel starts complaining on a filesystem errors);
 * at the same time, as I wrote before, there are NO error messages in the log-files outside of a VM which disk image is corrupted;
 * but when you run `qemu-img check ...` to check the image, you may see a lot of leaked clusters (that's why I'd strongly advice to check each and every image one each and every primary storage at least once per hour by a script being run by your monitoring system, something kind of `for imagefile in $(find /var/lib/libvirt/images -maxdepth 1 -type f); do { /usr/bin/qemu-img check "${imagfile}"; if [[ ${?} -ne 0 ]]; then { ... } fi; } done`);
 * when it happens you can also find a record in the snapshot_store_ref table that refers to the snapshot on a primary storage (see an example here https://pastebin.com/BuxCXVSq) - this record should have been removed when the snapshot's state is being changed from "BackingUp" to "BackedUp", but it isn't being removed in this case. At the same time, this snapshot isn't being listed in the output of `qemu-img snapshot -l ...`, so that's why I suppose that the image is being corrupted when ACS deletes the snapshot that has been backed up (it tries to delete the snapshot, but something goes wrong, image is being corrupted, but ACS thinks that everything's fine and changes the status to "BackedUp" without a bit of qualm);
 * if you're trying to restore this VM's image from the same snapshot that has caused destruction or any other snapshot that has been made after that, you'll find the same corrupted filesystem inside, but the snapshot's image that is stored in your secondary storage doesn't show anything wrong when you run `qemu-img check ...` (so you can restore your image only if you have a snapshot that had been created AND stored before the incident).

As I wrote, I saw several times in different environments and different versions of ACS. I'm pretty sure that it's not only me who had such a luck to experience the same glitch, so let's share our stories. Maybe together we'll find out why does it happen and how to prevent that in future.

Thanks in advance,
An Anonymous ACS Fan