This article explains how to troubleshoot orphaned VMware Consolidate Helper-0 snapshots caused by failed VMware backups.
Follow me

If you are using any of the snapshot-based backup solutions for your virtualized machines—Veeam Backup & Replication and vRanger being the two most popular—then from time to time you are probably going to see a failed backup. This is a common occurrence even with traditional backup solutions, but in this situation it can have a drastic effect on your production environment.

VMware Backups Troubleshoot the Consolidate Helper 0 backup error 1

While I can only directly speak for the effect with Veeam, as it is the only one I’ve used, the situation described here as I understand it is common to all of the VMware-centric backup systems. What happens is specific to situations where bandwidth is constrained, like when you are replicating over the WAN to off site. Backup job A runs and successfully completes. The last stage of the backup process is removing the temporary snapshot that is used to create the backup; this can be a time-intensive process depending on how large the backup (and thus the snapshot) is. During this process, backup job B attempts to create a snapshot and begins backing up the same virtual machine. When this happens, the VMware helper that was trying to merge the original snapshot (<disk>-delta.vmdk) back into the disk file (.vmdk) becomes orphaned. Furthermore, VMware still believes the snapshot to be locked by the original process.

In my experience, there are four steps to troubleshooting this issue and getting the delta files to roll back up into the disk files. You hopefully will not have to complete all four, as they are steps for “this fix isn’t working.” The first two can be done on a running VM, so you incur no downtime; the second two require you to take down the VM.

VMware Backups Troubleshoot the Consolidate Helper 0 backup error 2

Step 1: Unlock the snapshot ^

The first step is to get the snapshot unlocked. The trick to this is to migrate the VM from the host it is on to another and then back again. Those of you using a vCenter-based infrastructure should know all about this. Just vMotion the machine, not the datastore, from one host to another.

It is important to vMotion the machine back to the original host. If you don’t, you get introduced to another error. Once the process is finished, you should be able to go into Snapshot Manager and click the “Delete All” button to clear them all out. Don’t be alarmed if the process of “Remove all snapshots” seems to hang on 95%; this is normal operation and, depending on the size and quantity of the snapshots, this process can take many, many hours.

Step 2: Trick vCenter ^

In some cases, after you perform the migration and you open Snapshot Manager, it appears that the VM has magically gotten rid of all of its snapshots. If you browse the datastore where the VM resides, using either the GUI VI Client or the CLI via SSH, you will still see the delta files there. If this is the case, you can trick vCenter into showing you the snapshots again by creating another snapshot manually (right-click VM, choose snapshot, “Take Snapshot”). When done, all of your Consolidate Helpers will reappear. After that, try to Delete All from the Snapshot Manager again.

VMware Backups Troubleshoot the Consolidate Helper 0 backup error 3

Step 3: Migrate the datastore ^

If you have gotten to this point, I hate to tell you but you are now looking at some down time from here on out. If the snapshots still haven’t moved, the next step in my process is to shut down the virtual machine and migrate the datastore. Yes, I know, you are probably screaming at the screen about the fact that, with ESX 4, you don’t have to shut down to migrate the data any more. However, if you have reached this step, there are most likely more delta files than the VI client could handle before reaching the timeout limit, and shutting down makes the process faster and more robust. Migrating the datastore will have the effect of rolling the deltas back up if the process completes successfully.

Step 4: Convert the virtual machine ^

This step is for those of you (myself included) who ignore the problem too long. ESX is only capable of handling 32 snapshots for any given VM. Beyond that, trying to Delete All will not work. Neither will migrating the datastore. In this case, you need to install VMware’s standalone Converter tool on the VM and perform what’s referred to as a v2v, or virtual to virtual, conversion. I’ve seen reference to just using the VI client to clone the VM, but this process was defined to me by VMware support, so I’ll trust their judgment.

I have only reached this step once, and I hope to avoid it from here on. Current Windows activation will not survive this process, so at the least you will have to reactivate. At worst, you might actually have to call Microsoft and have them manually activate your server.

5 Comments
  1. Jeremy 11 years ago

    Thanks for this troubleshooting guide. I had to forcefully abort a Veeam network mode backup that left a locked snapshot on one of the VMs. I tried vMotioning to another host and rebooting vCenter and still had the problem. But then after reading your “Step 1” procedure I vMotioned it back to the original host and the snapshot was unlocked.

  2. Kevin Jackson 11 years ago

    Step #1 usually works for me, and that’s definitely the easiest. I’ve also had luck logging straight into the host gui that the guest resides on and deleting the snapshot from the host (bypassing vcenter). I had one time when a volume totally filled up with snapshots before I realized it, it took down all the guests on the volume. Not fun.

  3. raydha 11 years ago

    what do you mean by steps no.4
    “In this case, you need to install VMware’s standalone Converter tool on the VM”

    can i just install the vmware standalone converter on the vcenter and do the conversion from vcenter?

  4. Jason Lehman 10 years ago

    This is a great guide, thanks for posting it. I will add my experience.
    I got the whole way to step 4, my particular VM had 35 snapshots. I believe those snapshots were created by vRanger, but never deleted for some reason. Once I got to step 4, I called VMware. They suggested the same 4 steps to try (maybe they were referencing this post, lol) They had me install VMware’s standalone Converter tool (dl from VMWare’s website, dont use the add on, that in vCenter) on MY pc. I had to power down the VM w/ all the snapshots, then v2v it. Basically, the v2v consolidates all the vmdk files (snapshots) when it creates the “new” VM. I just added the letter B to the end of the “new” VM’s name. The only place you will see that is in vCenter, this will NOT change the VM’s computer name in the OS. For me it was Win2k8R2, & I did NOT have to reactivate the license. Here are the issues I ran into to.
    1. Once the “new” VM booted up, I had to do a repair install of VMWare tools. The mouse was terrible while in the VM console.
    2. I left the default network adapter type selected in the v2v setup, (which was type E1000)& we used the VMXNET 3, originally. So when Windows booted, it saw a new NIC & did NOT retain the static IP info. I deleted the E1000 & added the VMXNET 3; but still had to config the IP info. This could cause major issues if you have an app that is binded / bound (sorry) to a specific NIC (and yes the repair install of VMWare tools can mess up the bindings to.) Make sure you also select the correct Network Label/VLan; if you have multiple; in the settings of the Virtual Machine Properties.
    The server is back up & seems to be running fine. All the vmdk files have been consolidated. I still have the original powered off, just in case. Eventually I hope to delete it.

  5. Matthew Shapiro 10 years ago

    Great write up! It took me a long while to find this page, but it finally held the answer to my 3948 error with VMware Data Recovery. It did, in fact, turn out that the VMs that were reporting the error had over 35 snapshots so had to go the v2v conversion route, but that fixed the issue. Thank you!

Leave a reply

Your email address will not be published.

*

© 4sysops 2006 - 2022

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account