Reading Time: 2 minutes

Since version 4.x, VMware vSphere provides native API to perform image level backup of running VMs: VMware vSphere Storage APIs – Data Protection (formerly known as VMware vStorage APIs for Data Protection or VADP).

Those API are based on VMware snapshot to create a readable base VMDK disk that can be easily copied by a compatible backup solution using a full transfer or using incremental transfer with Changed Block Tracking (CBT) feature.

A backup product using VMware vSphere Storage APIs – Data Protection can backup vSphere virtual machines from a central backup server or virtual machine without requiring backup agents or requiring backup processing to be done inside each guest virtual machine on the ESX host. This offloads backup processing from ESX hosts and reduces costs by allowing each ESX host to run more virtual machines.

But those API are not always bug free… and can there can data corruption in specific build version and specific cases. Almost associated with the CBT based backup.

Here a brief table to summarize some build version affected by possible issues:

vShere version Solution Issues
8.0.2 Not available (yet) Possible CBT corruption after a virtual disk hot extend
KB 95965
6.7 Follow KB instruction The virtual disk is either corrupted or not a supported format when committing or creating snapshots
KB 2013520
6.5/6.7 Upgrade to VMware VDDK 6.7 or later version CBT reports larger area of changed blocks than expected if guest OS performed unmap on a disk
KB 59905
6.0.0 Update and follow KB instructions VMware vSphere 6 CBT issue
KB 2114076
KB 2136854
6.x Apply laters updates When users revert to a snapshot, it will automatically disable and enable CBT. If the snapshot is not a memory snapshot, enabling CBT will fail and CBT is disabled after reverting.
KB 71155
4.x/5.x Apply patches Possible backup data corruption after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled
KB 2090639

As you can notice there is actually on conditions that still can happen in an updated version of vSphere:

  • If you hot resize a VM disk on a vSphere 8.0.2 version, then CBT may have incorrect information. Actually this bug has no fix (yet). Funny that a similar bug was introducted in old 4.x and 5.x versions.
    So safer solution is don’t perform hot resize, but resize VMDK only on powered off VMs.
    Follow KB 95965 to know when a fix will be released!