Wednesday, December 23, 2015

Cleaning Up from the Change Block Tracking (CBT) Bug 2137546

There's a couple of ways to resolve the potential CBT corruption bug - as noted at VMware ESXi 6.0, Patch ESXi600-201511401-BG: Updates esx-base (2137546).

First, you need to load the latest build onto your ESXi hosts located at VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545) on affected hosts.

After you update your ESXi hosts, you can find directions at Changed Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201) to Reset CBT after you patch or by following an alternate method of resetting change block tracking by doing the following for each VM on a patched host:

  1. Disable CBT Tracking 
  2. Create a Snapshot - none can exist
  3. Remove the Snapshot
  4. Re-Enable CBT Tracking

Veeam has a nice script to resolve the issue - https://www.veeam.com/kb1113

The only thing lacking, for me, is change tracking for virtual machines that have already had the CBT reset. I'm not going to be able to patch all my hosts at once, but progress A.S.A.P - and not need to repeat resets unnecessarily.

For my backups, I've been kicking off a once a week disabled CBT backup on all my VMware jobs in Veeam Backup and Recovery with the following script: https://github.com/cajeeper/Veeam/blob/master/Run-JobsWithoutCBT.ps1

I wrote a script in PowerCLI - similar to Veeams - that resets the virtual machine's CBT tracking by following the mentioned change block tracking reset steps above, but I've added additional tagging to indicate prior CBT resets on VMs, so I'm not attempting to continuously repair them all and I have some change tracking for future reference.

You can find the latest version at: https://github.com/cajeeper/PowerCLI/blob/master/Reset-VM-CBT.ps1

<#  
 .SYNOPSIS  
  Script reset VMs CBT and tag them as being reset if ESXi600-201511001 has been loaded on its host.
    
 .DESCRIPTION
  CBT Bug VMWare KB 2137546 is no bueno. I wanted a way to patch VMs possibly affected
  and keep track of ones that were patched - so as I progress through and patch for
  this bug, I don't have to keep running the patch process on  VMs multiple time
  unnecessarily.
 
 .NOTES   
  Author   : Justin Bennett   
  Date     : 2015-12-23
  Contact  : http://www.allthingstechie.net
  Revision : v1.0
  Changes  : v1.0 Original
#>
#Connect-VIServer myvCenterServer.local

#Show Progress
$showProgress = $true

#Gather VM Hosts
$ESXHosts = Get-VMHost

#Gather VM Hosts that have corrected for CBT Bug build
$patchedESXHosts =  $ESXHosts | ? { $_.Build -ge 3247720 }

#Gather VMs from ESX hosts with corrected for CBT Bug build
$VMs = $patchedESXHosts | Get-VM

#Create tags if necessary
$nul = New-TagCategory PowerCLI -ErrorAction SilentlyContinue
$nul = New-Tag -Name ResetCBT -Category PowerCLI -Description "bit.ly/1U5nyxz" -ErrorAction SilentlyContinue

#Get the ResetCBT Tag
$Tag = Get-Tag ResetCBT

#VMs already patched
$existingResetCBTVMs = (Get-TagAssignment -Category "PowerCLI" | ? { $_.Tag.Name -eq $Tag.Name -and $_.Entity.Uid -like "*VirtualMachine=*"} | select Entity).Entity

#Gather VMs not already CBT Reset and ChangeTrackingEnabled is Enabled (No need to reset if CBT already disabled)
$resetCBTVMs = $VMs | ? { $_.Id -notin $existingResetCBTVMs.Id -and ($_.ExtensionData.Config).ChangeTrackingEnabled }
$resetCBTVMsCount = 0

if ($resetCBTVMs.Count -gt 0) { $resetCBTVMs | % {$i=0} {
  $i++
  if ($_.PowerState -eq "PoweredOn" -and ($_ | Get-View).snapshot -eq $null) {
   try {
    #Disable CBT Spec
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Disabling CBT" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $VMConf = New-Object VMware.Vim.VirtualMachineConfigSpec 
    $VMConf.ChangeTrackingEnabled = $false

    $_.ExtensionData.ReconfigVM($VMConf)

    #Creating snapshot
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Creating Snapshot to Clear CBT" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $snap=$_ | New-Snapshot -Name 'Clear CBT'
    
    #Removing snapshot
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Removing Snapshot" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $snap | Remove-Snapshot -confirm:$false
    
    #Enable CBT
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Enabling CBT" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $VMConf.ChangeTrackingEnabled = $true
    $_.ExtensionData.ReconfigVM($VMConf)
    
    #Tagging Reset CBT VM
    $nul = $_ | New-TagAssignment $Tag
    
    $resetCBTVMsCount++
   } catch { write-warning "Failed to reset CBT on VM: $($_.Name)" }
  } else { 
   if($_.PowerState -ne "PoweredOn") { write-warning "VM: $($_.Name) Not completed - Needs to be in PoweredOn state" }
   if(($_ | Get-View).snapshot -ne $null) { write-warning "VM: $($_.Name) Not completed - Needs to have no existing snapshots" }
  }
 }
} else { write-warning "No VMs to Reset CBT on" }

New-Object -TypeName PSCustomObject -Property ([ordered]@{
 ESXHosts = $ESXHosts.Count
 patchedESXHosts = $patchedESXHosts.Count
 VMCount = $VMs.Count
 existingResetCBTVMs = $existingResetCBTVMs.Count
 resetCBTVMs = $resetCBTVMs.Count
 CompletedCBTVMs = $resetCBTVMsCount
})
Formatted for web with http://codeformatter.blogspot.com/ 

Best of luck!

Script Running on Example VMs

Script Running on Example VMs

Script Finished running on Example VMs

vSphere Tags added to VMs with CBT Reset

vSphere Tag added to VM with CBT Reset

No comments: