Wednesday, December 23, 2015

Cleaning Up from the Change Block Tracking (CBT) Bug 2137546

There's a couple of ways to resolve the potential CBT corruption bug - as noted at VMware ESXi 6.0, Patch ESXi600-201511401-BG: Updates esx-base (2137546).

First, you need to load the latest build onto your ESXi hosts located at VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545) on affected hosts.

After you update your ESXi hosts, you can find directions at Changed Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201) to Reset CBT after you patch or by following an alternate method of resetting change block tracking by doing the following for each VM on a patched host:

  1. Disable CBT Tracking 
  2. Create a Snapshot - none can exist
  3. Remove the Snapshot
  4. Re-Enable CBT Tracking

Veeam has a nice script to resolve the issue -

The only thing lacking, for me, is change tracking for virtual machines that have already had the CBT reset. I'm not going to be able to patch all my hosts at once, but progress A.S.A.P - and not need to repeat resets unnecessarily.

For my backups, I've been kicking off a once a week disabled CBT backup on all my VMware jobs in Veeam Backup and Recovery with the following script:

I wrote a script in PowerCLI - similar to Veeams - that resets the virtual machine's CBT tracking by following the mentioned change block tracking reset steps above, but I've added additional tagging to indicate prior CBT resets on VMs, so I'm not attempting to continuously repair them all and I have some change tracking for future reference.

You can find the latest version at:

  Script reset VMs CBT and tag them as being reset if ESXi600-201511001 has been loaded on its host.
  CBT Bug VMWare KB 2137546 is no bueno. I wanted a way to patch VMs possibly affected
  and keep track of ones that were patched - so as I progress through and patch for
  this bug, I don't have to keep running the patch process on  VMs multiple time
  Author   : Justin Bennett   
  Date     : 2015-12-23
  Contact  :
  Revision : v1.0
  Changes  : v1.0 Original
#Connect-VIServer myvCenterServer.local

#Show Progress
$showProgress = $true

#Gather VM Hosts
$ESXHosts = Get-VMHost

#Gather VM Hosts that have corrected for CBT Bug build
$patchedESXHosts =  $ESXHosts | ? { $_.Build -ge 3247720 }

#Gather VMs from ESX hosts with corrected for CBT Bug build
$VMs = $patchedESXHosts | Get-VM

#Create tags if necessary
$nul = New-TagCategory PowerCLI -ErrorAction SilentlyContinue
$nul = New-Tag -Name ResetCBT -Category PowerCLI -Description "" -ErrorAction SilentlyContinue

#Get the ResetCBT Tag
$Tag = Get-Tag ResetCBT

#VMs already patched
$existingResetCBTVMs = (Get-TagAssignment -Category "PowerCLI" | ? { $_.Tag.Name -eq $Tag.Name -and $_.Entity.Uid -like "*VirtualMachine=*"} | select Entity).Entity

#Gather VMs not already CBT Reset and ChangeTrackingEnabled is Enabled (No need to reset if CBT already disabled)
$resetCBTVMs = $VMs | ? { $_.Id -notin $existingResetCBTVMs.Id -and ($_.ExtensionData.Config).ChangeTrackingEnabled }
$resetCBTVMsCount = 0

if ($resetCBTVMs.Count -gt 0) { $resetCBTVMs | % {$i=0} {
  if ($_.PowerState -eq "PoweredOn" -and ($_ | Get-View).snapshot -eq $null) {
   try {
    #Disable CBT Spec
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Disabling CBT" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $VMConf = New-Object VMware.Vim.VirtualMachineConfigSpec 
    $VMConf.ChangeTrackingEnabled = $false


    #Creating snapshot
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Creating Snapshot to Clear CBT" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $snap=$_ | New-Snapshot -Name 'Clear CBT'
    #Removing snapshot
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Removing Snapshot" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $snap | Remove-Snapshot -confirm:$false
    #Enable CBT
    if($showProgress) { Write-Progress -Activity "Reset-VM CBT" -Status "$($i)/$($resetCBTVMs.Count): VM:$($_.Name) - Enabling CBT" -PercentComplete (($i/$resetCBTVMs.Count)*100) }
    $VMConf.ChangeTrackingEnabled = $true
    #Tagging Reset CBT VM
    $nul = $_ | New-TagAssignment $Tag
   } catch { write-warning "Failed to reset CBT on VM: $($_.Name)" }
  } else { 
   if($_.PowerState -ne "PoweredOn") { write-warning "VM: $($_.Name) Not completed - Needs to be in PoweredOn state" }
   if(($_ | Get-View).snapshot -ne $null) { write-warning "VM: $($_.Name) Not completed - Needs to have no existing snapshots" }
} else { write-warning "No VMs to Reset CBT on" }

New-Object -TypeName PSCustomObject -Property ([ordered]@{
 ESXHosts = $ESXHosts.Count
 patchedESXHosts = $patchedESXHosts.Count
 VMCount = $VMs.Count
 existingResetCBTVMs = $existingResetCBTVMs.Count
 resetCBTVMs = $resetCBTVMs.Count
 CompletedCBTVMs = $resetCBTVMsCount
Formatted for web with 

Best of luck!

Script Running on Example VMs

Script Running on Example VMs

Script Finished running on Example VMs

vSphere Tags added to VMs with CBT Reset

vSphere Tag added to VM with CBT Reset

Monday, December 7, 2015

Abandon Ship!

At my work, we've been using Hyper-V as our virtualization hypervisor since 2008. We started down a new path of hypervisors since the beginning of this year. This last summer we acted and have been switching our virtual hosts to vSphere 6.0. The processes of converting virtual machines has been very uneventful - 93 conversions with only 3 resulting in post migration errors. We still have 16 VMs left to convert, but we should be done by January 2016 and Hyper-V will be a distant hypervisor memory.

The point of this post is just to explain what worked for us to perform these Virtual Machine to Virtual Machine (V2V) conversions.

Step 1: Shutdown the virtual machine.

Step 2: Backup the Hyper-V Virtual Machine - CYA

Step 3: Convert the VHD/VHDX files to VMDK files. For this, I preferred using StarWinds Converter Tool. VMware recommends using the VMware vCenter Converter Standalone. (Mainly because I was able to wrap the CLI tool into PowerShell - see below.)

Step 4: Create a new template Virtual Machine in vCenter matching the Hyper-V machine.

Step 5: Upload the converted VMDK files to your vSphere Datastore.
Connecting to vCenter with the old vSphere Client worked best for us.

Step 6: Clone the VMDK disks to ensure they're in the correct datastore format. (See for detailed steps)
This step was added do the StarWind Converter tool putting thin provisioned disks multiextent format, which can cause the virtual machines to crash when snapshot'd - particularly during Step 9 for us when Veeam Backup and Recovery would kick off a backup. I'm not sure what different scenarios that the StarWind Converter will not cause this problem, but after I lost one of our Active Directory (AD) servers to this bug, this step was added and no problems since. Unfortunately, that AD server was hours to repair. Just do this step.
If you use the VMware Standalone Converter, you should be safe to pass

Step 7: Attach VMDK disks.

Step 8: Turn on new VM and finish post configuration - Linux: As long as the MAC address matched the old VM, nothing; Windows: Activate Licensing, Reconfigure IP Addresses, etc.

Step 9: Verify the system is still working.

Step 10: Backup the new VMware virtual machine.

Step 11: Remove the old Hyper-V virtual machine.

Important take away's I've learned:

  • Ensure you process has plan B, C, D's, etc...
  • Backup, Backup, Backup
  • Did I mention backup?
  • Be cautious when converting Active Directory Servers and any database (SQL) systems. If you can avoid V2V of these types of systems, do avoid them.
  • Get 10Gbps networking. Unless you like to sit and wait for files to backup/copy/upload, make sure your network can take the hit.

Also, I've created a PowerShell cmdlet to automate and speed up step 3.

  Function to convert streamline conversion of multiple VHD/VHDX files from one directory to VMware VMDK files in a different directory.  
  The first VHD/VHDX found is assumed to IDE and all sub-sequential be SCSI adapters.
  This function requires the use the StarWinds V2V console conversion tool in order to make this function.
  Also, Windows 8 / Server 2012 or above is required for the StarWinds tool to convert any VHDX files.
  Author   : Justin Bennett   
  Date     : 2015-12-07
  Contact  :
  Revision : v1.1 
  Changes  : v1.0 Original
    v1.1 Added starwindsConvert, vmdkAdapterType, and vmdkType parameters
 .PARAMETER convertFromDir  
  Name of directory location containing VHD/VHDX files
 .PARAMETER convertFromTo
  Name of directory location where VMDK files will be deposited
 .PARAMETER vmdkAdapterType
  Set vmdk disk adapter to IDE, SCSI, or FirstIDE to first disk IDE with subsequential vmdk disks set to SCSI
 .PARAMETER vmdkType
  Set vmdk disk type to VMDK_F - VMWare pre-allocated image, VMDK_S - VMWare growable image,
 VMDK_SO - VMWare stream-optimized image, or VMDK_VMFS - VMWare ESX server image.
 .PARAMETER starwindsConvert
  File path of the StarWindows Converter Executable, StarV2Vc.exe 
  C:\PS> #Convert files in C:\DirA to C:\DirB
  C:\PS> Convert-VHDtoVMDK -convertFromDir 'C:\DirA' -convertToDir 'C:\DirB'
function Convert-VHDtoVMDK {
      param (  
           [parameter(Mandatory=$True)] [ValidateScript({Test-Path $_})] [string] $convertFromDir,
     [parameter(Mandatory=$True)] [ValidateScript({Test-Path $_})] [string] $convertToDir,
     [parameter(Mandatory=$False)] [ValidateSet("IDE","SCSI","FirstIDE")] [string] $vmdkAdapterType = "SCSI",
     [parameter(Mandatory=$False)] [ValidateSet("VMDK_F","VMDK_S","VMDK_SO","VMDK_VMFS")] [string] $vmdkType = "VMDK_F",
     [parameter(Mandatory=$False)] [ValidateScript({Test-Path $_})] [string] $starwindsConvert = 'C:\Program Files (x86)\StarWind Software\StarWind V2V Image Converter\StarV2Vc.exe'
 #initial vmdk set to 
 if($vmdkAdapterType -eq "SCSI") { $vmdkadtype = "SCSI" } else { $vmdkadtype = "IDE" }
 $vmdkType = "vmdk_s"
 $files = Get-ChildItem -Path $convertFromDir -Filter *.vhd*
 foreach ($file in $files) {
  $start = get-date
  $run = "`& `"$($starwindsConvert)`" if=`"$($file.fullname)`" of=`"$($convertToDir)\$($file.basename).vmdk`" ot=$($vmdkType) vmdktype=$($vmdkadtype)"
  #run conversion
  Write-Progress -Activity "Converting VHD(X) to VMDK: From $($convertFromDir), To $($convertToDir)" -Status "File $($i)/$($files.count): $($, Started at $(get-date -Format g $start)"
  try { $output = Invoke-Expression $run } catch { $output = "Failed to convert file $($, Error $($_)"; write-warning $output; }
  $end = get-date
  New-Object PSCustomObject -Property ([ordered]@{
   "File"= $
   "Start"= $start
   "RunCMD"= $run
   "Output"= $output
   "End"= $end
  #subsequential vmdk set to SCSI if FirstIDE set
  if($vmdkAdapterType -eq "FirstIDE") { $vmdkadtype = "SCSI" }
 if(!($i -ge 1)) { write-warning "No files found in $($convertFromDir)" }
Formatted for web with 

Screenshot of Convert-VHDtoVMDK

StarWind V2V Converter -
VMware vCenter Converter Standalone -
Powering on a virtual machine on an upgraded host fails with the error: File [VMFS volume] VM-name/VM-name.vmdk was not found (2036572) -

Tuesday, December 1, 2015

Remove Windows Update using PowerShell and KB

I would like a native cmdlet in PowerShell to remove Windows Updates.

Until that time comes, I've created my own function that takes the native comobjects and searches for the update I want to remove by KB ID.

Bon App├ętit

Current version at

  Remove One to Many Windows Updates
  Remove One to Many Windows Updates from OS.
  Author   : Justin Bennett   
  Date     : 2015-12-01  
  Contact  :
  Revision : v1  
  C:\PS> #Uninstall One Update 
  C:\PS> Remove-WindowsUpdate 123456
  C:\PS> #Uninstall Multiple Updates
  C:\PS> Remove-WindowsUpdate 123456,456123
Function Remove-WindowsUpdate {
      param (  
           [parameter(Mandatory=$True)] $RemoveKB

 $Searcher = New-Object -ComObject Microsoft.Update.Searcher
 $RemoveCollection = New-Object -ComObject Microsoft.Update.UpdateColl

 #Gather All Installed Updates
 $SearchResult = $Searcher.Search("IsInstalled=1")

 #Add any of the specified KBs to the RemoveCollection
 $SearchResult.Updates | ? { $_.KBArticleIDs -in $RemoveKB } | % { $RemoveCollection.Add($_) }

 if ($RemoveCollection.Count -gt 0) {
  $Installer = New-Object -ComObject Microsoft.Update.Installer
  $Installer.Updates = $RemoveCollection
 } else { Write-Warning "No matching Windows Updates found for:`n$($RemoveKB|Out-String)" }
Formatted for web with