Tuesday, October 11, 2016

SCOM Agent HealthService Not Starting

I have a few systems that developed an issue were the Microsoft Monitoring Agent service (HealthService), a System Center Operations Manager 2012 client agent service, would not start.

"Windows could not start the Microsoft Monitoring Agent on Local Computer. For more information, review System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code -2130771964"
"The description for Event ID 7024 from source Service Control Manager cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer. If the event originated on another computer, the display information had to be saved with the event. The following information was included with the event: Microsoft Monitoring Agent %%2164195332 The locale specific resource for the desired message is not present"

There's a couple TechNet articles in regards to this service not starting (permissions on registry entries and folder permissions), but I couldn't find my exact issue.
https://blogs.technet.microsoft.com/smsandmom/2008/04/30/opsmgr-2007-healthservice-service-fails-to-start-with-25362-warning/
https://blogs.technet.microsoft.com/henryzhu/2011/09/26/two-common-scom-agent-start-failure-issues/
https://support.microsoft.com/en-us/kb/2288515

I finally found the culprit in the Operations Manager Logs - it appeared to be a corrupt database issue.

"HealthService (4346) Health Service Store: Database recovery/restore failed with unexpected error -543."

"HealthService (4364) Health Service Store: Database C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\HealthServiceStore.edb requires logfiles 1441-1443 (C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\edb005A1.log - C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\edb.log) in order to recover successfully. Recovery could only locate logfiles up to 1442 (C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store\edb005A2.log)."

I found a non-official post that helped clear the database at systemcentercentral.com (http://www.systemcentercentral.com/flushing-the-health-service-and-state-and-cache/)

To resolve the issue:
  1. Move or Delete the database files located in "C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Health Service Store"
  2. Start the Microsoft Monitoring Agent service

    The database should be recreated if there's no other issues.
Moving corrupt database to a new folder.
Microsoft Monitoring Agent service running after corrupt database was moved.

Tuesday, September 13, 2016

Running a Linux Script for vSphere via Veeam Backup & Replication

I needed to do some proof of concept testing on the Pre-Freeze and Post-Thaw scripts to isolate some issues I've been having - rather than just beating my head against the wall all day.

Below are the steps I took to test Application-Aware processing for Linux Scripts for my test virtual machine.

Requirements:
Steps:
  1. Setup your Linux VM with SSH
    For Lubuntu

    sudo apt-get install openssh-server -y

    Installing SSH

  2. Copy your scripts to your Veeam Backup & Replication server

    Copying your scripts

  3. Add the script's execution to the Application-Aware Processing to your backup job.

    Edit the job,
    Go to Guest Processing section,
    Enable application-aware process and click Applications,
    Select your virtual machine and click Edit,
    Go to the Scripts tab, and
    Browse to your scripts that you want to run for Pre-freeze and Post-thaw

    Note: Don't forget to assign you local login username and password that has SSH and the correct file / execution permissions.
     


     
  4. Save the changes and run the backup job

    Veeam Backup and Recovery Backup Job Test Run

    Verfiying the scripts ran

Best of luck!

Wednesday, July 20, 2016

Automate Update of VM Template - Part 1 - Windows

Automate Update of VM Template - Part 1 - Windows
Automate Update of VM Template - Part 2 - Linux

Update: I've tested and now use this on a Windows Server 2016 VM template as well as 2012 R2.

I don't know about you, but I "love" having to maintain things manually and wasting my time.

Good news is that it only takes 200+ updates and hours of wasted time to patch a fresh installation of Windows Server 2012 R2 server.

One solution we've decided to streamline this task is to maintain a patched virtual machine (VM) template that has the most current Windows updates needed.

"That's a great idea, but you're still having to maintain a VM template." you might say.

Yeah, I don't have to maintain it - my scheduled task that kicks off a PowerCLI/PowerShell script maintains our VM template.

You can find the latest version at: https://github.com/cajeeper/PowerCLI/blob/master/Install-Windows-Update-for-Template.ps1

Here's a breakdown on what this script does:

  • Converts the VM template to a virtual machine
  • Starts the virtual machine
  • Issues PowerShell script block via VMWare Tools to trigger and install any needed Windows updates
  • Sleeps for 10 minutes to allow Windows Updates to reboot the OS as needed
  • Shuts the virtual machine down
  • Converts the virtual machine back to a VM template
  • Logs the results
Optionally:
  • Creates copy of template to another system/site
  • Shows the progress via the CLI
I've been using it for the last few months with success, so I figured its time to share - once again. Note: I've only been testing this on a Windows Server 2012 VM template.

Requirements for the script to function:
  • An existing Windows VM template that has VMware Tools installed and a network adapter to recieve Windows Updates
  • VMware vCenter - Tested on vCenter 6.0.0
  • A user/service account with permissions to Create, Remove, Start, Stop VMs and issue VM scripting commands in vCenter 6.0.0
  • A user/service account with permissions to install Windows Updates in the template virtual machine
  • Update the parameters with all your relevant template, location, etc. and if you want a post copy of the template to kick off.
Here's the last few months of Windows Updates that have been installed on every Fridays for my virtual machine template via this script:
Template Virtual Machine - Windows Update History

Template Virtual Machine - Recent Scripted Tasks and Copy of Template at Dev1 Datacenter
Best of luck!

Thursday, May 26, 2016

Configure Server 2012 R2 with RDP Session Host without the whole RDP-RDMS shebang! - Part 2 - Shadow Users

Configure Server 2012 and 2012 R2 with RDP Session Host without the whole RDP-RDMS shebang! - Part 1
Configure Server 2012 R2 with RDP Session Host without the whole RDP-RDMS shebang! - Part 2



Had a few people ask about shadowing users. If you've looked around, you'll see Microsoft dropped the ball from Windows Server 2008 R2 to 2012 - as they removed the shadowing capabilities.

For Windows 2012 R2, they've added it back. Now, if you've configured just a single RDP Session Host, like I've shown, you may be wondering "How can I shadow my users now that I'm all RDP-RDMS free?" Well, I'm glad to say that you're in luck.

A) Run the PowerShell commands below as Administrator:

Current full version of configuring RDP Session Host script at https://github.com/cajeeper/PowerShell/blob/master/Setup-Standalone-RDS-Host.ps1

 #Allow Shadowing Users
 # Values: 0 (No Remote Control), 1 (Full Control with user's permission), 2 (Full Control without user's permission), 3 (View Session with user's permission), 4 (View Session without user's permission)
 New-ItemProperty "hklm:\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" -Name Shadow -Value 2 -PropertyType "DWORD"

 #Update GPO for Shadowing Users
 gpupdate /force
Formatted for web with http://codeformatter.blogspot.com/ 

Example of running PowerShell script
 
B) Launch a shadowing session
  1. Open a PowerShell or Command Prompt as Administrator

    Example of running PowerShell as Administrator
     
  2. Find the session you want to shadow using:

    quser
       or
    qwinsta
     
  3. Connect to the session by using the following command on the server you're attempting to administrate (replace # with the correction session ID):

    mstsc /shadow:# /control
      or
    mstsc /shadow:# /control /noconsentprompt
Example of running quser and mstsc as Administrator

Example of shadowing user session


Cheers!

Thursday, January 28, 2016

PowerCLI to vMotion VMs off and Enable Maintenance for ESXi host

I'm using VMware vSphere 6 with no DRS enabled on my clusters. Initially, when I wanted to perform maintenance, it was a bit of process with many clicks through the web GUI.

I've been tinkering with PowerShell for years, so it was natural to start banging on PowerCLI to make my life easier administering my VMware clusters.

I've been using and growing the following script for the past 4 months. It seems to be making my ESXi patching and maintenance a little less laborious chore than the prior click-storms.

Enjoy!

You can find the latest version at: https://github.com/cajeeper/PowerCLI/blob/master/vMotion Off All VMs Host A to B.ps1

<#  
 .SYNOPSIS  
  Script to vMotion VMs off one ESXi host to do maintenance - non-DRS user.
    
 .DESCRIPTION
  I use this script to vMotion VMs off one ESXi host to another ESXi host (A to B),
  perform some maintenance on the host, then migrate the VMs back to host (B to A).
  Optional progress bar, unmounting of ISO files (sometimes causes issues),
  and maintenance mode for standard vs. Virtual SAN hosts.
 
 .NOTES   
  Author   : Justin Bennett   
  Date     : 2016-01-28
  Contact  : http://www.allthingstechie.net
  Revision : v1.1
  Changes  : v1.0 Original
    v1.1 Added Progress Bar and Notes
#>
#Connect-VIServer myvCenterServer.local

#Show Progress
$showProgress = $true
#maintenance host
$srchostesx = "esxA.domain.local"
#temp VM host
$dsthostesx = "esxB.domain.local"

#VMs to be migrated around
$VMs = Get-VM | ? { $_.VMHost -like $srchostesx }

#disconenct any ISOs as needed
# $VMs | % { get-vm -name $_.name | Get-CDDrive  } | ? { $_.IsoPath -like "*.iso" -OR $_.HostDevice -match "/" } | % { $_ | Set-CDDrive -NoMedia -Confirm:$false }

#move VMs off
if ($VMs.Count -gt 0) { $VMs | % {$i=0} { 
 $i++
 if($showProgress) { Write-Progress -Activity "vMotion Off All VMs: $($srchostesx) to $($dsthostesx)" -Status "$($i)/$($VMs.Count): VM:$($_.Name) - Attempting to vMotion to $($dsthostesx)..." -PercentComplete (($i/$VMs.Count)*100) }
 Move-VM $_ -Destination $dsthostesx -VMotionPriority High
 if($showProgress) { Write-Progress -Activity "vMotion Off All VMs: $($srchostesx) to $($dsthostesx)" -Status "$($i)/$($VMs.Count): VM:$($_.Name) - Pausing 10 seconds..." -PercentComplete (($i/$VMs.Count)*100) }
 sleep 10
 }
}

#enter host maintenance 
Get-VMHost -Name $srchostesx | Set-VMHost -State Maintenance
#enter host maintenance - VSAN
# Get-View -ViewType HostSystem -Filter @{"Name" = $srchostesx }|?{!$_.Runtime.InMaintenanceMode}|%{$_.EnterMaintenanceMode(0, $false, (new-object VMware.Vim.HostMaintenanceSpec -Property @{vsanMode=(new-object VMware.Vim.VsanHostDecommissionMode -Property @{objectAction=[VMware.Vim.VsanHostDecommissionModeObjectAction]::NoAction})}))}

#
# do my thang ¯\_(ツ)_/¯
#

#exit host maintenance
Get-VMHost -name $srchostesx | Set-VMHost -State Connected

#move VMs back
if ($VMs.Count -gt 0) { $VMs | % {$i=0} { 
 $i++
 if($showProgress) { Write-Progress -Activity "vMotion Off All VMs: $($dsthostesx) to $($srchostesx)" -Status "$($i)/$($VMs.Count): VM:$($_.Name) - Attempting to vMotion to $($srchostesx)..." -PercentComplete (($i/$VMs.Count)*100) }
 Move-VM $_ -Destination $srchostesx -VMotionPriority High
 if($showProgress) { Write-Progress -Activity "vMotion Off All VMs: $($dsthostesx) to $($srchostesx)" -Status "$($i)/$($VMs.Count): VM:$($_.Name) - Pausing 10 seconds..." -PercentComplete (($i/$VMs.Count)*100) }
 sleep 10
 }
}

Formatted for web with http://codeformatter.blogspot.com/ 

Script Running on Example VMs

Script Running on Example VMs