Archives For VMware HA

Last night during a patching job in a customer I found the following error for several VMs when I put a host in maintenance mode and DRS tried to evacuate the virtual machines to the other nodes of the cluster.


Very strange since as far as I could see the virtual machines were running without errors, I was logged into some of them through SSH, and they also appeared as powered on in vSphere Client.

I decided to go to Tech Support Mode on the ESXi and check the virtual machine power state.


Everything looked exactly as it should be, no error logs, nothing. At this point I decided to restart the ESXi management agents.


And it worked, after a few seconds I was able to perform a successful vMotion and the host could be evacuated.


Managing VMware vSphere cluster configuration with the vSphere Client can be, sometimes, a tedious task. In this post I’ll show you how to use PowerCLI to manage your cluster’s configuration. I did not pretend to write a full detailed guide but just to show the most common tasks I perform at work.

The first thing  to do is to properly setup the basics that will allow us to interact with the cluster. First get your cluster basic configuration and store it in a variable, $cldl380 in the example.

[vSphere PowerCLI] C:\> $cldl380 = get-cluster cluster-dl380-01

Use the cmdlet to Get-View to get the .Net objects of the cluster and store the result in another variable.

[vSphere PowerCLI] C:\> $viewdl380 = get-view $cldl380.Id
[vSphere PowerCLI] C:\> $viewdl380

Configuration       : VMware.Vim.ClusterConfigInfo
Recommendation      : {}
DrsRecommendation   : {}
MigrationHistory    : {}
ActionHistory       : {}
DrsFault            : {}
ResourcePool        : ResourcePool-resgroup-36
Host                : {HostSystem-host-37, HostSystem-host-39}
Datastore           : {Datastore-datastore-11, Datastore-datastore-12, Datastore-datastore-13, Datastore-datastore-38...}
Network             : {Network-network-21, Network-network-60, Network-network-22, Network-network-16}
Summary             : VMware.Vim.ClusterComputeResourceSummary
EnvironmentBrowser  : EnvironmentBrowser-envbrowser-35
ConfigurationEx     : VMware.Vim.ClusterConfigInfoEx
Parent              : Folder-group-h4
CustomValue         : {}
OverallStatus       : red
ConfigStatus        : red
ConfigIssue         : {0, 0}
EffectiveRole       : {-1}
Permission          : {}
Name                : cluster-dl380-01
DisabledMethod      : {}
RecentTask          : {}
DeclaredAlarmState  : {alarm-1.domain-c35, alarm-11.domain-c35, alarm-12.domain-c35, alarm-13.domain-c35...}
TriggeredAlarmState : {,}
AlarmActionsEnabled : True
Tag                 : {}
Value               : {}
AvailableField      : {}
MoRef               : ClusterComputeResource-domain-c35
Client              : VMware.Vim.VimClient

[vSphere PowerCLI] C:\>

This variable is the base we are going to use to get the cluster configuration, now we are going to use it.

– Get the cluster resources summary:

[vSphere PowerCLI] C:\> $viewdl380.Summary

CurrentFailoverLevel : 0
AdmissionControlInfo : VMware.Vim.ClusterFailoverLevelAdmissionControlInfo
NumVmotions          : 5
TargetBalance        : 200
CurrentBalance       : 16
CurrentEVCModeKey    :
TotalCpu             : 63984
TotalMemory          : 77287268352
NumCpuCores          : 24
NumCpuThreads        : 48
EffectiveCpu         : 56784
EffectiveMemory      : 62981
NumHosts             : 2
NumEffectiveHosts    : 2
OverallStatus        : red
DynamicType          :
DynamicProperty      :

[vSphere PowerCLI] C:\>

– Get VMware HA configuration:

[vSphere PowerCLI] C:\> $viewdl380.Configuration

DasConfig       : VMware.Vim.ClusterDasConfigInfo
DasVmConfig     :
DrsConfig       : VMware.Vim.ClusterDrsConfigInfo
DrsVmConfig     :
Rule            : {sql-cl01, hpsim}
DynamicType     :
DynamicProperty : 

[vSphere PowerCLI] C:\> $viewdl380.Configuration.DasConfig

Enabled                 : True
VmMonitoring            : vmMonitoringDisabled
HostMonitoring          : enabled
FailoverLevel           : 1
AdmissionControlPolicy  : VMware.Vim.ClusterFailoverLevelAdmissionControlPolicy
AdmissionControlEnabled : True
DefaultVmSettings       : VMware.Vim.ClusterDasVmSettings
Option                  : {das.isolationaddress1, das.isolationaddress2, das.usedefaultisolationaddress}
DynamicType             :
DynamicProperty         : 

[vSphere PowerCLI] C:\>

– Get cluster advanced options:

[vSphere PowerCLI] C:\> $viewdl380.Configuration.DasConfig.Option

Key                                   Value                                 DynamicType                          DynamicProperty                     
---                                   -----                                 -----------                          ---------------                     
das.usedefaultisolationaddress        false                                                                                                          

[vSphere PowerCLI] C:\>

– Get DRS basic configuration

[vSphere PowerCLI] C:\> $viewdl380.Configuration.DrsConfig

Enabled                   : True
EnableVmBehaviorOverrides : True
DefaultVmBehavior         : partiallyAutomated
VmotionRate               : 3
Option                    : {ForceAffinePoweron}
DynamicType               :
DynamicProperty           :

[vSphere PowerCLI] C:\> $viewdl380.Configuration.DrsConfig.Option

Key                                   Value                                 DynamicType                          DynamicProperty
---                                   -----                                 -----------                          ---------------
ForceAffinePoweron                    1

[vSphere PowerCLI] C:\>

– Get Virtual Port Groups of the cluster

[vSphere PowerCLI] C:\> Get-View $viewdl380.Network | Select-Object Name


[vSphere PowerCLI] C:\>

– Get the Datastores configured in the cluster

[vSphere PowerCLI] C:\> Get-View $viewdl380.Datastore | Select-Object Name


[vSphere PowerCLI] C:\>

Finally to ease things at work I created a bunch of scripts that implement some of the above tasks, here it is an example. The syntax I used for all of them is [script-name] <cluster-name>.

# Get-ClusterAdvancedOption.ps1
# PowerCLI script to get VMware HA advanced options of a given cluster
# Juan Manuel Rey - juanmanuel (dot) reyportal (at) gmail (dot) com
# Syntax: Get-ClusterAdvancedOption.ps1 &lt;cluster-name&gt;


$cluster = Get-Cluster $clustername
$viewcluster = Get-View $cluster.Id


From this point I’ll leave to you to investigate all the possibilities, as always if you know a better way to do it, have any question or want to say anything about the post please comment. I’m also preparing a new post with all my small scripts and hope to publish them next week :-)


Anyone with some experience and knowledge about VMware HA knows how to perform a Reconfigure for HA operation in a host from the vSphere client and I’m no exception to that rule. However I never did with PowerCLI.

I created a new cluster in my homelab with a problem in one of the hosts, I fixed the problem, put my mind to work and after an hour or so digging through PowerCLI and the vSphere API Reference Documentation I came up with the following easy way to do it.

First we are going to create a variable that contained the configuration of the ESXi I wanted to reconfigure.

[vSphere PowerCLI] % $vmhost = Get-VMHost esxi06.vjlab.local
[vSphere PowerCLI] %
[vSphere PowerCLI] % $vmhost | Format-List

State                 : Connected
ConnectionState       : Connected
PowerState            : PoweredOn
VMSwapfileDatastoreId :
VMSwapfilePolicy      : Inherit
ParentId              : ClusterComputeResource-domain-c121
IsStandalone          : False
Manufacturer          : VMware, Inc.
Model                 : VMware Virtual Platform
NumCpu                : 2
CpuTotalMhz           : 5670
CpuUsageMhz           : 869
MemoryTotalMB         : 2299
MemoryUsageMB         : 868
ProcessorType         : Intel(R) Core(TM)2 Quad CPU    Q9550  @ 2.83GHz
HyperthreadingActive  : False
TimeZone              : UTC
Version               : 4.1.0
Build                 : 260247
Parent                : cluster3
VMSwapfileDatastore   :
StorageInfo           : HostStorageSystem-storageSystem-143
NetworkInfo           : esxi06:vjlab.local
DiagnosticPartition   : mpx.vmhba1:C0:T0:L0
FirewallDefaultPolicy :
ApiVersion            : 4.1
CustomFields          : {[com.hp.proliant, ]}
ExtensionData         : VMware.Vim.HostSystem
Id                    : HostSystem-host-143
Name                  : esxi06.vjlab.local
Uid                   : /VIServer=administrator@vcenter1.vjlab.local:443/VMHost=HostSystem-host-143/

[vSphere PowerCLI] %

Next with the cmdlet Get-View I retrieved the .NET objects of the host ID and stored them in another variable.

[vSphere PowerCLI] % Get-View $vmhost.Id

Runtime             : VMware.Vim.HostRuntimeInfo
Summary             : VMware.Vim.HostListSummary
Hardware            : VMware.Vim.HostHardwareInfo
Capability          : VMware.Vim.HostCapability
ConfigManager       : VMware.Vim.HostConfigManager
Config              : VMware.Vim.HostConfigInfo
Vm                  : {}
Datastore           : {Datastore-datastore-144}
Network             : {Network-network-11}
DatastoreBrowser    : HostDatastoreBrowser-datastoreBrowser-host-143
SystemResources     : VMware.Vim.HostSystemResourceInfo
Parent              : ClusterComputeResource-domain-c121
CustomValue         : {}
OverallStatus       : red
ConfigStatus        : red
ConfigIssue         : {0}
EffectiveRole       : {-1}
Permission          : {}
Name                : esxi06.vjlab.local
DisabledMethod      : {ExitMaintenanceMode_Task, PowerUpHostFromStandBy_Task, ReconnectHost_Task}
RecentTask          : {}
DeclaredAlarmState  : {,,,}
TriggeredAlarmState : {}
AlarmActionsEnabled : True
Tag                 : {}
Value               : {}
AvailableField      : {com.hp.proliant}
MoRef               : HostSystem-host-143
Client              : VMware.Vim.VimClient

[vSphere PowerCLI] % $esxha = Get-View $vmhost.Id

Now through the $esxha variable I invoked the method ReconfigureHostForDAS to reconfigure the ESXi, this method is part of the HostSystem object and its description can be found here in the vSphere API reference.

As it can be seen in the above screenshot, the task is displayed in the vSphere client. You can also monitor the operation with the Get-Task cmdlet.

Finally I created the below script to simplify things in the future :-)

# Reconfigure-VMHostHA.ps1
# PowerCLI script to reconfigure for VMware HA a VM Host
# Juan Manuel Rey - juanmanuel (dot) reyportal (at) gmail (dot) com


$vmhost = Get-VMHost $esx
$esxha = Get-View $vmhost.Id