Find CPU hogging vms using PowerCLI

Hello,
this time i would like to show how can we check if we have any vms that are hogging cpu for too long. I bet you are already preventing this using VC alarms. So lets build a simple script that helps us out in getting list of vms that have cpu usage at near 100% for some time.
What we have to do here is:
1) figure out where alarms are triggered , for which vms
2) figure out how to get the root folder of alarms
3) figure out how to get definitions of alarms
4) figure out how to select vms that are using cpu too much for some predefined period of time
5) figure out how to get the cpu stats for vms from point number 4

Let’s go:
Lets first get the service instance object:

$si=get-view serviceinstance

I have decided to put more explanations in this post.
So, serviceinstance, what is that ? I think that at best it is described in the documentation. “The ServiceInstance managed object is the singleton root object of the inventory on both vCenter servers and servers running on standalone host agents. The server creates the ServiceInstance automatically, and also automatically creates the various manager entities that provide services in the virtual environment. Some examples of manager entities are LicenseManager, PerformanceManager, and ViewManager. You can access the manager entities through the content property.”
Make sure to use that link to read service instance description, it will really help to understand how it works.
We will be using alarm manager in order to achieve our goal, let’s have a look how we get there.
sicontent
And ? What to do now ? We see Alarm-AlarmManager. If we would like to return it, we will just get something with Type,Value properties. So how this can help us?
Let’s see what is this by checking the type, and then use get-help on get-view parameter called ID. This should give us some hint what to do next. So This magic Alarm-AlarmManager is actually a moref/ManagedObjectReference , having that we can get its view using get-view.
amn

Then define from when we should build the statistics for vm cpu hogging

$days=-7

Let’s get the VC root folder , and then find out which alarms are triggered there on.

$rootfolderviewalarms=(get-view -id $si.Content.RootFolder).TriggeredAlarmState

Then let’s get the alaram manager object:

$am=get-view -id $si.Content.AlarmManager

Let’s get defined alarms ids in our root vc folder.
We can do this for example using GetAlarm method from our AlarmManager. But..but..but.. how ?! Ok, lets take few steps back. What if we do not know how to do this, or if there even is a method that can do this ? First thing we can do is to inspect the alarm manager, to see what he can do for us.
On the screenshot below, you can see our alarm manager object $am. We want to see what it can do for us, we use get-member to get list of his methods. From there we can see that he has a method called GetAlarm. In order to check what it does we can use the documentation.
From the output of this method, we know that it will be returning morefs(defined alarm ids on particular entity), and in order to use it we need to give it a entity moref, a place where we look for alarms. Now the documentation for GetAlarm method also states that if the entity will not be set, then it will return all visible alarms. If you would like to use it like that, you would have to run it with $null as argument.
amm
In this example we will get only alarms defined on the root of our Virtual Center server.

$alarmids=$am.GetAlarm($si.Content.rootfolder)

Once again, what is it that was returned ? Morefs ! correct. As such they don’t hold too much information. We can get that information though. Using what ? Get-View , connected to that id. Screenshot below shows how to get from ids to actual alarm definition objects with information.
Lets take information about those alarms.

$alarmdefinitions=get-view -id $alarmids

alarmsa1
Let’s find the alarm id that describes the alarm for virtual machine cpu usage.
Now this part is bit tricky. I am making assumption here that we have only 1 alarm defined for vm cpu usage, and that it was defined in the root virtual center folder. This script will not work if you have defined more alarms than 1 for vm cpu usage because i am searching only for alarm system name ‘alarm.VmCPUUsageAlarm’, and i am not filtering by its name. So we are looking at $alarmdefinitions array that holds definitions of alarms with .info object that has a systemname property. We filter it so we can get in result the VM cpu usage alarm, and selecting its alarm moref.

$vmcpuusagealarmid=($alarmdefinitions|Where-Object {$_.Info.Systemname -eq 'alarm.VmCPUUsageAlarm' }).info.alarm
$vmcpuusagealarmid
Type                                              Value
----                                              -----
Alarm                                             alarm-6

So the alarm id that is about vm cpu usage is Alarm-alarm-6.
Let’s get now ids of virtual machines that have currently triggered alarm that we have found previously. Our root VC container had a property called TriggeredAlarmState that holds morefs of entities and alarm ids triggered on those entities. We will now filter them them to get only those that have vm cpu usage alarm triggered.
alarmsa2

$vmswithcpualarms=$rootfolderviewalarms|?{$_.Alarm -eq $vmcpuusagealarmid}

So now we have in $vmswithcpualarms only those triggered alarm states that match our vms with alarm of vm cpu usage.
Let’s change ids to vm view objects, so that we can grab vm names. So far we have only in $vmsswithcpualarms properties called Entity which is only a moref.

$cpuhoggingVMs=get-view -property name -id ($vmswithcpualarms | %{$_.Entity})

Now let’s build statistics from a ‘cpu.usage.average’ metric. We will use 2h intervals and we would like to get data from -7 days until now as stated in $days variable. PowerCLI gives us get-stat cmdlet that we will use. It accepts entity names. I am using $cpuhoggingVMS|%{$_.name} in order to return only names directly. Same as if you would type : -Entity ‘vm1′,’vm2′,’vm3′,’vm4’

$result=Get-Stat -Entity ($cpuhoggingVMS|%{$_.name}) -Start (Get-Date).AddDays($days) -Finish (get-date) -Stat 'cpu.usage.average' -IntervalMins 120

We have now our data stored in $result variable, we have a lot of data there, for each virtual machine statistics about its cpu usage.
Ok what if you say that you can not distinguish from which vm is that statistic data ? I say : “We need to go deeper” 🙂
So gm or get-member on the result entry shows that there are more properties than only those which are displayed.

alarmsa3

And the last line! We will group statistics for the vms by their name, and then for each of them we will measure their average cpu usage during that period of time.

$reportVMcpu=$result | select value,Entity | Group-Object -Property entity | % {$temp=$_; $temp.group | Measure-Object -Property value -average | select @{n='CPU % Average usage';e={[math]::round($_.average,3)}}, @{n='entity';e={$temp.name} }} | Sort-Object -Propert 'CPU % Average usage' -Descending

Now, if we will display our report, we will get a summary of vms and its corresponding average cpu usage through last 7 days in our example.
vmcpuhoggers
We can now tell that some vms here have an average of cpu usage for lat 7 days at 99%, that would indicate that something went bad inside this vm, and we need to investigate it. We do not like vms that hog cpus without any reason for too long 😉
Why would we want this report ?
Ok, i bet you are using alarms for VM cpu usage, and the alarm kicks in after 5..15..30.. minutes for example. You might assume that something went wrong inside the vm, but there are vms that for example are working really hard only during some specific time window. For example systems that are doing calculations at end of the month, or that use cpu only for few days in week/month as per design/function. Each time that alarms is triggered you would have to go to vm performance, and check if this is abnormal situation/call the vm owner/ or look for any pattern in its cpu usage. If you will see that this vm behaves as expected because it is normal to consume that amount of cpu only on Mondays, you would ignore that alarm and just wait to see if alarm gets cleared as previously.

I hope that this post will help you start using alarm manager and other managers, as well as understand morefs and using get-view.

Below is the code without any comments.

$si=get-view serviceinstance
$days=-7
$rootfolderviewalarms=(get-view -id $si.Content.RootFolder).TriggeredAlarmState
$am=get-view -id $si.Content.AlarmManager
$alarmids=$am.GetAlarm($si.Content.rootfolder)
$alarmdefinitions=get-view -id $alarmids
$vmcpuusagealarmid=($alarmdefinitions|Where-Object {$_.Info.Systemname -eq 'alarm.VmCPUUsageAlarm' }).info.alarm
$vmswithcpualarms=$rootfolderviewalarms|?{$_.Alarm -eq $vmcpuusagealarmid}
$cpuhoggingVMs=get-view -property name -id ($vmswithcpualarms | %{$_.Entity})
$result=Get-Stat -Entity ($cpuhoggingVMS|%{$_.name}) -Start (Get-Date).AddDays($days) -Finish (get-date) -Stat 'cpu.usage.average' -IntervalMins 120
$reportVMcpu=$result | select value,Entity | Group-Object -Property entity | % {$temp=$_; $temp.group | Measure-Object -Property value -average | select @{n='CPU % Average usage';e={[math]::round($_.average,3)}}, @{n='entity';e={$temp.name} }} | Sort-Object -Propert 'CPU % Average usage' -Descending

Enjoy!

Advertisements

How to copy folders structure between two virtual center servers ?

Hello !
It is some time since i posted last time. There was quite a lot happening in my life lately so that’s why 😉 I’ve spent huge amount of time on my first lecture that was given in Poland at PLVMUG,
plvmug.q1.2013-26-700x464 (1)
plvmug.q1.2013-24-700x464
after that i was back again in Poland to see my family on Easter. But anyway… i am lately having more free time, so i can share some of my thoughts with you again 😉
This time i want to show how we can copy folder structures between different virtual center servers. For example if you have to make a migration between virtual center servers, there is a lot to think of while doing this. You can’t forget about your annotations, folders(structure), location of virtual machines.
If you are migrating 20-50 vms, that’s not a big deal to move that kind of volume between VC, but what if you need to migrate 1000 or 2000 or 10000 vms ? And probably you don’t keep virtual machines inside ‘Discovered virtual machines’ folder, but you already have some structure.
This example was built based on example where cluster vms are attached to 1 folder structure. So if you have a virtual center and datacenter within, i assume here that structure looks like this:
‘blue\vm folders\’
‘\Datacenter X\ Cluster_Omega\Somefolder_with_Structure’

This function is able to move not only folder structures between VC but it also generates list of virtual machines and their folder ids, to which they should be moved. So if all my vms from cluster Omega resides somewhere in folder ‘Somefolder_with_Structure\x\y\z\’ you can grab the output of script to variable :
$mvlist=Copy-VCFolderStructure ….
While copying the folder structure this funtion will also return an array of vms with Name and Folder property, where the Folder property is actually an ID of a folder on the new virtual center server in which this vm should be placed.

function Copy-VCFolderStructure {
<#
	.SYNOPSIS
		Copy-VCFolderStructure copies folder and its structure from one VC to another..

	.DESCRIPTION
		Copy-VCFolderStructure can be handy when doing migrations of clusters/hosts between
		Virtual Center servers. It takes folder structure from 'old' VC and it recreates it on 'new'
		VC. While doing this it will also output virtualmachine name and folderid. Why would you
		want to have it ? Let's say that you have a cluster on old virtual center server
		oldvc.local.lab
		DC1\Cluster1\folder1
		DC1\Cluster1\folderN\subfolderN
		Copy-VCFolderStructure will copy entire folder structure to 'new' VC, and while doing this
		it will output to screen VMs that resides in those structures. VM name that will be shown on
		screen will show also folderid, this ID is the folderid on new VC.  After you have migrated
		your hosts from old cluster in old VC to new cluster in new VC, and folder structure is there,
		you can use move-vm cmdlet with -Location parameter. As location you would have give the
		folder object that corresponds to vm that is being moved. Property Name is the name of VM
		that was discovered in that folder and Folder is the folderid in which the vm should be moved
		into. This folderid has to first changed to folder, for example :
		$folderobj=get-view -id $folder|Get-VIObjectByVIView
		We can then use $folderobj as parameter to move-vm Location parameter

	.PARAMETER  OldFolder
		This should be the extensiondata of folder that you want to copy to new VC.
		$folderToRecreate=Get-Folder -Server oldVC.lab.local -Name teststruct
		Have in mind that this should be an single folder and not an array.

	.PARAMETER  ParentOfNewFolder
		When invoking the function this is the root folder where you want to attach the copied folder.
		Let's say you are copying folder from \DatacenterA\FolderX\myfolder
		If you will have the same structure on the new VC, you would have set ParentOfNew folder
		to FolderX. Still it's not a problem if you have a new structure on new VC. Let's say that on
		new VC you have folder: \DatacenterZ\NewStructure\FolderZ and you want to copy entire
		'myfolder' beneath the FolderZ. In that case, first create a variable that has desired folder
		$anchor=get-folder 'FolderZ' -Server newVC
		Make sure that $anchor variable will have only 1 element.

	.PARAMETER  NewVC
		This parameter describes virtual center to which we are copying the folder structure.
		Copy-VCFolderStructure works only when you are connected to both old and new vc at the
		same time. You need to set your configuration of PowerCLI to handle multiple connections.
		Set-PowerCLIConfiguration -DefaultVIServerMode 'Multiple'
		You can check if you are connected to both servers using $global:DefaultVIServers variable

	.PARAMETER  OldVC
		This parameter describes virtual center from which we are copying the folder structure.
		Copy-VCFolderStructure works only when you are connected to both old and new vc at the
		same time. You need to set your configuration of PowerCLI to handle multiple connections.
		Set-PowerCLIConfiguration -DefaultVIServerMode 'Multiple'
		You can check if you are connected to both servers using $global:DefaultVIServers variable

	.EXAMPLE
		PS C:\> Set-PowerCLIConfiguration -DefaultVIServerMode 'multiple'
		PS C:\> $DefaultVIServers
		Ensure that you are connected to both VC servers
		Establish variables:
		This will be the folder that we will be copying from old VC
		$folderToRecreate=Get-Folder -Server $OldVC -Name 'teststruct'
		This will be the folder to which we will be copying the folder structure
		$anchor=get-folder 'IWantToPutMyStructureHere' -Server $NewVC
		$OldVC='myoldvc.lab.local'
		$NewVC='mynewvc.lab.local'
		Copy-VCFolderStructure -OldFolder $folderToRecreate.extensiondata -NewVC $NewVC
		-OldVC $OldVC -ParentOfNewFolder $anchor
		$OldFolder expects to get extensiondata object from the folder, if you will not provide it, function will
		block it.

	.EXAMPLE
		If you are planning to move vms after hosts/vm/folders were migrated to new VC, you might use it in this way.
		By default Copy-VCFolderStructure will output also vms and their folder ids in which they should reside on new
		VC. You can grab them like this:
		$vmlist=Copy-VCFolderStructure -OldFolder $folderToRecreate.extensiondata -NewVC $NewVC
		-OldVC $OldVC -ParentOfNewFolder $anchor
		You can now export $vmlist to csv
		$vmlist |export-csv -Path 'c:\migratedvms.csv' -NoTypeInformation
		And once all virtual machines are in new virtual center, you can import this list and do move-vm operation on those
		vms. Each vm has name and folder properties. Folder is a folderid value, which has to be converted to Folder object.
		move-vm -vm $vmlist[0].name -Location (get-view -id $vmlist[0].folder -Server $newVC|get-viobjectbyviview)
		-Server $newVC
		This would move vm that was residing in previously on old VC in migrated folder to its equivalent on new VC.

	.NOTES
		NAME:  Copy-VCFolderStructure

		AUTHOR: Grzegorz Kulikowski

		NOT WORKING ? #powercli @ irc.freenode.net 

		THANKS: Huge thanks go to Robert van den Nieuwendijk for helping me out with the recursion in this function.

	.LINK
		https://psvmware.wordpress.com

#>

   param(
   [parameter(Mandatory = $true)]
   [ValidateNotNullOrEmpty()]
   [VMware.Vim.Folder]$OldFolder,
   [parameter(Mandatory = $true)]
   [ValidateNotNullOrEmpty()]
   [VMware.VimAutomation.ViCore.Impl.V1.Inventory.FolderImpl]$ParentOfNewFolder,
   [parameter(Mandatory = $true)]
   [ValidateNotNullOrEmpty()]
   [string]$NewVC,
   [parameter(Mandatory = $true)]
   [ValidateNotNullOrEmpty()]
   [string]$OldVC
   )
  $NewFolder = New-Folder -Location $ParentOfNewFolder -Name $OldFolder.Name -Server $NewVC
  Get-VM -NoRecursion -Location ($OldFolder|Get-VIObjectByVIView) -Server $OldVC|Select-Object Name, @{N='Folder';E={$NewFolder.id}}
  foreach ($childfolder in $OldFolder.ChildEntity|Where-Object {$_.type -eq 'Folder'})
                  {
                   Copy-VCFolderStructure -OldFolder (Get-View -Id $ChildFolder -Server $OldVC) -ParentOfNewFolder $NewFolder -NewVC $NewVC -OldVC $OldVC
                  }
}

My example:
We have our VcOld: at : vc1.lab.local My cluster is CL_Holandia
$oldvc=vc.lab.local
$newvc=vc2.lab.loacl

main_dc_holandia
Our NewVC before the movement looks like this:
As you can see in dc_new there is a folder that should keep my folder structure called: ‘Some_Holder’

before_newvc

On NewVC we have now structure like this:
We are connected to 2 virtual center servers : vc.lab.local and vc2.lab.local
Our cluster folder is named CL_Holandia where we keep all vms that are inside the cluster.
We are defining folder to be copied from old_VC:
$folderToRecreate=get-folder -Server $oldVC -Name ‘CL_Holandia’
we define to which folder this structure should go:
$anchor=get-folder’CL_Holandia’ -Server $newVC
Where:
$oldvc=’vc.lab.local’
$newvc=’vc2.lab.local’
We store VMs in variable $Vmlist
$vmlist=Copy-VCFolderStructure -OldFolder $folderToRecreate.extensiondata -ParentOfNewFolder $Anchor -NewVC $NewVC
-OldVC $OldVC

copyFolderS

newVCfS
When we will add the cluster to new VC we will have structure probably like this. There will be additional ‘Discovered virtual machine’ folder, where all vms will appear and the rest of our copied structure. That means that we should now move vms to their proper folders.
clusteradded

After the last movement we will see proper structure:
It’s time now to move our vms from cluster from old VC, to folder of NEW cluster in NEW NC. So we will use now
$Vmlist variable do to this:
foreach($vm in $vmlist) {move-vm -vm $vm.name -destination (get-view -id $vm.folder|get-viobjectbyviview)}
This should do it. We should see now our structure with vms and folder in proper place.
afterconfandmovement
I do realize that i write this post at 1 am, but this is the only time i guess now that i have it free… So i know there might be some issues/bugs with this code. Please post a comment whenever you see some error. I will try to fix it as soon as possible. Please note that not all of you should have the same setup as me. This solution in regards of moving vms on new VC to proper folders will work only if you have similar folder structure(1 CL x 1 main_folder). I will try to rewrite this function entirely in order to make it as compatible with all structures as it can be.
At the end i would like to thank to Robert van den Nieuwendijk for his time and effort in writing this function. Without him it would take ages for me to write this recursion…
I hope that this function will help some of you with similar setup when you need to do migrations.

*images from PLVMUG session were taken from:
Blog.Inleo.Pl

When was vm created, how vm was created and who has created this vm ?

I wanted to know where it is possible, who has created virtual machine. When the virtual machine was created and how it was created.
The idea is very simple. As an input i will take a virtual machine, and the output should be an object with few properties. VM can be created in several ways, right ? So i need to search for different events. When vm is being created/deployed it will create an event for this fact. Those events are as follows:

VmBeingCreatedEvent – Vm is created manually. Right-click New-VM. When we do VM in this way, a second event joins VmRegisteredEvent. So To check for vms there were manually created, we have to browse for events and check if 2 types of events have occured.
VmBeingDeployedEvent – When vm was deployed from template it will make this type of event
VmRegisteredEvent – When vm was registered, for example, open datastore browser, right click on vmx – register.
VmClonedEvent – When vm was cloned.
Continue reading

Issues with vmmonitoring in HA clusters while using vsphere client 5.0.0 build 455964, vm monitoring with vmware tools down will not restart failed vm

I wanted to share with everybody this ‘news’.

While trying to create vm monitoring within HA cluster using vm monitoring only i went into some issues. VM monitoring was not working. I thought there is not that much to configure to have this working right ? Open cluster settings, go to vm monitoring section

Continue reading

Change vnic type from / to e1000, Flexible, Vmxnet, EnhancedVmxnet, and Vmxnet3, and Unknown with Set-NetworkAdapter. UPDATED!

What if you noticed that your vms are running on wrong network card type ? Let’s say you want e1000 instead of vmxnet3 for some reason.There is a great cmdlet set-networkadapter. This will help us out in that kind of situation(Thx Rafael!).
Right, so what set-networkadapter can do for us ? (by this time you should read the get-help set-networkadapter -full). So now, like you already know 😉 it can do a lot for us! But what i want from it right now, is to help me with changing the adapter type. I want to change it from vmxnet 3 to e1000, would be cool if i will not need to shutdown vm too 😉 Unfortunately it will not be so cool at all !!! VM has to be powered off to make this change properly. Please do not do this while vm is up and running.
Continue reading

Get-view, list viewtypes, filter usage, Get-VIObjectbyVIView, and get-vm in powercli

From time to time i check my blog to see how people are getting to my blog 😉 Today i thought that if so many of you are trying to find this information although for some reason it is not in that post to which you are being redirected, i will create a new post about it. So that you will not be disappointed that the information is not there.
So few things about get-view. I think that VMware offers a GREAT documentation so why not take their explanation 😉

The vSphere PowerCLI list of cmdlets includes the Get-View and Get-VIObjectByVIView cmdlets, which enable access to vSphere PowerCLI views from .NET.

Using the vSphere PowerCLI views cmdlets for low-level VMware vSphere management requires some knowledge of both PowerShell scripting and the VMware vSphere APIs.

Continue reading

Change ESXi server license to evaluation while connected to esxi directly using powercli

In order to change license to the evaluation one when the host is still not connected to Virtual Center we can use this approach.

connect-viserver esxi1.local
$lm = Get-View -Id 'LicenseManager-ha-license-manager'
$lm.UpdateLicense("00000-00000-00000-00000-00000", $null)

This will change esxi licene to the evaluation one on Esxi 5.