Get-NtpdcFromVMHost to obtain ntp time sync status from vmhost using powershell

Some update to recent post about vmhost time report (Get-VMHostTimeReport Reporting time from vmhost system). Like i have mentioned earlier i guess that report might be handy when troubleshooting vmhosts for issues with ntp time sync. It is not 100% accurate as i was comparing my local pc time to vmhost reported time. This time i have written a function that utilizes ntpdc command from vmhost so the infromation is more accurate. In addition it will return all other imporant properties such as reach,offset,delay,poll,local interface.

I tried to document the function as much as i could, just run ‘get-help Get-NtpdcFromVMHost -full’ to see how to run it and so on. Function also helps to explain the reach value, it can utilize ntpdc to query remote vmhost without having ssh service running on remote vmhost. Right now it can query only 1 remote vmhost using the -remotevmhost, but i will try to add an option so it will handle more in future. If no remotevmhost parameter is specified it will use plink to start ntpdc -p on the vmhost that was given in vmhost parameter.

So now we can check what is the offset between our vmhost system and the ntp server. My last vmhost time report could sometimes show big diff between local time and vmhost time ~ 1-5 sec. If there are still doubts about what is going on with time on vmhost we can run Get-NtpdcFromVMHost to obtain more detailed informations. If there is something not clear let me know in the comments.
Please be sure that if you are querying vmhost A wtih -vmhost param, then this vmhost has to have ssh service turned on. Also make sure not to run the script with the -batchmode switch when you haven’t connected before to that particular vmhost as there will be a question about storing key in cache. While running the function with -batchmode plink will not show any questions to the screen.
On the screenshot you can see that parameters plinklocation and vmhostcredential are set to variables. I have defined them before so i can use them later again:
$pl=’c:\plink.exe’
$vmhostcredential=get-credential

Plink can be obtained from here: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
Useful pages/articles regarding this topic:
http://doc.ntp.org/4.2.2/ntpq.html
http://www.eecis.udel.edu/~mills/ntp/html/decode.html#peer
http://tech.kulish.com/2007/10/30/ntp-ntpq-output-explained/
http://rickardnobel.se/tcpdump-uw-for-troubleshoot-esxi-networking/#comment-36778
http://www.oit.uci.edu/dcslib/ntp/ntp-4.0.99k/ntpdc.htm
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005092
http://www.meinbergglobal.com/english/info/ntp.htm
http://www.linuxjournal.com/article/6812
http://www.arubanetworks.com/techdocs/ArubaOS_61/ArubaOS_61_CLI/showntp.htm
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036357

Below few screenshots and the code:
1stquery

2ndquery

3rdquery

function Get-NtpdcFromVMHost{
<#
    .SYNOPSIS
        Returns properties from ntpdc command on vmhost using plink.exe
 
    .DESCRIPTION
        This function helps troubleshooting vmhost regarding issues with ntp. It runs ntpdc via plink
        Then it parses the output to properties. It also adds properties like ReachExplanation and 
        NtpserverStatus. ReachExplanation describes value of reach that was returned. For example
        Reach of 377 can be presented as 8 last tries of pooling time from ntp. Reach value is written
        as an octal number. We can convert it to binary value. As a result we will see only 0 and 1.
        0 represents failure in getting time from ntp server and
        1 represents success.
        Reach=377
        11111111
        Would mean that last 8 times everything was pooled with a success.
        Reach=376
        11111110
        Would mean that the last pooling was not successful. The last try is always on the right side.
        Reach=375
        11111101
        Would mean that last try was successful, but before that there was a failure, and so on.
        If last digit in reach is odd and ends on 1,3,5,7 we can at least say that the LAST pooling was
        successful. 
        NtpserverStatus is the fist character in ntpserver property. Also known as 'tally code'.
        More information about different tally codes can be found : http://doc.ntp.org/4.2.2/ntpq.html
        or here http://tech.kulish.com/2007/10/30/ntp-ntpq-output-explained/
        If you do not see a '*' that would mean that there is no active ntp server with which time can 
        be synced.
        Offset, delay and dispersion are given in milliseconds. Poll time is in seconds, describes
        pooling interval.
        This function requires vmhost to have started ssh service. It is also possible to query other
        vmhost using ntpq without using ssh. This function has -remotevmhost parameter. Using it it can
        run remote query using particular vmhost.
        plink.exe ----> vmhost (ntpdc -p another_vmhost) ----> result
        Plink.exe can be obtained from 
        http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

         
 
    .PARAMETER  vmhost
        Specify vmhost to which the plink will be connecting. Make sure that ssh service is started on
        target vmhost, and that there is no issue with connecting with plink to it directly. If you have
        not connected to that host before you might get an question to answer from plink like:
        'Store key in cache? (y/n)'. That requires our input. When running this function with -batchmode
        switch this information will be not shown and connection failure will occur. Please make sure first
        that there is no issue with connecting to vmhost directly using plink.
         
    .PARAMETER  remotevmhost
        If you would like to query anothervmhost using the vmhost defined within vmhost parameter you can 
        use it with the remotevmhost parameter. It is doable to make ntpdc -p vmhost1 vmhost2 vmhost 3
        but this is not implemeneted within this function, i will try to add it soon. For now it works 
        only with 1 vmhost.

    .PARAMETER  vmhostcredential
        In this version it expects a credential object. You can create it by using get-credential. 
        For example : root//password.1 those credentials have to be valid on your target vmhost.

     .PARAMETER  plinklocation
        You should have plink.exe in order to use this function. This parameter should point to plink.exe
        for example: -plinklocation 'c:\software\plink.exe' 

     .PARAMETER  batchmode
        Disables all interactive prompts from plink


    .EXAMPLE
        PS C:\> Get-NtpdcFromVMHost -vmhost vmhost01.local -batchmode -vmhostcredential (get-credential) 
        -plinklocation 'c:\plink.exe'
        vmhost           : vmhost01.local
        remotevmhost     :
        ntpserver        : *myntpserver01.com
        NtpserverStatus  : *
        local            : 1192.168.0.112
        stratum          : 2
        poll             : 128
        reach            : 377
        ReachExplanation : 11111111
        delay            : 0.00070
        offset           : 0.004569
        disp             : 0.06561

        vmhost           : vmhost01.local
        remotevmhost     :
        ntpserver        : =*myntpserver02.com
        NtpserverStatus  : =
        local            : 192.168.0.112
        stratum          : 2
        poll             : 128
        reach            : 377
        ReachExplanation : 11111111
        delay            : 0.00026
        offset           : 0.004706
        disp             : 0.07617
        Will establish connection with vmhost using plink and run ntpdc -p in order to get iformation
        about ntp sync status.
        It will return as many objects as there are ntp servers configured. In this example 1 ntp server
        is marked as active (*) and one as a backup (=)
        
         
    .EXAMPLE
        PS C:\> Get-NtpdcFromVMHost -vmhost vmhost01.local -batchmode -vmhostcredential (get-credential)
        -plinklocation 'c:\plink.exe' -remotevmhost vmhost02.local
        vmhost           : vmhost01.local
        remotevmhost     : vmhost02.local
        ntpserver        : *myntpserver01.com
        NtpserverStatus  : *
        local            : 192.168.0.148
        stratum          : 2
        poll             : 128
        reach            : 377
        ReachExplanation : 11111111
        delay            : 0.00067
        offset           : 0.004723
        disp             : 0.06688

        vmhost           : vmhost01.local
        remotevmhost     : vmhost02.local
        ntpserver        : =myntpserver02.com
        NtpserverStatus  : =
        local            : 192.168.0.148
        stratum          : 2
        poll             : 128
        reach            : 377
        ReachExplanation : 11111111
        delay            : 0.00026
        offset           : 0.004706
        disp             : 0.09250
        Will produce report for the given remotevmhost. We will use vmhost01 to query for the ntp status
        on vmhost02. In this case vmhost02 does not need ssh service to be enabled. Some network ports
        should be open though in order to make this query successful. Have in mind that ESX/i 4/0 by
        default will not be able to answer this query as their ntp service by default are configured with
        noquery option in /etc/ntp.conf(KB:1036357)

         
 
    .NOTES
        NAME:  Get-NtpdcFromVMHost
         
        AUTHOR: Grzegorz Kulikowski
         
        NOT WORKING ? #powercli @ irc.freenode.net 
         
        THANKS: BartekB
 
    .LINK
 
https://psvmware.wordpress.com
 
#>

    param(
        [Parameter(Mandatory=$true)]
        [string]$vmhost,
        [string]$remotevmhost,
        [Parameter(Mandatory=$true)]
        [System.Management.Automation.PSCredential]$vmhostcredential,
        [Parameter(Mandatory=$true)]
        [ValidateScript({Test-Path $_ -PathType 'Leaf'})]
        [string]$plinklocation,
        [switch]$batchmode
    )

    $vmhostuser=$vmhostcredential.GetNetworkCredential().UserName
    $vmhostpasswd=$vmhostcredential.GetNetworkCredential().password
    if($remotevmhost){
        $cmd=$plinklocation+' -pw '''+$vmhostpasswd+''' -l '+$vmhostuser+' '+$vmhost+' ntpdc -p '+$remotevmhost
        if($batchmode){
            $cmd=$plinklocation+' -batch -pw '''+$vmhostpasswd+''' -l '+$vmhostuser+' '+$vmhost+' ntpdc -p '+$remotevmhost
        }
    }else {
        $cmd=$plinklocation+' -pw '''+$vmhostpasswd+''' -l '+$vmhostuser+' '+$vmhost+' ntpdc -p'
        if($batchmode){
        $cmd=$plinklocation+' -batch -pw '''+$vmhostpasswd+''' -l '+$vmhostuser+' '+$vmhost+' ntpdc -p'
        }
    }


    $output=Invoke-Expression -Command $cmd
    if ( $output ) {
        $output[2..($output.count)]|%{
            $temp=$_
            $splitted=$temp -split '\s+'
            $reachexplanation=[Convert]::ToString([Convert]::ToInt32($splitted[4],8),2)   
            ""|select @{n='vmhost';e={$vmhost}}, @{n='remotevmhost';e={$remotevmhost}}, @{n='ntpserver';e={$splitted[0]}},@{n='NtpserverStatus';e={($splitted[0])[0]}}, @{n='local';e={$splitted[1]}}, @{n='stratum';e={$splitted[2]}}, @{n='poll';e={$splitted[3]}}, @{n='reach';e={$splitted[4]}}, @{n='ReachExplanation';e={$reachexplanation}}, @{n='delay';e={$splitted[5]}},@{n='offset';e={$splitted[6]}},@{n='disp';e={$splitted[7]}} 
        }
    } else {
        ""|select @{n='vmhost';e={$vmhost}}, @{n='remotevmhost';e={$remotevmhost}}, @{n='ntpserver';e={'plink had an issue'}},@{n='NtpserverStatus';e={'plink had an issue'}}, @{n='local';e={'plink had an issue'}}, @{n='stratum';e={'plink had an issue'}}, @{n='poll';e={'plink had an issue'}}, @{n='reach';e={'plink had an issue'}}, @{n='delay';e={'plink had an issue'}},@{n='offset';e={'plink had an issue'}},@{n='disp';e={'plink had an issue'}} 
    }
}

* fixed issue today : when running script from powershell v2 not all parameters were not defined properly.
* fixed issue with user variable when running scripts from powershell v2
Now the function should run within powershell v2 and v3 properly

Get-VMHostTimeReport Reporting time from vmhost system

I wanted to check if all of my vmhost system are keeping time with ntp. I wrote recently how to query time from vmhost system using esxcli. Today i want to show how to do this without esxcli, and also produce nice report and a summary which will help investigate issue with host time. I am comparing host time with local time o/s that is converted to UTC. If you are not sure whether your time is accurate then it might be a problem. Solution for this is to query remote utc time. In internet you can find sites that offer time web services that can be easily integrated to this script.
Have in mind that property DiffToUTC is in seconds.
I have tried to optimize this report as much as i can. It was possible to execute this report in 1 minute and 8 seconds. But it was not that readable and did not contain all of this information. On 100 vmhost+ it should fit in 1 – 2 minutes. Before i had approach with obtaining time using esxcli, but that was slower then this approach.
If you know how to query it faster, post a comment. Have in mind that this function is not perfect. I believe that there are better ways to show weather vmhost system is out of time or not. Anyway, this function helps a lot in diagnosing problems time on esx/esxi host systems.
It will state if ntpd is running or not, or for example if you have >100 host you can spot a pattern for ntp servers. Sometimes it is easy to overlook something, for example ntp servers:
172.16.x.y
72.16.x.y
Where you can easily see that you had a typo.
Sometimes you are just not aware that ntp service is running.
Or sometimes… 😉
Few screenshots from running this function:
From this screenshot where the report was stored in $myreport, we can see that we have some issues with host time, as there is a difference of 76 second compared to my local os time.
srh2
If you see DiffToUTC like 0.0xxxxx that would mean that you are ok. If you get readings > 1 seconds that could indicate something is wrong.
We can also do report for a single vmhost system.
srh1
We can do a full report and using the -summary switch, make function to return some description about the report.
trsum
Documentation for
HostDateTimeSystem -> Managed Object – HostDateTimeSystem
HostServiceSystem -> Managed Object – HostServiceSystem
I have added now option to check your local time which will be converted to UTC and compare it to UTC time taken from internet. Internet connection should be in place in order to use it. That was added if you suspect that your local time might not be correct and you would like to check it with other source. Please also read disclaimer,usage restrictions before using this time web service. So if you spot that your time differs to much from time taken from that ‘internet returned time’, it might indicate some issues, maybe the web service has issues, or our system.
Function below:

function Get-VMHostTimeReport {
<#
	.SYNOPSIS
		Gets time from VM Hosts and checks with local time(to utc).

	.DESCRIPTION
		This function might help investigating issues with vm host time. If it is running without any
		parameter it checks time for all hosts registered in virtual center to which user is currently
		connected. Using parameter SingleVMHost will produce report for single vm host system.
		VMHost should be the name of the host, string.
		Report returns colums : Name (vmhost system name), VMHostTime (Time from vmhost),
		UTCTime (this is the utc time from our local os), NTPServers ( if any are in the vmhost
		configuration), NTPServiceRunning (checks if the ntp service is running on the vmhost),
		DiffToUTC (that's the difference in seconds between time reported by vmhost and our os)
		By default it sorts report by from lowest to highest time difference reported.
		

	.PARAMETER  SingleVMHost
		Specify single vmhost name that is registered in VC. This should be a string.
		
	.PARAMETER  Summary
		Indicate if you would like to receive short summary about produced report.

	.PARAMETER  CheckTimeFromInternet
		Indicate if you would like to see in summary information about your local time and remote 
		time. Time from remote web service will be checked and comapred to your system utc time.

	.EXAMPLE
		PS C:\> Get-VMHostTimeReport
		Will produce report for all vmhosts that are registered within VirtualCenter to which user
		is currently connected. It is possible to close the report into a variable for example:
		$timereport=Get-VMHostTimeReport
		You can then export this report to csv if needed for example, or view it again :
		$timereport | format-table -autosize
		For viewing convenience
		
		
	.EXAMPLE
		PS C:\> Get-VMHostTimeReport -SingleVMHost 'myesxhost.local.lan'
		Will produce report for given vmhost that is registered within VirtualCenter to which user
		is currently connected.
		

	.EXAMPLE
		PS C:\> Get-VMHostTimeReport -Summary
		Will produce report . and show a small summary which might indicate if there is a problem
		with time sync on vmhost systems. Example of the summary below:
		UTC time from the current system :Min and Max times during reporting period
		Min: 9/3/2013 4:17:38 PM
		Max: 9/3/2013 4:19:20 PM
		While function was creating this report, first date that was returned by local os was the Min
		and the last date that was returned by local os was the Max value.
		VMHosts reported Times :Min and Max date / time while creating report
		Min: 9/3/2013 4:16:35 PM
		Max: 9/3/2013 4:19:20 PM
		Our vmhost systems reported their date/time. If this time span is too big, this might indicate 
		issues with vmhost ntp sync.
		Time Difference between VMHost and UTC from local os time Min, Max, Avg
		Min: 0.03075
		Max: 76.20985
		Avg: 1.36604993421053
		This is summary for comparing UTC vm host time to UTC time from local os. If you see 
		big Max value >1/2 sec that it might indicate that there is an issue with vmhost ntp time sync
		If a switch parameter CheckTimeFromInternet is present in the summary section there will
		be small report generated about your local time converted to UTC and time taken from
		http://www.earthtools.org
		You can then quickly see if there is an issue with your local time

	.NOTES
		NAME:  Get-VMHostTimeReport
		
		AUTHOR: Grzegorz Kulikowski
		
		NOT WORKING ? #powercli @ irc.freenode.net 
		
		THANKS: http://www.earthtools.org

	.LINK

https://psvmware.wordpress.com

#>

   param(
   [string]$SingleVMHost,
   [switch]$Summary,
   [switch]$CheckTimeFromInternet
   )
$TimeReport=@()
if ($SingleVMHost) { $VMHosts=Get-View -ViewType HostSystem -Filter @{'name'=$SingleVMHost} }
else{
$VMHosts=get-view -viewtype hostsystem -property name,ConfigManager.DateTimeSystem,ConfigManager.ServiceSystem -Filter @{'runtime.ConnectionState'='connected'}
}
Foreach($VMHost in $VMHosts){
$VMHostDateTimeSystem=get-view -id $VMHost.ConfigManager.DateTimeSystem
$VMHostServiceSystem=get-view -id $VMHost.ConfigManager.ServiceSystem
$VMHostTime=$VMHostDateTimeSystem.QueryDateTime()
$NtpServiceState=($VMHostServiceSystem.ServiceInfo.Service|Where-Object {$_.Key -eq 'ntpd'}).Running
$NtpServers=$VMHostDateTimeSystem.DateTimeInfo.NtpConfig.Server
$UTCTime=(Get-Date).ToUniversalTime() 
$TimeReport+=$vmhost| Select-Object -Property Name, @{n='VMHostTime';e={$VMHostTime}},@{n='UTCTime';e={$UTCTime}},@{n='NTPServers';e={$NtpServers}},@{n='NTPServiceRunning';e={$NtpServiceState}},@{n='DiffToUTC';e={[Math]::Round([math]::abs(($VMHostTime - $UTCTime).TotalSeconds),5)}}
}
if($Summary){
$SummaryUTCTime=$TimeReport|Measure-Object -Property UTCTime -Min -Max
$SummaryVMHostTime=$TimeReport|Measure-Object -Property VMHostTime -Min -Max
$SummaryDiffToUTC=$TimeReport|Measure-Object -Property DiffToUTC -Min -Max -Average
Write-Host "UTC time from the current system :Min and Max date/time while creating this report."
Write-Host "Min: $($SummaryUTCTime.Minimum.ToString())"
Write-Host "Max: $($SummaryUTCTime.Maximum.ToString())"
Write-Host "VMHosts reported Times :Min and Max date/time while creating this report"
Write-Host "Min: $($SummaryVMHostTime.Minimum.ToString())"
Write-Host "Max: $($SummaryVMHostTime.Maximum.ToString())"
Write-Host "Time Difference between VMHost and UTC from local os time Min, Max, Avg"
Write-Host "Min: $($SummaryDiffToUTC.Minimum.ToString())"
Write-Host "Max: $($SummaryDiffToUTC.Maximum.ToString())"
Write-Host "Avg: $($SummaryDiffToUTC.Average.ToString())"
if($CheckTimeFromInternet){
[datetime]$TimeFromInternet=(Invoke-RestMethod -Uri 'http://www.earthtools.org/timezone/52.35000/4.86660').timezone.utctime
$CurrentSystemTimeUTC=(Get-Date).ToUniversalTime()
Write-Host "We took UTC time from internet and compared it to your local time converted to UTC time"
$TimeDiff=[math]::abs(($CurrentSystemTimeUTC-$TimeFromInternet).TotalSeconds)
Write-Host "Reported local time converted to UTC $CurrentSystemTimeUTC"
Write-Host "Reported time taken from internet(http://www.earthtools.org/webservices.htm) $TimeFromInternet"
Write-Host "Difference: $TimeDiff seconds"
}
}
return $TimeReport | Sort-Object -Property DiffToUTC
}