vSphere 6.0 and 6.5 have an interesting bug in the vSphere Web Client: a DirectPath I/O option is enabled by default for a new virtual machine with VMXNET3 network adapter provisioned.
A New Virtual Machine dialogue looks good.
However, when I open the VM settings, surprise surprise, it is quite different.
I have done my research and found the DirectPath I/O option was enabled regardless the VM hardware version (checked all versions starting from 8 to 13). VMware explains this behaviour in a KB # 2145889 with the recommendation to disable this parameter.
My next thought was to find all VMs that had the same problem. Thanks to VMware and its vast community, I was able to find some useful examples (here, here, and here) for a PowerCLI script below.
# File name: Get-DirectPathIOStatus.ps1 | |
# Description: VMXNET3 - Network DirectPath I/O Status | |
# VMware KB: https://kb.vmware.com/kb/2145889 | |
# | |
# 06/07/2018 - Version 1.1 | |
# - Added the PowerCLI version check | |
# - Improved the host connection block | |
# - Added an export to CSV | |
# 26/04/2017 - Version 1.0 | |
# - Initial Release | |
# | |
# Author: Roman Dronov (c) | |
# Initial variables | |
$outputCollection = @() | |
$newModule = "VMware.PowerCLI" | |
$oldModule = "VMware.VimAutomation.Core" | |
# Check the PowerCLI module availability | |
Clear-Host | |
Write-Host "`n Checking if the VMware PowerCLI modules are loaded...`r" | |
if ($(Get-InstalledModule | ? {$_.Name -like $newModule}) -eq $null) { | |
if ($(Get-Module | ? {$_.Name -like $oldModule}) -eq $null -and $(Get-Module -Name $oldModule -ListAvailable -ErrorAction Ignore) -eq $null) { | |
Write-Host "`n You need to install the VMware PowerCLI module to run this script. Exiting...`r" -ForegroundColor Yellow | |
exit | |
} | |
else { | |
Import-Module -Name $oldModule -ErrorAction Ignore -WarningAction Ignore | |
$moduleVersion = $(Get-Module -Name $oldModule).Version | |
Write-Host "`n The VMware PowerCLI module version $moduleVersion is loaded successfuly.`r" -ForegroundColor Green | |
} | |
} | |
else { | |
$moduleVersion = $(Get-InstalledModule -Name $newModule).Version | |
Write-Host "`n The VMware PowerCLI module version $moduleVersion is detected.`r" -ForegroundColor Green | |
} | |
# Set the VIServer connection mode to Multiple | |
$connectionMode = $(Get-PowerCLIConfiguration -Scope Session).DefaultVIServerMode | |
if ($connectionMode -like 'Single') { | |
Set-PowerCLIConfiguration -Scope Session -DefaultVIServerMode Multiple -Confirm:$false | Out-Null | |
} | |
# Connect to the host(s) | |
function Enter-Credentials { | |
$defaultUser = $env:USERDOMAIN + "\" + $env:USERNAME | |
if ($(Read-Host "`n Username [default: $($defaultUser)]") -eq '') { | |
$viUser = $defaultUser | |
} | |
else { | |
$viUser | |
} | |
$viPassword = Read-Host " Password" -AsSecureString | |
$script:viCredential = New-Object System.Management.Automation.PSCredential ($viUser,$viPassword) | |
} | |
function Verify-ViServer { | |
$viServer = Read-Host -Prompt "`n Input the host name and then press Enter" | |
Write-Host "`n Checking connection to ""$viServer""..." -ForegroundColor Green | |
while ($(Test-Connection -ComputerName $viServer -Count 2 -ErrorAction Ignore) -eq $null) { | |
Write-Host "`n Host ""$viServer"" is not reachable." -ForegroundColor Yellow | |
$viServer = Read-Host -Prompt "`n Input the correct host name and then press Enter" | |
} | |
$script:viServer = $viServer | |
} | |
if ($global:DefaultVIServers.Count -ne '0'){ | |
Write-Host "`n Connection has already established with the following host(s):" | |
$vcsaConnections = $global:DefaultVIServers | |
foreach ($vcsaServer in $vcsaConnections) { | |
Write-Host "`t* $vcsaServer`r" -ForegroundColor Yellow | |
} | |
$userAnswer = Read-Host "`n Would you like to continue using current connection(s)? [Yes/No]" | |
while ("Yes","No" -notcontains $userAnswer) { | |
$userAnswer = Read-Host "`n Please choose correct answer [Yes/No]" | |
} | |
if ($userAnswer -eq 'No') { | |
Verify-ViServer | |
Enter-Credentials | |
Write-Host "`n Connecting to ""$viServer""..." -ForegroundColor Green | |
Connect-VIServer -Server $viServer -Credential $viCredential -ErrorAction Ignore -WarningAction Ignore | Out-Null | |
} | |
else { | |
continue | |
} | |
} | |
else { | |
Verify-ViServer | |
Enter-Credentials | |
Write-Host "`n Connecting to ""$viServer""..." -ForegroundColor Green | |
Connect-VIServer -Server $viServer -Credential $viCredential -ErrorAction Ignore -WarningAction Ignore | Out-Null | |
} | |
# Read all VMs that have VMXNET3 | |
$vmNetworks = Get-VM | Get-NetworkAdapter | Where-Object {$_.Type -eq 'Vmxnet3'} | |
# Get information about the network card | |
for ($i=0; $i -lt $vmNetworks.Count; $i++) { | |
$vm = $vmNetworks[$i].Parent | |
$vmNetworkName = $vmNetworks[$i].NetworkName | |
$vmNetworkAdapter = $vmNetworks[$i].ExtensionData.DeviceInfo.Label | |
$vmNetworkAdapterDPIOStatus = $vmNetworks[$i].ExtensionData.UptCompatibilityEnabled | |
if ($vmNetworkAdapterDPIOStatus -ne 'True') { | |
$dpioStatus = "Disabled" | |
} | |
else { | |
$dpioStatus = "Enabled" | |
} | |
$properties = @{'Virtual Machine' = $vm.Name; | |
'HW Version' = $vm.Version; | |
'Network Adapter' = $vmNetworkAdapter; | |
'Network Name' = $vmNetworkName; | |
'DPIO Status' = $dpioStatus} | |
$object = New-Object –TypeName PSObject –Property $properties | |
$outputCollection += $object | |
} | |
# Disconnect from the host(s) and clear the screen | |
Disconnect-VIServer * -Confirm:$false | |
Clear-Host | |
# Print out the results | |
$dpioResults = $outputCollection | Sort-Object -Property "Virtual Machine" | select Virtual*,*Version,*Adapter,*Name,*Status | |
Write-Host "`n DirectPath I/O status`r" | |
$dpioResults | |
# Save results to CSV | |
$csvPath = $env:TEMP + "\DPIOStatus-" + $(Get-Date -Format 'ddMMyyyy') + ".csv" | |
$dpioResults | Export-CSV -Path $csvPath -NoTypeInformation -Force -Confirm:$false | |
Invoke-Item $csvPath |
Feel free to use this code.
28/11/2016 – Update 1: vSphere Client (C#) and vSphere HTML5 Web Client (Fling) also create VMs with the DirectPath I/O option enabled.
28/11/2016 – Update 2: The workaround is to use PowerCLI to provision a new VM.
26/04/2017 – Update 3: The solution to this problem has been provided by LucD. I am working on an updated script which automate the process of disabling DirectPath I/O for the VMs.
10/07/2018 – Update 4: The script was updated to provide better reporting, including export to the CSV format.
Hi, any idea how I could possibly check this out on HW v10? For some reason creating a new VM from powercli is enabling DirectPath I/O on the network card and I can’t seem to get it disabled (The template is disabled) and can’t find .ExtensionData.UptCompatibilityEnabled. Any help is appreciated!
LikeLiked by 1 person
Hi simnether,
Thank you for your question.
I have updated my script to show information about the DirectPath I/O status regardless the VM hardware version.
LikeLike
Can ‘DirectPath I / O’ option be disabled by PowerCLI?
LikeLike
Hi @seishimurakami,
LucD has provided a working script that enables / disabled the DirectPath I/O setting. Please see Update 3 in my post for a link.
LikeLike
Mr. LucD has answered how to disable/enable DirectPath IO by PowerCLI in VMware Communities. It’s amazing!!
https://communities.vmware.com/message/2669952#2669952
LikeLike
Where is the recommendation to disable the parameter? I see that they describe how to change the behavior when creating a VMXNET3 NIC, but not a specific recommendation as to how/why the setting should be set. I’m asking because I just discovered the setting was enabled on my templates and so there are several VMs deployed with this setting enabled now, but the “DirectPath I/O” status is “Inactive” on all the VMs, probably because there’s no backing PCI device configured for DirectPath. So I’m just wondering if anyone has any information about the impact of having the checkbox for DirectPath enabled but DirectPath itself being inactive, whether this causes any issues or not.
LikeLiked by 1 person
Hi Steve,
Thank you for asking this.
The recommendation has come from the VMware GSS engineer when they’re troubleshooting one of the cases.
I understand, it’s more about the good housekeeping rather than any problems caused by having DirectPath I/O enabled for VMXNET3 adapters. However, this feature was designed for the hardware directly presented to the VM and not for the emulated or para-virtualised devices (https://blogs.vmware.com/performance/2010/12/performance-and-use-cases-of-vmware-directpath-io-for-networking.html).
So I prefer DirectPath I/O being disabled on VMXNET3 by default.
LikeLike
Any update on the script that disables this?
LikeLike
Sorry for not answering earlier. It was quite busy month for me. I am still working on the script and let you know when it’s ready.
LikeLike
hi, allow me to ask a dummy quesiton, is directI/O path = upt (universal pass through) ? TQ
LikeLike
Hi Raymond,
I’ve never heard about upt before.
However, you can learn about DirectPath I/O following this link – https://blogs.vmware.com/performance/2010/12/performance-and-use-cases-of-vmware-directpath-io-for-networking.html.
Hope it helps.
LikeLike
hi i tried the script above in VW 6.0 pwrcli 5.5 and happened to see the output is “disabled”. just to cross check i picked a server and saw the directio setting is “enabled”. is there something im missing or do i need to run this script with pwrcli6.0?
LikeLike
Hi Yash,
I haven’t tested the script with PowerCLI 5.5.
Just to confirm, what was the hardware version of those VMs and which version of vSphere they were running on?
LikeLike
thanks for the reply, here is the details. hardware Version is V8 and they are running on 6.0
LikeLike
just asked simsync to share the version of pcli in the other post (update 3 in your post) as i understand he had same problem and got it fixed.
LikeLike
[…] 6.x: VMXNET3 and DirectPath IO issue In this article it talks of an issue from long ago. However, I heard about it as it was causing performance […]
LikeLike
[…] I noticed on a bunch of my VMs deployed in the last year or two, that they had Direct IO Path enabled. Thanks for the guys in the vExpert Slack channel for cluing me in. At least one of them had performance issues related to this odd issue. […]
LikeLike
Does anyone have a script for changing this setting across multiple VM’s? LucD’s script is good for changing a single VM but it would be good to be able to change this across a selection of VM’s.
LikeLike
Hi Fred, Will be available ober this weekend. Thanks.
LikeLike
Apologies for the daft question but my PS skills aren’t fantastic.
What’s the best way to export this report to a CSV/Excel file?
I’ve used ‘export-csv’ but I get a long string list when trying to do this.
Final line:
$OutputCollection | Sort-Object -Property “Virtual Machine” | Format-Table Virtual*,*Version,*Adapter,*Name,*Status -AutoSize | export-csv “C:\export\DPIO_Status.csv” -notypeinformation
I also put the export into the For loop to append to a text file but it still produces the same result.
LikeLike
Hi Nick, Thank you for providing this feedback. I am working on improving this script and will publish an updated version over the weekend.
LikeLike
Hi Nick, The script has been updated with the CSV export. Please let me know if any questions.
LikeLike
It’s November 2018,I’m just looking at 6.7U1 and Directpath I/O is still default enabled.
LikeLike
Correct, the quality control is one of the things that always require attention.
LikeLike
[…] to Roman over at Virtual Nomad I found a PowerCLI script to quickly obtain a list of the VMs that have DirectPath I/O enabled. […]
LikeLike
Hello Roman,
Disabling DirectPath I/O went smoothly for most VM’s. (no eventlog messages)
However, our Windows clusters were triggered to perform failovers.
Apparently disabling DirectPath I/O for a short moment disrupts network traffic, eventhough DirectPath I/O is inactive. I did not expect that.
“Cluster node ‘######’ was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.”
LikeLike
Thank you for sharing this information Marcel.
LikeLike