January 8, 2015 - David Sokolik
Delayed ACK in ESXi
Disable Delayed ACK in VMware ESXi for iSCSI:
A lot of system administrators I encounter are faced with various levels of performance issues most of them are related to host resources over commitment and some are because of poor I/O performance.
If you are using iSCSI protocol for your main storage communication then you might have issues with read/write latency, one the of the ways to deal with this issue is to disable delayed ACK.
This has been described in VMware KB1002598 ESX/ESXi hosts might experience read or write performance issues with certain storage arrays.
Now I’ve noticed that most storage vendors HP,DELL and even NetApp have addresses this issue and all have recommended to disable delayed ACK as the first step.
So what is Delayed ACK anyway?
When talking about ACK and the affects of Delayed ACK we first need to understand where the idea of ACK comes from, it all begins with the main difference between the TCP and UDP protocols
TCP is a trusted form of communication while UDP is untrusted.
When your computer sends a TCP packet it expects an ACK packet to be return acknowledging that the packet was successfully received by the remote side.
When you computer send a UDP packet it is unaware of it has arrived or if was even in its complete form.
ACK is explained in great detail in RFC 813 but basically the network traffic flow looks something as described in this drawing:
So if we take that and apply it to our server <-> storage scenario what happens is that only the most recent or “highest numbered command” as it is referred to in the RFC is acknowledged, which is basically acknowledging an unspecified number of previous commands.
Which results in a delay that is by design occurs for efficiency and congestion avoidance, this delay in ESXi can be anywhere between 100–500ms.
The iSCSI storage array needs to verify that their data was been properly received. so they throttle the network to make sure they receive their ACK packet with an expected time frame of a few ms
But if we turn off the delayed ACK for iSCSI traffic our network traffic flow will look something similar to:
And when the Delayed ACK has been turn off this will increase the amount of bandwidth used and the usage of CPU, but it will allow the storage array to confirm the receipt of data much faster and will improve I/O performance.
Note: Since iSCSI traffic should be on its own switch or at least on its own vLAN this change shouldn’t affect your overall network performance
How to disable Delayed ACK:
I will only address ESXi 5.x, if you have running ESXi 4.x please refer to VMware KB1002598
There are two ways to disable Delayed ACK within the VMware ESXi platform both are from within the vSphere client
First we need to open our vSphere client and connect to either our vCenter or directly to ESXi host
Go into the ‘Inventory‘ view and select the desired ‘Host’
Select the ‘Configuration‘ Tab and navigate to the ‘Storage Adapters‘ sections (right side in the middle)
Right click the ‘iSCSI Software Adapter‘ and select ‘Properties‘
From where we can either choose to change the setting globally or per iSCSI connection (network connection)
Click on ‘Advanced‘ go all the way to the button and uncheck the ‘DelayedACK‘ checkbox
If you wish to remove the setting on per iSCSI connection basis go to the ‘Dynamic Discovery‘ and/or ‘Static Discovery‘ Tabs select the ‘Server‘
Click on ‘Settings‘ and go into ‘Advanced‘ go all the way to the button. Uncheck the ‘Inherit from parent‘ checkbox and then uncheck the ‘DelayedACK‘ checkbox
Click ‘OK‘ twice and repeat the process for the desired connections.
After you have completed the change the ESXi host will require a reboot