VM in new host slower than VM in older host

Few months back, I led a project to upgrade an in-house application to a newer version running on new hardware.  New ESXi hosts with better configuration were added to the existing cluster and new Virtual Machines were created for the new version of the application.

One of the objectives of the upgrade is to increase the performance of the application based on the fact that the new VMs will be on new and more powerful hardware. 

So the team conducted performance testing but was getting some weird results.

Initial test result showed that the new version of the application in the new VMs transact slower than the old one in the old VMs.  This is even so when the new VMs were running in the new ESXi hosts which supposed to be more powerful.  This is definitely not the result we were expecting.

Further testing showed that if we moved the old VMs to the new ESXi hosts, those older version of the application also transact slower compared to older version of the application in the old ESXi hosts.

We then compared the new VMs in the new ESXi hosts with the new VMs in the old ESXi hosts and found that the later performs better.

So the conclusion seems to be the new ESXi hosts was performing worse than the old ESXi hosts.  Now this led us to re-check the settings of the new ESXi hosts and perform more testing.  Every setting is correct until we compare the CPU Power Management.

Below is the CPU Power Management of the old ESXi host; it is a Dell server.  It was set to “Maximum Performance”.

Below is the CPU Power Management of the new ESXi host.  The setting was “Performance Per Watt (DAPC)” which is the default and provides a mix of performance balanced with power consumption reduction.

We changed the setting to “Performance” which matches the “Maximum Performance” of the old server.

We did a few round of re-tests and finally get to the result we have been expecting.