One of our new customers using VMware has not been happy with the performance of some of their virtual machines that had been setup before they were our client. Specifically, a couple of their Citrix VMs and a SQL Server 2005 VM have been “sluggish since they were built” according to the IT staff. I did some basic diagnostics on the SQL Server VM and it did seem to have some performance problems. However, since they had already bought a new physical server and started moving the databases to the new install we didn’t spend much time trying to make the VM run well. We decided to upgrade to VMware vSphere v4.1 upgrade before attempting to address any of the performance issues, since less than stellar performance on virtual terminal servers was normal on VMware v3.5.
During the upgrade, I needed to vMotion some of the VMs around to take down one of the ESX hosts. I kept getting a very generic error on several of the VMs and the migration would fail. I lmust have ooked at every setting a dozen times until I finally just shut the VM down and opened up the VMX file to see what might be causing the issue. The problem was that the person who had built the VMs originally had included processor affinity settings in the VMX file. This binds a VM to a specific subset of the physical processors/cores on the ESX hosts. For example, on the SQL Server VM, it was bound to cores 0 & 1. With this setting, ESX was forced to schedule core 0 & core 1 for all operations even though the server had 8 cores. Additionally, on ESX, the service console has processor affinity on cores 0 & 1, but it holds the highest priority. So basically the SQL Server VM (and the other VM I found that was co-scheduled on cores 0 & 1) were fighting with the service console for processor cycles. After removing the processor affinity, the CPU wait time counter in vCenter for that VM dropped 6x. I ended up finding 10-12 VMs with processor affinity set, so that explained why the performance was terrible.
The moral of this story is to not manually schedule the processors when you configure ESX or Virtual Machines. Chances are the ESX schedule will be much better than any manual configuration you could put together.