VMware ESX Server guest OS performance tips, part one

David Marshall and Wade Reynolds, contributers
11.27.2006

VMware ESX Server is one of the highest performing virtualization platforms on the market, but that does not mean that you don't have to optimize. In order to extract as much performance as is possible, you need to focus on three main areas.

First, the configuration of the ESX host server must be scrutinized at both the physical hardware layer and the virtualization platform layer. The decisions in this area will affect the performance of every virtual machine hosted on the ESX server. The second area of performance optimization is at the virtual machine layer with the various virtual hardware configuration options and factors. The third and final area is the configuration of the guest operating systems running inside the virtual machines.

In this three-part article, we'll list twelve top tips that can boost the performance of your virtual environment in all three of these areas. We'll cover everything from upgrades to remote access to configuration tweaks. Are you ready? Let's get started.

Tip 1: Upgrade to VI3

If you are still running VMware ESX Server 2.x, you ought to know that the new VMware Virtual Infrastructure 3 (VI3) could bring significant performance improvements. That's not to mention major scalability, usability and hardware compatibility enhancements. Upgrading your ESX 2.x hosts and virtual machines to VI3 (using ESX 3.0.1) could bring performance improvements straight out of the box.

VMware has optimized several subcomponents of ESX Server in version 3. In particular, you can expect faster handling of page faults, context switches and memory management unit operations within virtual machines. Virtual machines can now use Physical Address Extensions (PAE) to access up to 16 GB, more than the previous limit of 3.6GB of RAM, without taking a severe performance hit.

Virtual networking within ESX 3.0 has been improved, both in the vmxnet virtual network adapter and in VM-to-VM networking. VMFS3 is a new version of the VMFS family of file systems that offers enhanced performance and scalability. Virtual machines are less dependent upon the Service Console, and their user-level virtualization components can now run on any available processor core. According to VMware, this also allows ESX 3.0 to scale up to 50% more virtual machines per host over ESX 2.5.

ESX 3.0 can use the ACPI power saving mode for better handling of idle virtual machines. Virtual machines can take advantage of up to four virtual processors. Other features in VI3 that also enhance performance and scalability are the distributed resource scheduler (DRS), resource pools, high availability (HA) and VMware Consolidated Backup. ESX 3.0 hosts can now support up to 20 physical network adapters.

If you do nothing else, look into upgrading to VI3. The difference is akin to jumping from Windows NT Server 4.0 to Windows Server 2003.

Tip 2: Improve your host's processor and memory

Your host server's physical hardware greatly affects the performance of all virtual machines that reside upon it. Processor and memory are two vital resources in this area.

Although VMware ESX Server is compatible with a broad range of server hardware, including older server equipment, a top-notch CPU and more memory will improve the performance of your guest environments. Always use the highest performing processors you can afford.

When making processor decisions, consider the amount of power draw each processor incurs as well as its heat output. Some processors are more efficient than others, and some processors that have massive brute strength may have hidden costs of more power consumption.

Also be aware that some processors actually can downgrade their performance on the fly when thermal thresholds are reached. Although this may sound like a great feature for increasing your processor's life expectancy, it will destroy the performance of your virtual machines. It is usually better to select a more efficient processor that will not have any thermal issues in your use case.

In addition to using high performance processors, having more processors is usually better. Most new processors are available in multi-core versions that increase processor density while maintaining physical server densities in the racks. VMware ESX Server supports many processor cores in both the 2.5 and 3.0 versions. Having more available processor resources will allow higher virtual machine densities on each physical host server and will increase the performance of existing virtual machines by reducing CPU contention.

It is also important to optimize the ratio of virtual machines to physical processor cores on the host server. More cores equal higher densities, in most cases. Running too many virtual machines per processor core will adversely affect performance. The optimal ratio depends on the resource needs and utilization of the individual virtual machines themselves. The DRS feature in VI3, along with resource pools, can help automate this part of the equation.

Add as much RAM as possible to your ESX hosts so long as it matches up to proper virtual machine densities that can be supported by the processors. VMware ESX Server hosts have a minimum of 8GB RAM; 16GB, 32GB or even 64GB are not uncommon.

When raw virtual machine performance is the most important metric, avoid over-committing memory. Although ESX can handle over-committed memory, it causes swapping to the virtual machine swap file located on disk. Disk is always slower than RAM, so avoid swapping to disk in order to enjoy the benefits of using physical memory.

Tip 3: Enhance your host's storage

This may sound obvious, but using the highest performing disk I/O subsystem possible will improve guest performance. If you plan on using local or direct-attached disk storage, use a RAID-5 array of Ultra320 SCSI or SAS disks. Many disk controllers support multiple channels, and splitting your disks across those channels will improve performance.

For example, if you have six SCSI disks and a two-channel controller, place three disks on each channel and configure them in a six-disk RAID-5 array, if possible, with your server hardware. Note that some servers' storage backplanes may not allow splitting the I/O in this manner.

You can install multiple disk controllers and additional disks in a host and use multiple file systems, placing some virtual machines on each of those file systems. This can be especially useful for partitioning virtual machines that are I/O intensive. For example, if you have twenty virtual machines on a host, and two of them are I/O intensive, you can add the additional controller and disks and either move the I/O intensive virtual machines onto the second controller or split all virtual machines across each controller, ensuring that only one I/O intensive virtual machine is allocated to each file system.

Just like in the physical world, if you need to improve disk seek time, put faster disks in the server. Use 15K RPM disks in the ESX host to improve the I/O performance for all of the virtual machines on that host.

Using a SAN will offload I/O operations from the ESX host server, which leaves more resources to the virtual machines. Depending on your situation, having more available bandwidth may also increase your I/O performance. Use a 4Gb Fibre Channel SAN infrastructure when possible.

Your virtual machines will inevitably share access to the ESX host server's I/O subsystem. You should use the I/O share allocation for each virtual machine to adjust the amount of I/O resources that the virtual machine should be given, as appropriate. For virtual machines that are not very I/O intensive, you can set resource shares to 500 (low), and for the more resource intensive virtual machines that require more priority to I/O resources, set their shares to 2000 (high).

Tip 4: Optimize your host's networking

When your physical server is running VMware ESX Server, you should use Gigabit Ethernet network adapters for improved networking performance. This means that you must have Gigabit switches as well. Gigabit network adapters allow more virtual machines to share each physical network adapter over Fast Ethernet and greatly improve network bandwidth for network intensive virtual machines.

When configuring your physical network adapters, the speed and duplex settings on each network adapter must match the speed and duplex setting used on the switch port to which it is connected. Having conflicting settings between the network adapter and the switch can be not only a performance problem but also a connectivity issue.

When using 100Mbit Fast Ethernet adapters, set the network adapter and switch port speed/duplex settings to 100/Full. When using Gigabit Ethernet, set the network adapter and switch port speed duplex settings to Auto Negotiate, which is usually the default setting.

You can increase the available network bandwidth and increase network fault tolerance by teaming multiple Gigabit network adapters into a bond. This can simplify the number of virtual switches mapped to physical network adapters as well. In order to take advantage of the Gigabit network adapters in your virtual machines, configure your virtual machines to use vmxnet or e1000 virtual network adapters and avoid vlance when possible.

In the next tip, we'll discuss VM-to-host placement, processors, unnecessary hardware and more.