VMware ESX Server guest OS performance tips, part two

David Marshall and Wade Reynolds
11.28.2006

Even though VMware ESX Server is known for high performance, performance tweaks and a little know how can make it even better. In this series, we are examining 12 tips for top-notch ESX Server performance.

Tips one through four discussed VI3, host processors and memory, storage and host networking. Now we'll turn our attention to VM-to-host placement, remote access, virtual processors and memory and more.

Tip 5: Be aware of VM-to-host placement

Within an ESX 2.x environment, it is important to keep track of what is going on inside of the virtual machine. By distributing heavy load virtual machines on different hosts or by placing virtual machines that operate at different times throughout the day on the same host, you can increase performance of your virtual machines by reducing resource contention. This is a manual task when using VMware ESX Server 2.x.

Because virtual machines are easy to create and distribute, it's common to have a problem known as virtual machine sprawl. If it isn't managed properly, you can have unmanaged virtual machines out there that people forget about.

It's important to either shut down/power off or suspend unused virtual machines so that they aren't wasting valuable resources on the ESX Server. Even an idle virtual machine will chew up memory and processing power, taking it away from other virtual machines that could use it.

Upgrading to VI3 can help out with your VM-to-host placement problem. Rather than manually trying to figure out where a virtual machine best fits on which host server and with which other virtual machines, VI3's DRS and resource pool features will help do the job for you.

When deploying a virtual machine in VI3 with DRS, you no longer need to figure out which ESX Server is going to host the virtual machine. You simply assign the virtual machine to a resource pool. DRS will automatically move the virtual machine to the right ESX Server so that it can get the resources that it needs. DRS will automatically balance your virtual machines across all of the ESX Servers assigned to the resource pool, even after virtual machines are added, removed and modified.

If a virtual machine isn't getting enough resources, then DRS can automatically balance the virtual machine in the resource pool to ensure that all of the virtual machines are receiving enough resources. If a resource pool itself is running out of resources, additional resources can be added by simply adding additional ESX servers to the resource pool.

VI3's DRS feature requires VMotion capabilities to perform the automatic migrations. It can move virtual machines from one ESX Server to another without any downtime. It takes some of the guesswork out, and by doing so, virtual machine performance is optimized because the VM no longer has to deal with resource contention as it would if it were placed on an improper host server.

Tip 6: Use remote access carefully

Some virtual machine performance enhancements can be as simple as closing an unnecessary VMware Remote Console (VMRC) session. Although you might not think that this would cause a problem, every remote console session that is open is consuming valuable CPU resources in the service console.

You can improve performance of the virtual machine by simply not connecting through the remote console. The VMRC protocol is not optimized and was never designed to be used as a standard remote technology. Exceptions to this rule would be using the VMRC when KVM type access to the virtual machine is needed for doing occasional administrative functions or when console access is explicitly required.

To optimize performance, you should use third-party remote control software like the tools you use to remotely manage your physical servers. By using lighter-weight remoting technologies like the Citrix ICA client, Microsoft Terminal Services or Remote Desktop Protocol (RDP), Telnet or SSH, you can connect to your virtual machines without adversely consuming CPU resources from the ESX host's service console.

Keep in mind that not all guest-oriented remote access methods are created equally. As an example, some organizations use the Virtual Network Computing (VNC) viewer, and while it may consume more resources than some of the other viewers, it too is still a better choice than using the VMRC client.

Tip 7: Virtual machine processors and memory

In a physical environment, for better or worse, we are taught that increasing the number of processors or adding additional memory to a server increases the performance of the operating system and its applications. This same idea has migrated over to the virtualized environment. Unfortunately, multiple processors and sharing of memory will increase the load on your ESX server. And this is exactly what you are trying to eliminate.

If you notice slow performance on your virtual machine, examine its CPU usage. You should check to see how much idle time each processor has, and you should examine the overall system CPU utilization through the VMware Management Interface.

Adding additional processors to a virtual machine by using VMware's Virtual SMP is not always going to solve performance problems. Not all applications are able to take advantage of having multiple CPUs. Guest operating systems and applications should be analyzed to determine whether or not Virtual SMP would improve throughput.

If these applications are not multi-threaded or cannot use multiple processes in execution, adding Virtual SMP may consume valuable physical processor resources without actually offering the virtual machine any added performance benefit, and as a result, may take away resources from other virtual machines on the same physical host server.

Virtual SMP should be used sparingly. Running a virtual machine with a single virtual CPU will, in many cases, outperform the same virtual machine with Virtual SMP turned on. It is definitely a case-by-case scenario, and you should test each virtual machine environment.

If the slow performance is not caused by the CPU, examine the virtual machine's use of memory. It is important to determine if the guest operating system is paging or swapping memory. Since disk is slower than RAM, this bottleneck will need to be identified and corrected.

A number of tools and options are available to determine if paging is taking place. Within a Linux guest operating system, you can use the vmstat command. For a Windows guest operating system, use the Performance tool under Administrative Tools to check the value for pages/second. If a virtual machine has a high number of page faults, such as 1000 pages/second, increase its minimum amount of memory to eliminate the excessive paging taking place. If the minimum memory size is fast approaching the maximum memory size, then increase that resource setting.

Remember, you should allocate only as much memory to the virtual machine as is needed. Some due diligence and testing is needed here; giving a virtual machine additional memory will not always increase performance. In fact, that is a wasteful practice, because you are taking away valuable memory that could be allocated to an additional virtual machine or used to increase your virtual machine density on the host server.

Altering the virtual machine's minimum and maximum CPU resource allocation percentages is another way to affect performance. If you want to avoid CPU starvation for a virtual machine, set its minimum percentage to something other than zero. Conversely, to keep low priority virtual machines from consuming too many CPU cycles, you can set its maximum percentage to something lower, like 50%, thereby effectively throttling down that virtual machine and allowing other more pertinent virtual machines to make use of those valuable CPU cycles.

You can also control which physical processor or processors each virtual machine can use. This control is called processor affinity. The default setting is to use no affinity, and this is usually the best choice for most situations. You should only set a virtual machine's CPU affinity when absolutely necessary.

If you have a very resource intensive virtual machine running on a host server, you might want to set its CPU affinity to isolate that virtual machine and to protect its performance. Doing so will also protect the performance of all other virtual machines running on that same host server by changing each of their affinity settings to a different processor from that of the resource-intensive virtual machine.

Tip 8: Remove unneeded virtual hardware

In a physical server, it can be extremely difficult and time consuming to add/remove hardware components that are not constantly being used by the system. And typically, unused hardware devices on a physical server don't usually hinder system performance. This is not the case with a virtual machine.

Disabling or removing any virtual hardware components that are not being used by the virtual machine is a good way to increase a guest server's performance even if it is only by a very small amount. With a virtual machine, small performance boosts can make a world of difference across the entire host server.

If your virtual machine environment does not need a CD/DVD ROM drive, floppy drive, network adapter or COM and LPT ports, remove or disable them. If you later decide that your virtual machine needs one or all of these devices, virtualization makes it quick and easy to add them back at any time.

In the next part of this tip, we'll look at updating VMware tools, optimizing your guest operating systems and effectively using antivirus and backup tools.