Hyper-V Performance Tuning - Part 4: Monitoring Hyper-V

This article gives some tips about Hyper-V performance monitoring. Among other things you learn how to use Hyper-V spefic counters, how to monitor comitted memory and memory buffers.
Profile gravatar of Paul Schnackenburg

Paul Schnackenburg

Paul Schnackenburg works part time as an IT teacher as well as running his own business in Australia. He has MCSE, MCT, MCTS and MCITP certifications. Follow his blog TellITasITis.
Profile gravatar of Paul Schnackenburg

Seasoned sys admins have a good idea of how to use Task Manager and Performance Monitor to understand what’s going on when there’s a server performance issue. Those skills transfer well to the virtual world but there are some gotchas that we’ll cover in this article.

Task Manager in VMs lies ^

The first rule is don’t ever measure performance of a VM from within a VM. Most sys admins first reaction to performance complaints will be to have a look in Task Manager. Unfortunately that doesn’t work in a VM because it can only see its little keyhole view of the world. Here’s a simple test to understand this; run one VM on a host, run an application (such as Prime95) that maxes out the virtual CPU(s). Task Manager will report 100% CPU but if you now start another VM and run the same application in that VM, Task Manager will still report 100% but it’s actually running at half the speed compared to earlier. This example disregards additional processors in the host but clearly demonstrates: Task Manager in VMs lies.

Create baselines ^

The second rule for performance monitoring, whether physical or virtual is to create baselines when things are humming along. Unless you’ve got a baseline when performance was good and users happy you have nothing to compare with when issues arise. Learn how to create Data Collector Sets; log for a few days and archive the results.

Use Hyper-V specific counters ^

When using Performance Monitor in the host, be aware that it might lie as well, don’t use normal counters as they often don’t understand VMs, use Hyper-V specific counters. For processor performance use the Hyper-V Hypervisor Logical Processor\% Total Run Time counter; this will let you monitor physical processors in the host. For VMs use Hyper-V Hypervisor Virtual Processor\% Guest Run Time which lets you monitor virtual CPUs for each running VM (or as a total). As a rule of thumb, less than 75% for this counter (total for all VMs) is healthy, over 75% is a warning sign and more than 85% should definitely be investigated.

Hyper-V Monitoring Hyper-V - Performance Monitor CPU

Committed Memory ^

In today’s Hyper-V world, B DM (Before Dynamic Memory), you should monitor \Memory\Available Mbytes on the host, healthy is more than 10% of RAM free, at less than 10% it’s a warning than something is going on and if it’s less than 100 MB it needs to be looked at. Old hands will know that “High pages per second” means a low memory condition but there are cases where this isn’t true (due to memory mapping of files), for more information see here. Use the \Memory\Committed Bytes counter to see how much memory each VM is using when planning your VM memory allocation.

Average Pressure ^

In the After Dynamic Memory (with SP1) world watch the \Hyper-V Dynamic Memory Balancer\Average Pressure counter, healthy is less than 80; a value between 80 and 100 deserves attention whilst over 100 indicates a critical condition.

Memory buffer ^

In the physical world as sys admins we always like to have a bit extra up our sleeve. If you were designing a server that you knew would use about 1.5 GB of memory you’d install 2 GB just to be on the safe side. In the VM world this is adjusted with the buffer, the default is 20%. For workloads that need a lot of file cache increase this value. This screen is also where you can assign a higher or lower weight to each VM which will allow a VM to have a higher (or lower) priority for memory as the host starts running low on available memory.

Hyper-V Monitoring Hyper-V - Memory Config SP1

Disk latency ^

Monitoring disk is done via the \LogicalDisk(*)\Average Disk Sec\Read or Write which indicates your disk latency. Healthy is less than 10ms (0.010), if it goes up to 15ms or above (0.015) it’s time to check it out, at 25ms or above (0.025) the situation is critical.

Network monitoring ^

For network monitoring use the counter \Network Interface (*)\OutputQueue Length, less than 1 on average is healthy, warning is when it’s above 1 on average, critical is when it’s 2 or more on average.

Conclusion ^

In this article series we’ve looked at tips for how to design a Hyper-V environment for best performance, some tricks and gotchas to look out for and finally we covered how to monitor performance in a virtualized world.

Resources ^

Here are a list of websites and whitepapers that you can use to delve deeper into this fascinating topic.

Virtual Reality Check compares performance of hypervisors.

Download a performance tuning whitepaper for both the physical and virtual world from Microsoft for Windows Server 2008 R2 here.

In-depth information on performance tasks

The Performance Analysis of Logs is a PowerShell script that can help you out in deciding which performance counters to collect and how to analyse them.

For disk performance tasks have a look at Iometer

Detailed technical information on how to measure performance in Hyper-V

Take part in our competition and win $100!

4 Comments
  1. avatar
    Bart Jacobs 6 years ago

    Hello,

    I wouldn't call Task Manager to be liar. I get your point, but in my opinion an important factor is left out, namely the difference between a vCPU and a "real" CPU. Task Manager in a VM controls the vCPU and how that vCPU is "translated" to the physical CPU is a core task of the hypervisor who spreads the load of vCPU's over the available physical cores.

    So it's not true to claim that when you start another VM, the 100% CPU load vCPU will be running at half the speed. The hypervisor will schedule all vCPU's of both VM's equally to the physical CPU's. There is no direct affinity between vCPU's and physical cores.

    What will happen is that the overall load on the physical CPU's will increase off course. This is why there is a best practice to "divide by 2". Say you have an Hyper-V server with a single QC CPU, you should limit VM's to 2 vCPU's to provide efficient scheduling.

    0
  2. avatar
    Paul Schnackenburg 6 years ago

    Hi Bart,

    Thanks for your comments. I agree that I could have explained the difference between a vCPU and a physical CPU / Cores better. But my advice still stands, if you have performance issues with a VM and your boss tells you to go and look at it you *can't* rely on what Task Manager inside the VM tells you. It doesn't really know about real CPU load etc because the "CPU(s)" it's looking at don't really have specific boundaries as the hypervisor is dynamically sharing physical CPU resources between all running VMs. And that sharing is affected by how many vCPUs/logical processors each VM has assigned to it, the number of physical cores and their speed, and the "VM Reserve", "VM limit" and "Relative Weight" assigned to each VM. The point is that all of that invisible from within the VM. I also don't know that the "divide by 2" best practice always applies, I think it's more important to look at the workload in each VM, if it's a multithreaded load which can be CPU bound that VM probably will benefit from 4 logical processors (provided it's running Win 08/08R2) whereas other VMs on the same host might work well with only one logical processor. In the physical world we need to apply performance monitoring and design when sizing systems, the VM world is no different.

    Thanks again for your comments - feedback is always welcome.

    0
  3. avatar
    Bart Jacobs 6 years ago

    Hi Paul, glad to see your appreciation. Bottom line is: when trying to figure out performance issues look at the VM and the host, instead of just one of them.

    I also have more experience with Citrix XenServer than with Hyper-V, but they are very much alike in concepts and architecture, and that's were I got that "divide by 2" rule

    0
  4. avatar
    Greg 5 years ago

    Hi Paul,
    Nice article, just for your and others information I had to use following:
    \LogicalDisk(*)\Avg. Disk sec/Read
    \LogicalDisk(*)\Avg. Disk sec/Write
    \Network Interface(*)\Output Queue Length

    Small spelling differences, the rest was ok.
    I tested Hyper-V R2 SP1.

    0

Leave a reply

Your email address will not be published. Required fields are marked *

*

CONTACT US

Please ask IT administration questions in the forum. Any other messages are welcome.

Sending
© 4sysops 2006 - 2017
Do NOT follow this link or you will be banned from the site!

Log in with your credentials

or    

Forgot your details?

Create Account