VMware memory optimization

• Author
• Recent Posts

Vladan Seget

Vladan Seget is an independent consultant, professional blogger, vExpert 2009-2021, VCAP-DCA/DCD and MCSA. He has been working for over 20 years as a system engineer.

Latest posts by Vladan Seget (see all)

• How to use VMware vSAN ReadyNode Configurator - Fri, Dec 17 2021
• VMware Tanzu Kubernetes Toolkit version 1.3 new features - Fri, Dec 10 2021
• Disaster recovery strategies for vCenter Server appliance VM - Fri, Nov 26 2021

VMware vSphere uses different memory optimization techniques (also called memory overcommitting). It doesn't use them all the time and all at the same time but rather one after another depending on the level of memory resource oversubscription. They include:

• Transparent page sharing
• Memory ballooning
• Memory compression
• Swap to host cache
• Regular swapping

All of these techniques dynamically reduce the amount of machine physical memory required for each virtual machine (VM).

Virtualization increases the amount of physical memory required due to the extra memory ESXi needs for its own code and for data structures. This additional memory requirement comprises two main components:

1. A system-wide memory space overhead for the VMkernel and various host agents (hostd, vpxa, etc.)
2. An additional memory space overhead for each VM

Transparent page sharing

Transparent page sharing (TPS) is a technique for deduplicating RAM used in VMs, within themselves or between them. TPS allows you to save money on memory resources and consolidate them.

If the VMs running on the ESXi hosts have the same data in RAM (for example, the same OS and same application), this will deduplicate these memory spaces and use only a copy of the data on the physical memory module.

Notably, this feature is disabled by default since the discovery of security vulnerabilities regarding access to data through sharing RAM pages. In this way, an attacker could determine an AES encryption key of another VM sharing the same memory pagination on the same processor by forcing a flush and reloading of the memory cache.

However, this would be unlikely to happen in a production environment. But since VMware's policy is to deliver secure products, they've disabled this feature by default.

To reactivate the TPS feature, you only need to change the advanced settings of the ESXi host. To do this, first understand how you want to share identical memory pages—only between VMs hosted on an ESXi host or between all VMs, or even between specific VMs only?

To reactivate TPS, here are the parameters to modify for particular cases:

• Between VMs running on the same ESXi host (we will enable this solution): Change the ShareForceSalting = 0 value (in the ESXi host advanced settings)

The sched.mem.pshare.salt value is useless because it's ignored (in the advanced settings for each VM).

• Between all VMs: Change the ShareForceSalting = 1 value (in the advanced ESXi host settings)

The sched.mem.pshare.salt value is not present (in the advanced settings for each VM).

• Between specific VMs (by groups of VMs you trust for using TPS): Change the ShareForceSalting = 1 value (in the advanced ESXi host settings)

Add the sched.mem.pshare.salt = value dedicated to the VM group (in the advanced settings of each VM in the group).

• Only deduplication in the VM itself but not between VMs: use the value ShareForceSalting = 2

How do we change the Mem.ShareForceSalting value?

Select the ESXi host on the vSphere Client, go to the Configuration tab, select Advanced Settings, the Mem section, and change the Mem.ShareForceSalting value.

How to change the Mem.ShareForceSalting value

Disable large memory pages

It is not enough to activate TPS to take advantage of its functionality because the larger the memory pages, the less deduplication.

So disabling large memory pages will divide them into smaller fractions that will be easier to deduplicate. However, this also means there will be more CPU processing on a larger number of memory pages.

To disable "large pages," you have to do the same thing as for Mem.ShareForceSalting, but this time, we will modify the Mem.AllocGuestLargePage value.

Give the following value: Mem.AllocGuestLargePage = 0.

Ballooning

Ballooning is another optimization technique that VMware vSphere uses. If the host memory gets lower and lower, and the VM's memory usage approaches its memory target, ESXi will use ballooning to reduce that VM's memory demands.

It uses the vmmemctl module installed on the guest operating system (OS) as part of VMware Tools. ESXi can cause the guest OS to flush the memory pages it considers least valuable.

To use ballooning, you must configure the guest OS with sufficient swap space. There is no special configuration needed, but you must install VMware Tools within the guest OS of the VM.

Memory compression

This is the third option to save some memory that ESXi uses. If the VM's memory usage approaches the level at which host-level swapping will be required, ESXi will use memory compression to reduce the number of memory pages it will need to swap out.

Because the decompression latency is much smaller than the swap-in latency, compressing memory pages has significantly less impact on performance than swapping out those pages.

Compression takes place in memory where swapping takes place within the storage system. And storage in general is slower, with more latency.

Swap to host cache

Swap to host occurs only when exhausting the above methods. If memory compression doesn't keep the VM's memory usage low enough, ESXi will forcibly reclaim memory next using host-level swapping to a host cache.

The swap to host cache feature allows users to configure a special swap cache on SSD storage. In most cases this host cache (being on SSD) will be much faster than the regular swap files (typically on hard disk storage), significantly reducing access latency.

System swap configuration options

Thus, although some pages ESXi swaps out might be active, swap to host cache has a far lower performance impact than regular host-level swapping. We will cover this further in the next paragraph.

Regular host-level swapping

If the host cache is full or isn't configured, ESXi will reclaim memory next from the VM by swapping out pages to a regular swap file.

Like the swap to host cache feature, some of the pages ESXi swaps out might be active. This mechanism can cause VM performance to degrade a lot compared to all other methods described above. This is because there is usually a high access latency.

Final words

VMware memory overcommitting uses some powerful optimization techniques that allow optimizing memory within your datacenter for your datacenter workloads. It is especially useful for virtual desktop infrastructure (VDI) workloads where you have VMs with the same OS running side by side.

The transparent page sharing, ballooning, memory compression, and swap to host cache features allow you to achieve some great memory overcommitment—usually with little or no impact on performance.

However, you should avoid overcommitting memory to the point where you might use host-level swapping to swap out active memory because then the performance impact might be significant.

All VMware vSphere memory overcommitting techniques are safe and do not harm your workloads any way. The only attention you need to pay is for regular host swapping, which, as the last option, can cause performance degradation.