Over time with the evolution of VMware vSphere, there have been some changes to VMware memory optimization techniques and their implementation.

VMware vSphere uses different memory optimization techniques (also called memory overcommitting). It doesn't use them all the time and all at the same time but rather one after another depending on the level of memory resource oversubscription. They include:

  • Transparent page sharing
  • Memory ballooning
  • Memory compression
  • Swap to host cache
  • Regular swapping

All of these techniques dynamically reduce the amount of machine physical memory required for each virtual machine (VM).

Virtualization increases the amount of physical memory required due to the extra memory ESXi needs for its own code and for data structures. This additional memory requirement comprises two main components:

  1. A system-wide memory space overhead for the VMkernel and various host agents (hostd, vpxa, etc.)
  2. An additional memory space overhead for each VM

Transparent page sharing ^

Transparent page sharing (TPS) is a technique for deduplicating RAM used in VMs, within themselves or between them. TPS allows you to save money on memory resources and consolidate them.

If the VMs running on the ESXi hosts have the same data in RAM (for example, the same OS and same application), this will deduplicate these memory spaces and use only a copy of the data on the physical memory module.

Notably, this feature is disabled by default since the discovery of security vulnerabilities regarding access to data through sharing RAM pages. In this way, an attacker could determine an AES encryption key of another VM sharing the same memory pagination on the same processor by forcing a flush and reloading of the memory cache.

However, this would be unlikely to happen in a production environment. But since VMware's policy is to deliver secure products, they've disabled this feature by default.

To reactivate the TPS feature, you only need to change the advanced settings of the ESXi host. To do this, first understand how you want to share identical memory pages—only between VMs hosted on an ESXi host or between all VMs, or even between specific VMs only?

To reactivate TPS, here are the parameters to modify for particular cases:

  • Between VMs running on the same ESXi host (we will enable this solution): Change the ShareForceSalting = 0 value (in the ESXi host advanced settings)

The sched.mem.pshare.salt value is useless because it's ignored (in the advanced settings for each VM).

  • Between all VMs: Change the ShareForceSalting = 1 value (in the advanced ESXi host settings)

The sched.mem.pshare.salt value is not present (in the advanced settings for each VM).

  • Between specific VMs (by groups of VMs you trust for using TPS): Change the ShareForceSalting = 1 value (in the advanced ESXi host settings)

Add the sched.mem.pshare.salt = value dedicated to the VM group (in the advanced settings of each VM in the group).

  • Only deduplication in the VM itself but not between VMs: use the value ShareForceSalting = 2

How do we change the Mem.ShareForceSalting value?

Select the ESXi host on the vSphere Client, go to the Configuration tab, select Advanced Settings, the Mem section, and change the Mem.ShareForceSalting value.

How to change the Mem.ShareForceSalting value

How to change the Mem.ShareForceSalting value

Disable large memory pages ^

It is not enough to activate TPS to take advantage of its functionality because the larger the memory pages, the less deduplication.

So disabling large memory pages will divide them into smaller fractions that will be easier to deduplicate. However, this also means there will be more CPU processing on a larger number of memory pages.

To disable "large pages," you have to do the same thing as for Mem.ShareForceSalting, but this time, we will modify the Mem.AllocGuestLargePage value.

Give the following value: Mem.AllocGuestLargePage = 0.

Ballooning ^

Ballooning is another optimization technique that VMware vSphere uses. If the host memory gets lower and lower, and the VM's memory usage approaches its memory target, ESXi will use ballooning to reduce that VM's memory demands.

It uses the vmmemctl module installed on the guest operating system (OS) as part of VMware Tools. ESXi can cause the guest OS to flush the memory pages it considers least valuable.

To use ballooning, you must configure the guest OS with sufficient swap space. There is no special configuration needed, but you must install VMware Tools within the guest OS of the VM.

Memory compression ^

This is the third option to save some memory that ESXi uses. If the VM's memory usage approaches the level at which host-level swapping will be required, ESXi will use memory compression to reduce the number of memory pages it will need to swap out.

Because the decompression latency is much smaller than the swap-in latency, compressing memory pages has significantly less impact on performance than swapping out those pages.

Compression takes place in memory where swapping takes place within the storage system. And storage in general is slower, with more latency.

Swap to host cache ^

Swap to host occurs only when exhausting the above methods. If memory compression doesn't keep the VM's memory usage low enough, ESXi will forcibly reclaim memory next using host-level swapping to a host cache.

The swap to host cache feature allows users to configure a special swap cache on SSD storage. In most cases this host cache (being on SSD) will be much faster than the regular swap files (typically on hard disk storage), significantly reducing access latency.

System swap configuration options

System swap configuration options

Thus, although some pages ESXi swaps out might be active, swap to host cache has a far lower performance impact than regular host-level swapping. We will cover this further in the next paragraph.

Regular host-level swapping ^

If the host cache is full or isn't configured, ESXi will reclaim memory next from the VM by swapping out pages to a regular swap file.

Like the swap to host cache feature, some of the pages ESXi swaps out might be active. This mechanism can cause VM performance to degrade a lot compared to all other methods described above. This is because there is usually a high access latency.

Final words ^

VMware memory overcommitting uses some powerful optimization techniques that allow optimizing memory within your datacenter for your datacenter workloads. It is especially useful for virtual desktop infrastructure (VDI) workloads where you have VMs with the same OS running side by side.

The transparent page sharing, ballooning, memory compression, and swap to host cache features allow you to achieve some great memory overcommitment—usually with little or no impact on performance.

However, you should avoid overcommitting memory to the point where you might use host-level swapping to swap out active memory because then the performance impact might be significant.

Subscribe to 4sysops newsletter!

All VMware vSphere memory overcommitting techniques are safe and do not harm your workloads any way. The only attention you need to pay is for regular host swapping, which, as the last option, can cause performance degradation.

avataravataravataravataravatar
2 Comments
  1. Leos Marek 3 years ago

    Hi Vladan,

    nice post there. Some comments from my side:

    TPS – nice feature, usefull for VDI yes. But saying that the vulnerability is unlikely to happen in production environment – well, its the case for many (or most) such vulnerabilities, but still this needs to be taken into consideration. Same as you pay life insurance when the probability of happening something is really small.

    Compression – this brings some CPU overhead and the gain is not so big. 

    Swapping to host cache – even if going to SSD, the speed compared to RAM is still reduced by large numbers there.

    In general I would say all of these technologies were not really created to be used on daily basis in production env, but rather handling overcommit situations in order to prevent the VM stop. Also, overcommitting memory is not likely to be seen in production env as the RAM modules are quite cheep and the performance degradation is too high. For dev/test its OK.

    Cheers Leos

  2. Benny Tate 3 years ago

    Excellent blog.  This helped me better understand these memory management setup options.  I'm not a production environment, just a home lab.   I have 64GB total on my host. I have numerous Oracle application and database VMs which requires about 16GB per VM.  along with that i have other VMs running for veeam, VCSA, Plex media server, and SVN repository.

     

    I'm experimenting with the shareForceSalting value and also disabling large page size. 

    I've also enabled 35GB host caching on an SSD datastore

    I'm going to try starting three of my Linux VMs that use 16GB each when their database and application services are starting up.

     

    Curious to see how this goes!

     

    Thanks again

Leave a reply

Your email address will not be published.

*

© 4sysops 2006 - 2022

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account