VMware vSphere NVMe High-Performance Plug-in (HPP)

• Author
• Recent Posts

Brandon Lee

Brandon Lee has been in the IT industry 15+ years and focuses on networking and virtualization. He contributes to the community through various blog posts and technical documentation primarily at Virtualizationhowto.com.

Latest posts by Brandon Lee (see all)

• Docker logs tail: Troubleshoot Docker containers with real-time logging - Wed, Sep 13 2023
• dsregcmd: Troubleshoot and manage Azure Active Directory (Microsoft Entra ID) joined devices - Thu, Aug 31 2023
• Ten sed command examples - Wed, Aug 23 2023

There is no doubt that storage performance has increased exponentially over the past few years. One of the major advancements in the world of storage has been the introduction of devices that make use of the NVMe protocol.

NVMe devices have revolutionized storage performance across the board, including the world of virtualized environments. VMware vSphere has steadily been making advancements in how it handles NVMe devices and improving its ability to take advantage of the performance that NVMe brings.

In vSphere 6.7, VMware introduced a new component called the High-Performance Plug-in (HPP). By enabling the HPP, VMware vSphere ESXi leverages NVMe devices and maximizes throughput. In this post, we will take a look at NVMe storage and how to make use of HPP and activate it in VMware vSphere for your NVMe devices.

The rise of NVMe in the data center

Traditionally, storage has been the bottleneck of performance. Compute and network have remained ahead of the performance curve when compared to storage in years past. NVMe devices have changed that paradigm.

NVMe devices have become increasingly popular across the board in the modern data center and provide incredible performance advantages over traditional spinning disk and even flash storage. NVMe is actually a storage protocol designed specifically for very high-performing, multi-queue storage communication.

With NVMe, production workloads benefit from the extremely low latency, high-bandwidth, and queue-depth advantages it offers. ESXi supports the NVMe protocol to connect to local and networked storage devices.

Pluggable storage architecture

How does VMware ESXi handle storage management, specifically multipathing? VMware makes use of what is referred to as Pluggable Storage Architecture (PSA), which is a special VMkernel layer component. The PSA is a modular framework built on open standards that allows for coordinating the software modules that are responsible for multipathing operations.

Let's first take a look at the native storage plugin, which is a generic multipathing module provided by default by VMware, prior to and including ESXi 6.7 and higher. The Native Multipathing Plug-in (NMP) is the plugin included with ESXi that handles multipathing by default. It is an extensible module that enables having sub-plugins controlled by the module.

There are two sub-plugins that are provided for by NMP:

• Storage Array Type Plug-ins (SATPs)
• Path Selection Plugins (PSPs)

Both of these sub-plugins (SATP and PSP) can be provided by VMware or a third-party provider. The NMP provides many capabilities for the ESXi hypervisor and how it interacts with storage, such as:

• Manages claiming and unclaiming physical paths
• Associates physical paths with logical devices
• Registers and unregisters logical devices
• Processes I/O requests to logical devices
• Handles failures and storage request retries
• Optimizes load balancing of physical paths
• Halts or rests logical storage devices

VMware HPP

Starting with VMware ESXi 6.7, VMware introduced the HPP. Think of the HPP as the generic "high performance" version of the NMP that is provided by default for storage devices in VMware ESXi. The NMP was designed and included in ESXi before the widespread adoption of NVMe.

The HPP is purpose-built for ultra-fast, high-performance flash devices that are locally installed in your ESXi host. Note that it is selected by default when NVMe-oF targets are provisioned in ESXi. For multipathing, HPP selects and uses the Path Selection Schemes (PSS).

When looking at my ESXi hosts that have locally provisioned and directly installed NVMe devices, I found that the HPP is not automatically selected to handle NVMe storage. Below is an example of viewing the NMP devices. Notice I have an NVMe drive that has been claimed by NMP and not HPP. This is an ESXi 7.0 host.

esxcli storage nmp device list

VMware ESXi using NMP by default for NVMe drive

Switching NVMe Device from NMP to HPP

So we have an NVMe device that is using the legacy storage module. Let's see the process for switching the NVMe device to use the newer HPP to maximize performance. There are only three steps in the process. These include:

1. Create an HPP claim rule—Claim rules determine which multipathing module owns the paths to a particular storage device. They also define the type of multipathing support that the host provides to the device.
2. Configure the PSS—Handles multipathing for NVMe storage devices.
3. Reboot the host.

To create the claim rule, the syntax is fairly simple. You can create the claim rule based on the NVMe controller model or based on the PCI vendor ID and subvendor ID. Below, I am creating a new claim rule based on the NVMe controller model. The syntax allows for wildcards, which makes it easier to create the rule.

esxcli storage core claimrule add --rule 200 --type vendor --nvme-controller-model "Samsung*" --plugin HPP

Now for configuring the PSS for the NVMe device. Following the VMware documentation here, I used the following:

esxcli storage core claimrule add --rule 300 -t vendor -V "Samsung*" -M "Samsung*" -P HPP --config-string "pss=FIXED"

Next, reboot your ESXi host.

As you can see below, after a reboot, the Samsung NVMe device is now claimed using HPP. You can see the NMP and HPP devices with the following commands:

esxcli storage nmp device list
esxcli storage hpp device list

Checking the nmp and hpp storage devices after rebooting ESXi host

HPP best practices

There are a few best practices related to using HPP with NVMe storage devices. These include:

• Use a vSphere version that supports HPP when using NVMe storage.
• Use HPP for NVMe that is either local or network-attached.
• HPP should not be used for traditional HDD storage or slower flash devices. A good rule of thumb is that the device needs to be capable of at least 200,000 IOPs.
• Use the ParaVirtual storage controller with your virtual machines.
• If a single VM takes the majority of the IOPs available, spread the IO over several VMDKs and use multiple virtual storage controllers.

Wrapping up and thoughts

As NVMe becomes more and more common in hardware running in the data center, it is good to understand how various platforms and solutions interact with and handle NVMe compared to traditional storage and flash. The selection of NMP or HPP underscores how you can squeeze even more performance out of your NVMe devices with VMware vSphere.

Switching to HPP compared to NMP is an easy process that involves simply creating a matching claim rule and path selection scheme. After this, a reboot makes the claim rule active. Once your NVMe storage device is matched, HPP becomes active in managing your NVMe device.