Install ESXi in a VirtualBox VM

• Author
• Recent Posts

Brandon Lee

Brandon Lee has been in the IT industry 15+ years and focuses on networking and virtualization. He contributes to the community through various blog posts and technical documentation primarily at Virtualizationhowto.com.

Latest posts by Brandon Lee (see all)

• Windows doesn’t start: Recover partitions, copy files, and reset password with SystemRescue - Tue, Mar 28 2023
• ManageEngine OpManager: Comprehensive monitoring for on-prem, cloud, and containers - Thu, Mar 23 2023
• Install K3s, a lightweight, production-grade Kubernetes distro - Mon, Mar 20 2023

When I attempted to install ESXi 7.0 and ESXi 8.0, I received various purple screen of death (PSOD) messages.

AP 1 is incompatible with BSP; cannot start

In addition, when I did get the ESXi 7.0 installation to work, there were network issues with a lack of network adapter drivers.

PSOD when installing ESXi 7 or 8

I decided to install VMware ESXi 6.7 in VirtualBox for various testing purposes, as it does not appear to have the issues associated with the newer releases.

Create the ESXi-VM in VirtualBox

Once you have VirtualBox installed and configured on your host system, the first step is to create the virtual machine that will be used for the ESXi installation. Click the New button in the VirtualBox GUI.

Creating a new VM in VirtualBox

This launches the Create Virtual Machine wizard. On the first screen, you need to name the virtual machine, choose the storage location for the VM files, and point it to an ISO file for installation. I downloaded the ESXi 6.7 installation ISO and configured the path in the ISO image configuration. Choose Linux 2.6 / 3.x / 4.x / 5.x (64-bit) for the version.

Create a new VM for ESXi and point to the ISO

On the Hardware configuration screen, set the values for RAM and CPU. When configuring your virtual hard disk, you might factor in the storage space demand for VMs you run on top of ESXi.

Create a new virtual hard disk for installing ESXi

Finally, you will see the Summary screen. Click Finish.

Changing the settings of the VirtualBox VM

After we create the virtual machine that will house the ESXi installation, we need to make a few changes to its settings. Under System > Motherboard, select the following:

• Chipset: ICH9
• Pointing Device: PS/2 Mouse

Changing the Chipset and Pointing Device configuration

On the Processor tab, enable the Extended Features > Enable PAE/NX and Nested VT-x/AMD-V settings.

Change the Processor configuration of the ESXi VM in VirtualBox

For the Network configuration, select the Host-only Adapter. For Promiscuous Mode, select Allow All.

Updating the network configuration

Install ESXi

The final step is installing VMware ESXi. After updating the settings, we power on the ESXi VM in VirtualBox.

Boot into the ESXi installer

This will launch the familiar installer that guides you through all steps of setting up VMware's hypervisor.

If you receive the warning below, you need to make sure you have set the Enable Nested VT-x/AMD-V setting in the properties of your VirtualBox VM. The installer will proceed with the warning. However, this may prevent you from running nested virtual machines on the ESXi installation.

Warning about hardware virtualization

Hyper-V-enabled VirtualBox hosts

VirtualBox virtual machines can run in parallel with Hyper-V with a few settings changes. Keep in mind that if you are running modern security technologies such as virtualization-based security (VBS), they use Hyper-V as the underlying virtualization technology.

To run VirtualBox with Hyper-V technologies enabled, you must add the Virtual Machine Platform optional component.

Adding a Virtual Machine Platform to Windows hosts

Then, in VirtualBox, you can select Hyper-V for the Paravirtualization Interface.

Enabling the Hyper V paravirtualization interface

Wrapping up

VirtualBox has long been a popular enthusiast virtualization platform used for running various operating systems on top of desktop operating systems. It is free to download and provides various installation capabilities and configuration features.

As shown, it is a viable option to install and play around with VMware ESXi. However, there appear to be incompatibilities with ESXi 7 and higher, both from a CPU and network perspective. In my testing, installing these versions of ESXi ended with mixed results.

There may be a workaround for this issue or an ESXi kernel option that allows bypassing a check initiated with newer ESXi installations to prevent PSODs. It may also be possible to slipstream additional network drivers to work with VirtualBox.