- What’s your ENow AppGov Score? Free Microsoft Entra ID app security assessment - Thu, Nov 30 2023
- Docker logs tail: Troubleshoot Docker containers with real-time logging - Wed, Sep 13 2023
- dsregcmd: Troubleshoot and manage Azure Active Directory (Microsoft Entra ID) joined devices - Thu, Aug 31 2023
There are many different tools and utilities that can be used for troubleshooting a VMware ESXi host. You're probably familiar with ESXTOP, various logging tools, vRealize Log Insight, vRealize Operations Manager, or other tools you can use to narrow down issues.
However, you may get to a point where you know that your ESXi installation is now corrupt, you have taken a patch you didn't intend to take, or you have found compatibility issues in your environment with a newer ESXi version. Of course, you can always simply boot from an ESXi ISO and reload ESXi from scratch, and then restore your configuration. However, there is another way with the potential to restore a good working state with your VMware ESXi host – rolling back the ESXi version. In this post, we will take a look at how to roll back and downgrade ESXi versions to get back to a good state.
Use cases for rolling back ESXi
We have already touched on one potential issue in the introduction. However, there are several reasons why you might want to roll back the ESXi version that is installed on your ESXi host. These include:
- You have taken a patch that is causing issues in your environment.
- You need to revert to a previous version of ESXi for a particular reason due to compatibility, supportability for a specific set of hardware, etc.
- You are bringing a standalone ESXi host into an existing vSphere cluster. Your standalone host may be at a higher version and you need to downgrade it.
- Your upgrade failed in some way and the ESXi installation is not functioning correctly.
- Rolling the ESXi version back is easier than reloading ESXi from scratch and restoring configurations.
- General troubleshooting.
These are just a few use cases you may encounter. Rolling ESXi back may make sense and save you a lot of time in reloading, troubleshooting, or performing other time-intensive tasks to get to a known good state.
How ESXi version rollback works
You may be asking how the ESX version rollback process works. There are a couple of special partitions to make note of with your vSphere ESXi installation. These are the /bootbank and /altbootbank partitions. When you install the ESXi hypervisor, the hypervisor image is installed in the /bootbank partition, which is a special 250MB partition that houses the ESXi image.
The bootbank partition contains the hypervisor image, which is decompressed at boot
ESXi is actually contained in a compressed image called s.v00. When the hypervisor boots, this special compressed package is decompressed and instantiated with the boot process. When you perform a clean install of ESXi, the /altbootbank partition is essentially blank, with only a boot.cfg file present.
Looking at the :altbootbank partition on a clean server build
Before the ESXi upgrade process replaces the image with the upgraded image, the contents of the /bootbank partition are copied to the /altbootbank partition. When you roll back your ESXi installation, the system copies the /altbootbank partition contents to the /bootbank location and, as we will see, actually copies the newer image to the /altbootbank location. The older image copied over to /bootbank is still the booted ESXi image location.
Altbootbank after an upgrade houses the previous ESXi hypervisor boot image version
It's easy to see which versions you have available without having to reboot your ESXi host and checking the rollback options. Use the following command to see the versions contained in the boot.cfg files on both partitions, SSH'ed into your ESXi host:
tail -2 /*bootbank/boot.cfg
Looking at the boot.cfg to see the versions contained for ESXi rollback
What are the requirements for rolling back to a previous version of ESXi? Let's take a look.
Requirements for ESXi rollback
There are a few requirements and things to keep in mind when deciding to roll your vSphere ESXi installation back to a previous version of the hypervisor. These are described below.
1) You need to have access to the Direct Console User Interface (DCUI) on the ESXi host that you want to roll back.
2) Make sure to get a configuration backup of your ESXi host. This is easily done using PowerCLI:
Get-VMHostFirmware -VMHost $host -BackupConfiguration -DestinationPath <your directory>
3) Reverting an ESXi host is only valid if the host was updated using one of these methods:
- VIB installation or removal
- Profile installation or removal
- ESXi host update using VMware Update Manager
- Updated from an ISO
4) Perform all the necessary housekeeping, including migrating any running VMs off the ESXi host
Once you have met the above requirements and you have verified you have a valid version of the ESXi hypervisor contained in the /altbootbank location, you should be good to roll back ESXi. Let's take a look at the process, which is very straightforward.
How to roll back and downgrade the ESXi version
The process to roll back and downgrade ESXi involves pressing the SHIFT+R key sequence when prompted while the ESXi host is booting. When you see the Loading VMware Hypervisor message, there is a short delay before the hypervisor image is decompressed. This is the moment to press the SHIFT+R key combination.
When ESXi is booting, press SHIFT+R to enter the rollback screen
This launches the VMware Hypervisor Recovery screen. It is a simple display with the current (Default) image listed as well as the alternate ESXi image. As you can see in the following screen, I have the following ESXi versions listed:
- 7.0-3.73.14320388 – Default
- 7.0-10302608 – Alternate
***Note*** – Rolling back your ESXi version is a permanent operation. In other words, you can't decide after the fact to return to the previous default ESXi image, which may be newer.
Press Y to roll back to the previous version in :altbootbank
Again, it is not possible to roll back to the previous default after initiating the rollback operation. If you reboot your ESXi host after the roll back operation, you will see the following message:
No alternate hypervisor to roll back to
After rolling back, SHIFT+R will show no alternate versions to roll back to
Interestingly, after rolling back ESXi, if you run the “tail” command to look at your boot.cfg in both locations, you will see that the newer image has been copied to /altbootbank, while the older image, which is now the default, is running in /bootbank.
After rolling back the ESXi version, :altbootbank contains the newer version and :bootbank contains the older version
Conclusion
The rollback option for ESXi presents a unique troubleshooting or testing tool to go back to a previous version of the ESXi hypervisor. There are definitely issues to consider before invoking this capability. As shown, you can use the command line to see your rollback options beforehand and before rebooting the host.
Additionally, ESXi merely swaps out the images that are located in the /bootbank and /altbootbank locations, so that the previous image replaces the newer image that resides in the /bootbank location. It is important to remember this is a one-way operation to roll back your ESXi image. Once you roll back, you cannot undo this operation, at least in a supported way.
Read the latest IT news and community updates!
Join our IT community and read articles without ads!
Do you want to write for 4sysops? We are looking for new authors.
