System Partition

vSphere 7 – ESXi System Storage Changes

Amit Arora vSphere 7.x , , ,

VMware changed the lay-out for ESXi system storage partitions on its boot device after a decade. This is done to be more flexible, and to support other VMware, and 3rd party solutions.

Prior to vSphere 7, the ESXi system storage lay-out had several limitations. The partition sizes were fixed and the partition numbers were static, limiting partition management. This effectively restricts the support for large modules, debugging functionality and possible third-party components.

They have increased the boot bank sizes, and consolidated the system partitions and made them expandable.

Partition Lay-out in vSphere 6.x

The partition sizes in vSphere 6.x are fixed, with an exception for the scratch partition and the optional VMFS datastore. These are created depending on the used boot media and its capacity.

Partition Lay-out in vSphere 6.x

Consolidated Partition Lay-out in vSphere 7

Consolidated Partition Lay-out in vSphere 7

The ESXi 7 System Storage lay-out only consists of four partitions.

  • System boot
    • Stores boot loader and EFI modules.
    • Type: FAT16
  • Boot-banks (x2)
    • System space to store ESXi boot modules
    • Type: FAT16
  • ESX-OSData
    • Acts as the unified location to store extra (nonboot) modules, system configuration and state, and system virtual machines
    • Type: VMFS-L
    • Should be created on high-endurance storage devices

The OSData partition is divided into two high-level categories of data called ROM-data and RAM-data.

Frequently written data, for example, logs, VMFS global traces, vSAN EPD and traces, and live databases are referred to as RAM-data.

ROM-data is data written infrequently, for example, VMtools ISOs, configurations, and core dumps.

ESXi 7 System Storage Sizes

Depending the boot media used, the capacity used for each partition varies. The only constant here is the system boot partition. If the boot media is larger than 128GB, a VMFS datastore is created automatically to use for storing virtual machine data.

System Partition Size in 7.x

NOTE: For storage media such as USB or SD devices, the ESX-OSData partition is created on a high-endurance storage device such as an HDD or SSD. When a secondary high-endurance storage device is not available, ESX-OSData is created on USB or SD devices, but this partition is used only to store ROM-data. RAM-data is stored on a RAM disk.

ESXi 7 System Storage Contents

The sub-systems that require access to the ESXi partitions, access these partitions using the symbolic links. For example: /bootbank and /altbootbank symbolic links are used for accessing the active bootbank and alternative bootbank. The /var/core symbolic link is used to access the core-dumps.

Review the System Storage Lay-out

When examining the partition details in the vSphere Client, you’ll notice the partition lay-out as described in the previous chapters. Use this information to review your boot media capacity and the automatic sizing as configured by the ESXi installer.

A similar view can be found in the CLI of an ESXi host. You’ll notice the partitions being labeled as BOOTBANK1/2 and OSDATA.

NOTE: When the OSDATA partition is placed on a SDD or NVMe device, VMFS-L is labeled as VFSS.

System Storage Requirements When Upgrading

The boot media requirements differ between a new vSphere 7 install, and an upgrade to vSphere 7.

There’s a requirement for boot media to run a 4 GB storage device at minimum, when upgrading to vSphere 7. 

To install fresh ESXi 7, the following boot media requirements must be met:

  • Boot media of at least 8GB for USB or SD devices
  • 32GB for other boot devices like hard disks, or flash media like SSD or NVMe devices.
  • A boot device must not be shared between ESXi hosts.

Upgrading to from ESXi 6.x to ESXi 7.0 requires a boot device that is a minimum of 4 GB.

If you install ESXi 7.0 on a very small system disk (for example 2 GB) you will notice this error from the installer:

Note: if you install ESXi 7 on a M.2 or other non-USB low-end flash media, beware that the storage device can be worn out quickly if you, for example, host a VM on the VMFS datastore on the same device. Be sure to delete the automatically configured VMFS datastore on the boot device when using low-end flash media. It is highly recommended to install ESXi on high-endurance flash media.

All the scenarios in this diagram are supported when upgrading to vSphere 7. Again, the recommended boot device would be a high endurance disk or flash device.

Upgrade Scenarios

But what happens from a ESXi system storage perspective if you have ESXi installed on a USB or SD device together with a HDD/SSD or you when you have a USB-only system?

Upgrade Scenario -1 : USB with HDD

When the storage media is a USB  or SD card, and the scratch partition is on HDD or SDD storage media, the upgrade process is as follows:

  1. Backup potential partner VIBs (kernel modules), contents of the active boot-bank, locker and scratch partitions to memory (RAM).
  2. Cleanup all system partitions, non-datastore partitions are not destroyed.
  3. If the upgrade media does not have a VMFS partition, create a GPT partition layout.
  4. Create partitions (book-banks and ESX-OSData)
    • The dedicated scratch partition is converted to the ESX-OSData partition
  5. Restore the contents from RAM to the appropriate partitions.

In this scenario, the scratch partition on the hard drive is converted to ESX-OSDATA. Its size is limited to 4 GB because of VFAT restrictions. This size might be too small for customers, who have large memory systems and require a large core dump file.

In this case, customers can take the following actions:

  • Create a core dump file on a datastore. To create a core dump file on a datastore, see the KB article 2077516.
  • Assign scratch to a directory in the datastore. To assign the scratch location to a directory in a datastore, see KB article 1033696.

Upgrade Scenario -2 : USB-only

Having a USB or SD-only device setup in your ESXi host, you can still upgrade to vSphere 7 if the storage device is at least 4 GB. Although a higher endurance and capacity device is strongly recommended. To support the 4GB minimum when upgrading to vSphere 7, there’s a couple of things happening with the storage partition layout.

Note: using vSAN in a cluster with ESXi hosts that have more than 512GB of memory require larger than 4 GB boot devices (when upgrading to vSphere 7) because of a larger core dump partition.

In the scenario of using a 4 GB boot device and no local disk is found, ESXi is running in a so-called ‘degraded mode‘. This means ESXi is not running in an optimal state, with some functionalities disabled. Also, running in a degraded state could mean ESXi loses its state on a power cycle. Solving this requires adding a local disk or flash device and run the instructions found in KB article 77009.

This diagram shows the typical setup when upgrading using USB boot media. This scenario is also applicable for a setup where the scratch location points to a datastore. For security reasons, ESX-OSData cannot exist in those locations. The upgrade steps using USB or SD media are:

  1. Request remediation of unavailable scratch volume:
    • The user is prompted to create a compatible scratch volume or add a spare disk. The upgrade process terminates if the user chooses to remediate.
  2. If remediation is ignored, then a fallback mode will be used:
    • ESX-OSData is created on USB.
      • USB flash MUST be >= 4GB otherwise upgrade will terminate because VMFS requires at least 1.3GB. This space is necessary for pre-allocation of the core file, vmtools and vSAN traces.
    • RAM-disk is used for frequently written data.
    • Subsystems that require persistence storage of data, implement an equivalent backup.sh capability to allow buffered saving of the data from RAM-disk to ESX-OSData
    • This is a highly degraded mode of operation with boot messages displaying so. The user must accept that there is potential for data to be lost because of the use of a RAM-disk which may be storing data that ESXi considers to be persistent across reboots.

The backup.sh script is run at regular intervals to save the system state and sticky bit files to the boot-bank.

First Boot Tasks

After the ESXi upgrade, or a new installation, the first boot tasks of the ESXi host are executed. These include:

  • Scan for unpartitioned disks and partition them as datastores.
  • Create symbolic links to file systems, ensuring that the necessary namespaces for subsystem are available.
  • Initialize subsystems to reference the correct storage location. For example, logs and core dump locations.