VMware Cloud Foundation 4

VMware Cloud Foundation

Let’s start with the full stack! Here are all of your must-have Cloud Foundation 4 resources:

VMware vSphere

vSphere is the compute component of Cloud Foundation stack. See what’s new with vSphere 7:

VMware VSAN

vSAN storage goes hand-in-hand with vSphere. Check out the exciting features of vSAN 7:

VMware NSX

NSX brings networking and security to Cloud Foundation. Learn about the advancements in NSX-T 3:

VMware vRealize

vRealize Cloud Management helps you manage it all, securely and consistently. Check out all the latest developments in this suite of products and see what they bring to Cloud Foundation:

Useful Links

Top 20 articles for vRealize Operations Manager

  1. How to reset the root password in vRealize Operations
  2. How to reset the admin password in vRealize Operations Manager
  3. Adding additional storage to a node in vRealize Operations
  4. vRealize Operations Manager Sizing Guidelines
  5. Configure a Certificate For Use With vRealize Operations
  6. /storage/log is full on vRealize Operations
  7. Replace expired internal certificate in vRealize Operations Manager 6.3 and later
  8. Upgrade to vRealize Operations Manager 8.0 fails due to the admin or root account password
  9. Upgrade to vRealize Operations Manager 8.0 hangs on step 4 of 9
  10. Shutdown and Startup sequence for a vRealize Operations Manager cluster
  11. Clearing the Alerts and Alarms Tables in vRealize Operations
  12. Upgrade to vRealize Operations Manager 8.x fails due to low space on /dev/sda
  13. Upgrade to vRealize Operations Manager 8.0 fails due to low space on /dev/sdc
  14. Continuous disk space alerts for /storage/archive Guest File System in vRealize Operations Manager
  15. Rebooting nodes in vRealize Operations Manager
  16. Minimum Collection User Permissions in vRealize Operations Manager 6.x and later
  17. How to take a Snapshot of vRealize Operations
  18. Change the IP Address of a vRealize Operations Manager Multi Node Deployment
  19. vRealize Operations Data Collection
  20. Reload the default certificate in vRealize Operations Manager

vSphere 7 – ESXi System Storage Changes

VMware changed the lay-out for ESXi system storage partitions on its boot device after a decade. This is done to be more flexible, and to support other VMware, and 3rd party solutions.

Prior to vSphere 7, the ESXi system storage lay-out had several limitations. The partition sizes were fixed and the partition numbers were static, limiting partition management. This effectively restricts the support for large modules, debugging functionality and possible third-party components.

They have increased the boot bank sizes, and consolidated the system partitions and made them expandable.

Partition Lay-out in vSphere 6.x

The partition sizes in vSphere 6.x are fixed, with an exception for the scratch partition and the optional VMFS datastore. These are created depending on the used boot media and its capacity.

Partition Lay-out in vSphere 6.x

Consolidated Partition Lay-out in vSphere 7

Consolidated Partition Lay-out in vSphere 7

The ESXi 7 System Storage lay-out only consists of four partitions.

  • System boot
    • Stores boot loader and EFI modules.
    • Type: FAT16
  • Boot-banks (x2)
    • System space to store ESXi boot modules
    • Type: FAT16
  • ESX-OSData
    • Acts as the unified location to store extra (nonboot) modules, system configuration and state, and system virtual machines
    • Type: VMFS-L
    • Should be created on high-endurance storage devices

The OSData partition is divided into two high-level categories of data called ROM-data and RAM-data.

Frequently written data, for example, logs, VMFS global traces, vSAN EPD and traces, and live databases are referred to as RAM-data.

ROM-data is data written infrequently, for example, VMtools ISOs, configurations, and core dumps.

ESXi 7 System Storage Sizes

Depending the boot media used, the capacity used for each partition varies. The only constant here is the system boot partition. If the boot media is larger than 128GB, a VMFS datastore is created automatically to use for storing virtual machine data.

System Partition Size in 7.x

NOTE: For storage media such as USB or SD devices, the ESX-OSData partition is created on a high-endurance storage device such as an HDD or SSD. When a secondary high-endurance storage device is not available, ESX-OSData is created on USB or SD devices, but this partition is used only to store ROM-data. RAM-data is stored on a RAM disk.

ESXi 7 System Storage Contents

The sub-systems that require access to the ESXi partitions, access these partitions using the symbolic links. For example: /bootbank and /altbootbank symbolic links are used for accessing the active bootbank and alternative bootbank. The /var/core symbolic link is used to access the core-dumps.

Review the System Storage Lay-out

When examining the partition details in the vSphere Client, you’ll notice the partition lay-out as described in the previous chapters. Use this information to review your boot media capacity and the automatic sizing as configured by the ESXi installer.

A similar view can be found in the CLI of an ESXi host. You’ll notice the partitions being labeled as BOOTBANK1/2 and OSDATA.

NOTE: When the OSDATA partition is placed on a SDD or NVMe device, VMFS-L is labeled as VFSS.

System Storage Requirements When Upgrading

The boot media requirements differ between a new vSphere 7 install, and an upgrade to vSphere 7.

There’s a requirement for boot media to run a 4 GB storage device at minimum, when upgrading to vSphere 7. 

To install fresh ESXi 7, the following boot media requirements must be met:

  • Boot media of at least 8GB for USB or SD devices
  • 32GB for other boot devices like hard disks, or flash media like SSD or NVMe devices.
  • A boot device must not be shared between ESXi hosts.

Upgrading to from ESXi 6.x to ESXi 7.0 requires a boot device that is a minimum of 4 GB.

If you install ESXi 7.0 on a very small system disk (for example 2 GB) you will notice this error from the installer:

Note: if you install ESXi 7 on a M.2 or other non-USB low-end flash media, beware that the storage device can be worn out quickly if you, for example, host a VM on the VMFS datastore on the same device. Be sure to delete the automatically configured VMFS datastore on the boot device when using low-end flash media. It is highly recommended to install ESXi on high-endurance flash media.

All the scenarios in this diagram are supported when upgrading to vSphere 7. Again, the recommended boot device would be a high endurance disk or flash device.

Upgrade Scenarios

But what happens from a ESXi system storage perspective if you have ESXi installed on a USB or SD device together with a HDD/SSD or you when you have a USB-only system?

Upgrade Scenario -1 : USB with HDD

When the storage media is a USB  or SD card, and the scratch partition is on HDD or SDD storage media, the upgrade process is as follows:

  1. Backup potential partner VIBs (kernel modules), contents of the active boot-bank, locker and scratch partitions to memory (RAM).
  2. Cleanup all system partitions, non-datastore partitions are not destroyed.
  3. If the upgrade media does not have a VMFS partition, create a GPT partition layout.
  4. Create partitions (book-banks and ESX-OSData)
    • The dedicated scratch partition is converted to the ESX-OSData partition
  5. Restore the contents from RAM to the appropriate partitions.

In this scenario, the scratch partition on the hard drive is converted to ESX-OSDATA. Its size is limited to 4 GB because of VFAT restrictions. This size might be too small for customers, who have large memory systems and require a large core dump file.

In this case, customers can take the following actions:

  • Create a core dump file on a datastore. To create a core dump file on a datastore, see the KB article 2077516.
  • Assign scratch to a directory in the datastore. To assign the scratch location to a directory in a datastore, see KB article 1033696.

Upgrade Scenario -2 : USB-only

Having a USB or SD-only device setup in your ESXi host, you can still upgrade to vSphere 7 if the storage device is at least 4 GB. Although a higher endurance and capacity device is strongly recommended. To support the 4GB minimum when upgrading to vSphere 7, there’s a couple of things happening with the storage partition layout.

Note: using vSAN in a cluster with ESXi hosts that have more than 512GB of memory require larger than 4 GB boot devices (when upgrading to vSphere 7) because of a larger core dump partition.

In the scenario of using a 4 GB boot device and no local disk is found, ESXi is running in a so-called ‘degraded mode‘. This means ESXi is not running in an optimal state, with some functionalities disabled. Also, running in a degraded state could mean ESXi loses its state on a power cycle. Solving this requires adding a local disk or flash device and run the instructions found in KB article 77009.

This diagram shows the typical setup when upgrading using USB boot media. This scenario is also applicable for a setup where the scratch location points to a datastore. For security reasons, ESX-OSData cannot exist in those locations. The upgrade steps using USB or SD media are:

  1. Request remediation of unavailable scratch volume:
    • The user is prompted to create a compatible scratch volume or add a spare disk. The upgrade process terminates if the user chooses to remediate.
  2. If remediation is ignored, then a fallback mode will be used:
    • ESX-OSData is created on USB.
      • USB flash MUST be >= 4GB otherwise upgrade will terminate because VMFS requires at least 1.3GB. This space is necessary for pre-allocation of the core file, vmtools and vSAN traces.
    • RAM-disk is used for frequently written data.
    • Subsystems that require persistence storage of data, implement an equivalent backup.sh capability to allow buffered saving of the data from RAM-disk to ESX-OSData
    • This is a highly degraded mode of operation with boot messages displaying so. The user must accept that there is potential for data to be lost because of the use of a RAM-disk which may be storing data that ESXi considers to be persistent across reboots.

The backup.sh script is run at regular intervals to save the system state and sticky bit files to the boot-bank.

First Boot Tasks

After the ESXi upgrade, or a new installation, the first boot tasks of the ESXi host are executed. These include:

  • Scan for unpartitioned disks and partition them as datastores.
  • Create symbolic links to file systems, ensuring that the necessary namespaces for subsystem are available.
  • Initialize subsystems to reference the correct storage location. For example, logs and core dump locations.

ESXTOP HANDY TOOL

                                                             Metrics and Thresholds

Display

Metric Threshold Explanation

CPU

%RDY

10

Overprovisioning of vCPUs, excessive usage of vSMP or a limit (check %MLMTD) has been set. See Jason’s explanation for vSMP VMs

CPU

%CSTP

3

Excessive usage of vSMP. Decrease number of vCPUs for this particular VM. This should lead to increased scheduling opportunities.

CPU

%SYS

20

The percentage of time spent by system services on behalf of the world. Most likely caused by high IO VM. Check other metrics and VM for possible root cause

CPU

%MLMTD

0

The percentage of time the vCPU was ready to run but deliberately wasn’t scheduled because that would violate the “CPU limit” settings. If larger than 0 the world is being throttled due to the limit on CPU.

CPU

%SWPWT

5

VM waiting on swapped pages to be read from disk. Possible cause: Memory over commitment.

MEM

MCTLSZ

1

If larger than 0 host is forcing VMs to inflate balloon driver to reclaim memory as host is overcommitted.

MEM

SWCUR

1

If larger than 0 host has swapped memory pages in the past. Possible cause: Over commitment.

MEM

SWR/s

1

If larger than 0 host is actively reading from swap(vswp). Possible cause: Excessive memory over commitment.

MEM

SWW/s

1

If larger than 0 host is actively writing to swap(vswp). Possible cause: Excessive memory over commitment.

MEM

CACHEUSD

0

If larger than 0 host has compressed memory. Possible cause: Memory over commitment.

MEM

ZIP/s

0

If larger than 0 host is actively compressing memory. Possible cause: Memory over commitment.

MEM

UNZIP/s

0

If larger than 0 host has accessing compressed memory. Possible cause: Previously host was overcommitted on memory.

MEM

N%L

80

If less than 80 VM experiences poor NUMA locality. If a VM has a memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM and “remotely” uses memory via “interconnect”. Check “GST_ND(X)” to find out which NUMA nodes are used.

NETWORK

%DRPTX

1

Dropped packets transmitted, hardware overworked. Possible cause: very high network utilization

NETWORK

%DRPRX

1

Dropped packets received, hardware overworked. Possible cause: very high network utilization

DISK

GAVG

25

Look at “DAVG” and “KAVG” as the sum of both is GAVG.

DISK

DAVG

25

Disk latency most likely to be caused by array.

DISK

KAVG

2

Disk latency caused by the VMkernel, high KAVG usually means queuing. Check “QUED”.

DISK

QUED

1

Queue maxed out. Possibly queue depth set to low. Check with array vendor for optimal queue depth value.

DISK

ABRTS/s

1

Aborts issued by guest(VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason.

DISK

RESETS/s

1

The number of commands reset per second.
DISK CONS/s 20

SCSI Reservation Conflicts per second. If many SCSI Reservation Conflicts occur performance could be degraded due to the lock on the VMFS.

 

 

Conversion of Dynamic Disk to Basic Disk by VMware Converter Tool.

First Power off your Source Virtual Machine and open VMware Converter tool.

Select Radio button on Source Type “Powered Off ” & choose VMware Infrastructure Virtual Machine in drop down list.

Mention your Source vCenter/Host name to connect it.

C1

Select the location of destination server in left pane and server name in right and click on next.

Select the Destination vCenter/Host where new VM needs to be presented and click next.

Enter the new machine name which should be different from original one in destination Virtual machine wizard and click next.

Change the hardware version if you want and click next.

Click Edit on Data to Copy section.

C2

Now choose “Select Volumes to Copy” in drop down list of Data Copy Type.

C3

You can convert your Virtual disk from Thick to Thin as by default it select Thick type and click next.

Review the wizard and finish it.

Once conversion will be completed you can power on new VM

You will see new VM will have all Basic disk.

 

Cannot add VMFS datastore to ESXi host

ISSUE

Keep existing signature” and “Assign a new signature” is grayed  out and only option is “Format the disk” is available while adding a VMFS Datastore to ESXi host in Cluster.

CAUSE

This issue is related to Non-Persistent Mount Volume between hosts in Cluster.

SOLUTION

First find the volume which are Non-Persistent by running below command on ESXi Host.

#esxcfg-volume -l

By running this you can find the volumes and names.

To persistently mount the volume use, “esxcfg-volume -M” followed by either the UUID or the name of the volume

#esxcfg-volume -M TEMPDATASTORE

After running above commands then rescan storage on hosts and you will see the datastore listed.

Find VM Snapshots and Space They are Consuming with Creation Date

Snapshots are one of the best features of virtualizing a server. Snapshots have saved my bacon countless times. It’s easy to get used to simply right-clicking on a VM, creating a snapshot and forgetting about it.
However, this can cause a problem by consuming lots of storage if not kept in check.
When a snapshot is created it does a couple things; it creates a differencing disk to begin to accumulate all changes since the snapshot time and, if a snapshot is made while the VM is running, it commits the memory of that VM to disk.
Let’s find out how we can use PowerShell to build a report to figure out just how much space each checkpoint is consuming.
Luckily, finding out how much space a snapshot is consuming is pretty easy. It’s just a property that’s returned whenever you run Get-Snapshot. All you need to is run below command in PowerCLI to enumerate all of the VMs, pass that to Get-Snapshot and, for this case, I wanted to see the VM name Snapshot size in GB in decimal value 2 with Snapshot Creation Date.

Connect-VIServer “vCenter Server Name” -User “User Name” -Password “password”
Get-VM | Get-Snapshot | select VM,@{Name=’SizeGB’;Expression={[math]::Round($_.SizeGB,2)}}, Created |Export-Csv d:\temp\list.csv
Disconnect-VIServer -confirm A

Exit

Backup of & Restore of ESXi Configuration

Using the vSphere CLI

To back up the configuration data for an ESXi host using the vSphere CLI, run this command:

vicfg-cfgbackup --server=ESXi_host_IP_address --username=root -s
output_file_name


If you are using vSphere CLI for Windows, run this command:

vicfg-cfgbackup.pl –server=ESXi_host_IP_address –username=root -s output_file_name

Where ESXi_host_IP_address is the IP address of the ESXi host and output_file_name is the name of the backup file you create.

Note: From vSphere CLI for Windows, ensure you are executing the command from C:\Program Files\VMware\VMware vSphere CLI\bin

For example:
vSphere CLI:

vicfg-cfgbackup --server=10.0.0.1 --username=root -s ESXi_test1_backup.tgz

vSphere CLI for Windows:

vicfg-cfgbackup.pl –server=10.0.0.1 –username=root -s ESXi_test1_backup.tgz

Note: Use the --password=root_password option (where root_password is the root password for the host) to avoid being prompted for the root user password when you run the script.
A backup text file is saved in the current working directory where you run the vicfg-cfgbackup script. You can also specify a full output path for the file.

Using the vSphere PowerCLI

To back up the configuration data for an ESXi host using the vSphere PowerCLI, run this command:

Get-VMHostFirmware -VMHost ESXi_host_IP_address -BackupConfiguration -DestinationPath output_directory

Where ESXi_host_IP_address is the IP address of the ESXi host and output_directory is the name of the directory where the output file will be created.

For example:

Get-VMHostFirmware -VMHost 10.0.0.1 -BackupConfiguration -DestinationPath C:\Downloads

Note: A backup file is saved in the directory specified with the -DestinationPath option.

Using the ESXi Command Line

To synchronize the configuration changed with persistent storage, run this command:

vim-cmd hostsvc/firmware/sync_config

To backup the configuration data for an ESXi host, run this command:
vim-cmd hostsvc/firmware/backup_config
Note: The command should output a URL in which a web browser may be used to download the file. The backup file is located in the/scratch/downloads directory as configBundle-HostFQDN.tgz

Restoring ESXi host configuration data

Using the vSphere CLI

Note: When restoring configuration data, the build number of the host must match the build number of the host that created the backup file. Use the -f option (force) to override this requirement.

To restore the configuration data for an ESXi host using the vSphere CLI:

  1. Power off all virtual machines that are running on the host that you want to restore.
  2. Log in to a server where the vCLI is installed.
  3. Run the vicfg-cfgbackup script with the -l flag to load the host configuration from the specified backup file:

    vSphere CLI:

    vicfg-cfgbackup --server=ESXi_host_IP_address --username=root -l backup_file

    vSphere CLI for Windows:

    vicfg-cfgbackup.pl –server=ESXi_host_IP_address –username=root -l backup_file

    Where ESXi_host_IP_address is the IP address of the ESXi host and backup_file is the name of the backup file to use for the restore.

    For example:

    vicfg-cfgbackup --server=10.0.0.1 --username=root -l ESXi_test1_backup.txt

    vSphere CLI for Windows:

    vicfg-cfgbackup.pl –server=10.0.0.1 –username=root -l ESXi_test1_backup.txt

    Notes:

    • When you run this command, you are prompted for confirmation before proceeding. You can override this safety feature using the -q option.
    • Use the --password=root_password option (where root_password is the root password for the host) to avoid being prompted for the root user password when you run the script.
To restore an ESXi host to the stock configuration settings, run the command:

vicfg-cfgbackup –server=ESXi_host_IP_address –username=root -r

For example:

vicfg-cfgbackup --server=10.0.0.1 --username=root -r

Using the vSphere PowerCLI

Note: When restoring configuration data, the build number of the host must match the build number of the host that created the backup file. Use the -force option to override this requirement.

  1. Put the host into maintenance mode by running the command:

    Set-VMHost -VMHost ESXi_host_IP_address -State ‘Maintenance’

    Where ESXi_host_IP_address is the IP address of the ESXi host.

  2. Restore the configuration from the backup bundle by running the command:

    Set-VMHostFirmware -VMHost ESXi_host_IP_address -Restore -SourcePath backup_file -HostUser username -HostPassword password

    Where ESXi_host_IP_address is the IP address of the ESXi host, backup_file is the name of the backup bundle to use for the restore, and username and password are the credentials to use when authenticating with the host.

    For example:

    Set-VMHostFirmware -VMHost 10.0.0.1 -Restore -SourcePath c:\bundleToRestore.tgz -HostUser root -HostPassword exampleRootPassword

Using the ESXi Command Line:

Note: When restoring configuration data, the build number of the host must match the build number of the host that created the backup file.

  1. Put the host into maintenance mode by running the command:

    vim-cmd hostsvc/maintenance_mode_enter

  2. Copy the backup configuration file to a location accessible by the host and run the command:

    In this case, the configuration file was copied to the host’s /tmp directory. For more information, see Using SCP to copy files to or from an ESX host (1918).

vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz
 
Note: Executing this command will initiate an automatic reboot of the host after command completion.