Bootbank loads in /tmp

 

Bootbank loads in /tmp after reboot of ESXi 7.0 Update 1 host.

From the ESXi shell, you see the bootbank, altbootbank and scratch partitions directing to /tmp, that means that they are mounted to the in-memory image of ESXi. 

For example:

Cause

The storage-path-claim service claims the storage device ESXi is installed on during startup. If this takes to much time to claim it, then this will cause the bootbank / altbootbank image to become unavailable and the ESXi host reverts to ramdisk (/tmp).

Resolution

This issue has been fixed in VMware ESXi 7.0 Update 1c.

Workaround

To workaround this issue, delay the storage-path-claim service to allow ESXi to retrieve the correct bootbank image from its storage device.

Step 1 - Determine the required delay (in seconds)

  • On the ESXi shell or SSH, run the following command:

 grep storage-path-claim /var/log/sysboot.log

  • Note the time highlighted (hh:mm:ss) in the example log output below (log timestamps will vary)

[2020-10-23 22:51:03.218874 - 2020-10-23 22:51:33.436044] sysboot: storage-path-claim

  • Run the following command to determine when the volume has mounted
grep 'mounted.*rw' /var/run/log/vobd.log|tail -1
  • Note the timestamp in the log output (hh:mm:ss), example below:

2020-10-23T22:52:02.413Z: [vmfsCorrelator] 66257428us: [esx.audit.vmfs.volume.mounted] File system [LOCKER-12345678-12345678-abcd-1234567890abcd, ....] on volume 12345678-12345678-abcd-1234567890abcd has been mounted in 'rw' mode. The datastore is now accessible on this host.

  • Determine the delay (in seconds) required for the next set of instructions.

22:52:02 - 22:51:33 = 29 seconds

Note: it is recommended to use a value of 30 seconds or higher for the next steps. In rare cases this value has been observed to go up to 240 seconds (4 minutes).

Perform the following steps to apply the workaround.

Step 2 - Applying the workaround (via Putty login or direct console access to ESXi server if available),

 
 Reboot the ESXi host.
  •  During the pre-boot 5 seconds splash screen, press SHIFT+O to modify the boot option line.
  •  In the resulting screen, move to the end of the boot line.
  •  Add the following to the same line: 

devListStabilityCount=30

For example:
  • Press the enter key to resume boot.
  • Login to the ESXi shell or SSH after the system has rebooted
  • Run the following command:

ls -al /

The output will list the links for bootbank and altbootbank. Confirm the path shows as /vmfs/volumes/UUID for both.
Bootbank Corrected

Note: If the path remains as /tmp/, you must reboot the ESXi host and increase the devListStabilityCount value in the boot option line (see earlier steps).
Do not proceed further with the instructions until this has been corrected.

  • Once the bootbank / altbootbank directories are directed to /vmfs/volumes/UUID, navigate to the bootbank directory:

cd /bootbank/

  • Make a backup of the boot.cfg

cp boot.cfg boot.cfg.bak

  • Edit boot.cfg to add the following setting to the line beginning with kernelopt
devListStabilityCount= the value you set in the above boot option line
Before:
 
After:
  • Save the changes, and reboot the ESXi host to apply the workaround.

Menu UK