Friday, August 18, 2023

Recovering a Corrupted NSX-T Manager

If your NSX-T Manager cluster is running as a cluster of VMs, if one is corrupted, there is a good chance they all are if the issue was related to storage connectivity. Or, maybe it is just one. If you are running a cluster, and don't have a backup to restore, these steps can be used to repair the file system. Mileage varies on repairing file systems, so there is no guarantee this will work, but this is the process to attempt nontheless.

1.  Connect to the console of the appliance.

2.  Reboot the system.

3.  When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again. Press e to edit the menu.

4.  Keep the cursor on the Ubuntu selection.

5.  Press e to edit the selected option.

6.  Enter the user name ( root) and the GRUB password for root (not the same as the appliance's user root).Password "VMware1" before release 3.2 and "NSX@VM!WaR10" 3.2 and beyond.

7.  Search for the line starting with "linux" having boot command.

8. Remove all options after root= (Starting from UUID) and add "rw single init=/bin/bash".

9.  Press Ctrl-X to boot.

10.  When the log messages stop, press Enter. You will see the prompt root@(none):/#.

11.  Run following commands to repair the file system.

  • e2fsck -y /dev/sda1
  • e2fsck -y /dev/sda2
  • e2fsck -y /dev/sda3
  • e2fsck -y /dev/mapper/nsx-config
  • e2fsck -y /dev/mapper/nsx-image
  • e2fsck -y /dev/mapper/nsx-var+log
  • e2fsck -y /dev/mapper/nsx-repository
  • e2fsck -y /dev/mapper/nsx-secondary

No comments:

Fixing Clustering and Disk Issues on an N+1 Morpheus CMP Cluster

I had performed an upgrade on Morpheus which I thought was fairly successful. I had some issues doing this upgrade on CentOS 7 because it wa...