The Chaos Drill: How I Rebuilt a Production Hypervisor from Bare Metal in 28 Minutes

There is an unwritten law in enterprise infrastructure: Nobody cares about your backups; they only care about your restores. Almost every systems engineer has a backup dashboard glowing with reassuring green checkmarks. But until you have physically unplugged a production server, wiped its boot media, and timed yourself restoring the environment while a stopwatch ticks, your disaster recovery plan is nothing more than theoretical fiction. Last weekend, I ran a live Chaos Drill to rebuild our primary enterprise hypervisor from bare metal.

The goal was simple and unforgiving: simulate total boot disk failure on our Dell PowerEdge R710 enterprise virtualization host, install the OS from scratch over the network, re-attach our SAN storage fabric, and have all virtual machines operational in under 30 minutes. Here is the minute-by-minute playbook of how it went down and the exact architecture that made it possible.

1. The Architectural Golden Rule: Decouple Compute from Storage

Before touching the server, understanding the architectural prerequisite is critical. If your virtual machine virtual hard disks (VHDs) live on local hypervisor boot drives, a drive failure turns into a multi-hour data recovery ordeal restoring hundreds of gigabytes over the network.

The Core Principle: Hypervisor hosts are disposable compute nodes. All persistent VM storage must live off-host on an enterprise SAN or clustered storage pool. When compute dies, your data remains untouched.

In our lab, the Dell PowerEdge runs XCP-ng. Its local drives contain nothing except the base hypervisor operating system and kernel logs. All VM storage repositories live on our Dell EqualLogic PS6100 SAN, connected over dual 10GbE bonded iSCSI fabrics on dedicated NET-STORAGE-VLAN99.

2. The Clock Starts: Minute 0 to 8 (Network PXE Re-image)

At T-00:00, I pulled the server’s drive sled, inserted a wiped drive, and powered on the chassis. I did not reach for a USB thumb drive. Instead, the machine utilized our enterprise PXE environment:

1. iDRAC & Network Boot: The PowerEdge booted via UEFI Network PXE, querying our UniFi Dream Machine DHCP server on NET-MGMT-VLAN1.
2. DHCP Option 66 & 67: DHCP handed out Option 66 (10.1.20.10 pointing to HV2) and Option 67 (iventoy_loader_16000_uefi).
3. Automated Kickstart: The iVentoy menu streamed the XCP-ng network installer ISO. With an answer file mounted via HTTP, the installer automatically partitioned the local drive, assigned the static management IP (10.1.20.14), and initiated reboot by T-07:45.

3. Minute 8 to 17: Multi-Path iSCSI Re-attachment

At T-08:30, the base hypervisor was booted and listening for SSH. Now came the crucial step: reconnecting the storage fabric without corrupting existing LUNs.

First, I configured the dedicated storage VLAN interface and restored the host’s authorized iSCSI Qualified Name (IQN), which I retrieved directly from our central Vaultwarden instance via Bitwarden CLI:

# Set the hypervisor IQN to match the SAN ACL
xe host-set-iscsi-iqn host-uuid= \
  iqn=iqn.2024-09.com.ittechstl:xcpng-r710-01

# Configure VLAN 99 on the 10GbE dual-port interface
xe pif-reconfigure-ip uuid= \
  mode=static IP=10.1.99.14 netmask=255.255.255.0

Next, rather than creating a new storage repository, I executed an sr-probe against the EqualLogic discovery IP to find the existing storage pool and introduced it back into the hypervisor:

# Discover LUNs on the EqualLogic SAN
xe sr-probe type=lvmoiscsi \
  device-config:target=10.1.99.100 \
  device-config:targetIQN=iqn.2001-05.com.equallogic:0-8a0906-xxxx-equallogic-pool

# Re-introduce the existing Storage Repository without formatting
xe sr-introduce uuid=a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d \
  type=lvmoiscsi \
  name-label="EqualLogic-SAN-StoragePool" \
  content-type=user

By T-16:20, the multi-path iSCSI fabric established four active active-optimized sessions across both Dell controller modules. The 1.8TB storage repository mounted instantly. Every single virtual disk was intact—zero gigabytes had to be copied over the wire.

4. Minute 17 to 25: VM Metadata Restoration

While the virtual disks were present on the SAN, the fresh hypervisor OS didn’t know which VHD belonged to which virtual machine. This is where automated metadata backups shine:

# Restore XCP-ng pool database from automated TrueNAS backup
xe pool-restore-database file-name=/mnt/backups/pool_metadata_backup.bak

The metadata restore re-populated the complete pool inventory: VM CPU topologies, RAM limits, MAC addresses, and network bridge assignments. At T-24:10, the entire fleet of virtual machines reappeared in Xen Orchestra.

5. Minute 25 to 28: Service Verification

I fired off the sequential startup script:

1. Core Services First: Primary Active Directory Domain Controller (AD1) and internal DNS.
2. Storage & Monitoring: TrueNAS SCALE backup VM and Zabbix proxy.
3. Application Tier: Production container hosts and web proxies.

At T-28:14, the final Zabbix trigger cleared. All 42 synthetic HTTP web checks, ping latencies, and service daemons flipped from warning yellow back to solid green. Stopwatch stopped: 28 minutes and 14 seconds.

6. Lessons Learned from the Drill

Two Crucial Lessons:
1. Never host your recovery tools on the system you need to recover: If our iVentoy PXE server or Vaultwarden keystore had lived as VMs on this PowerEdge host, we would have been completely dead in the water. They live on independent nodes.
2. Document Initiator IQNs: SANs authenticate hypervisors by IQN. If you install a clean OS, it generates a random IQN string that the SAN rejects. Storing our production IQN in Vaultwarden saved at least 15 minutes of troubleshooting access control lists.

Running real disaster recovery drills hurts. It elevates your heart rate and forces you to confront the missing steps in your documentation. But when a real drive fails at 2:00 AM on a Tuesday, you won’t be guessing—you’ll be executing a proven 28-minute runbook.