When building a couple of new ESX hosts based on Dell R620 systems, I used the Dell customized iso VMware-VMvisor-Installer-5.1.0-799733.x86_64-Dell_Customized_RecoveryCD_A01.iso to install ESXi
Those Dell systems had 4 Broadcom nics (2 x 1Gb + 2 x 10Gb) and 2 Intel 10Gb nics
Install went fine, and I decided to upgrade to the latest patches using esxcli since the hosts had no access to vCenter. All went fine till after the reboot. I noticed all Broadcom nics where missing from my hosts, most likely due to a driver issue, so time to investigate.
Here is what a host looks like before I installed the update vib ESXi510-201212001.zip
As you can see the hosts has 6 nics, four using the Broadcom bnx2x driver and two using the Intel ixgbe driver.
After the upgrade the host looks like this:
So my Broadcom nics where gone …
Lets have a look at the drivers before and after the upgrade. Before the upgrade, the Broadcom OEM drivers version 1.74.17.v50.1-1 where installed, which are part of the Dell iso:
So apparently the “upgrade” downgrades the Broadcom bnx2x drivers from 1.74.17.v50.1-1 to 1.16.15.v50.3-1, and the 1.16.15.v50.3-1 did not work on the nics in the R620 …
Looking at the output of the vib install on an other host confirms this behavior:
As you can see, the 1.16.15.v50.3-1 drivers are installed and the 1.74.17.v50.1-1 drivers are removed.
Sow how to fix?
The good thing is I always try to use nics of different vendors in my hosts, to prevent catastrophic issues with drivers like these. So my hosts still had access to my iSCSI array trough one of the intel 10Gb nics, and I could access the console via the Dell iDRAC.
Looking at the vSphere Download page I found an Offline Bundle for the Broadcom drivers: BCM-NetXtremeII-5.0-offline_bundle-940344.zip
I was able to put in on a VMFS volume using the Datastore Browser on a host I had not upgraded yet, and install it trough the iDRAC on the broken hosts:
After a reboot my host was back online and connectivity was restored.
So what did I learn?
- Do not trust the ESXi update procedure blindly. I did lots of upgrades in the past, but the process is not flawless and might install older non functioning drivers. If you relay want to be sure, use the vSphere ESXi Image Builder CLI to compare driver versions between the ISO and the update
- Test the upgrade on a single host (which I usually do, but since this was a new build, I did some hosts in parallel)
- Make sure you have access to the console or physical system when you do upgrades
- It does not hurt to have nics from different vendors in a host. I always try to do that in my designs, but this time it saved me a trip to the Datacenter …