Connectivity issue when upgrading Dell R620 to ESXi 5.1 build 914609

When building a couple of new ESX hosts based on Dell R620 systems, I used the Dell customized iso VMware-VMvisor-Installer-5.1.0-799733.x86_64-Dell_Customized_RecoveryCD_A01.iso to install ESXi

Those Dell systems had 4 Broadcom nics (2 x 1Gb + 2 x 10Gb) and 2 Intel 10Gb nics

Install went fine, and I decided to upgrade to the latest patches using esxcli since the hosts had no access to vCenter. All went fine till after the reboot. I noticed all Broadcom nics where missing from my hosts, most likely due to a driver issue, so time to investigate.

Here is what a host looks like before I installed the update vib 

As you can see the hosts has 6 nics, four using the Broadcom bnx2x driver and two using the Intel ixgbe driver.

After the upgrade the host looks like this:

So my Broadcom nics where gone …

Lets have a look at the drivers before and after the upgrade. Before the upgrade, the Broadcom OEM drivers version 1.74.17.v50.1-1 where installed, which are part of the Dell iso:

After the upgrade, the driver version was 1.16.15.v50.3-1, and these where VMware drivers:

So apparently the “upgrade” downgrades the Broadcom bnx2x drivers from 1.74.17.v50.1-1 to 1.16.15.v50.3-1, and the 1.16.15.v50.3-1 did not work on the nics in the R620 …

Looking at the output of the vib install on an other host confirms this behavior:

As you can see, the 1.16.15.v50.3-1 drivers are installed and the 1.74.17.v50.1-1 drivers are removed.

Sow how to fix?

The good thing is I always try to use nics of different vendors in my hosts, to prevent catastrophic issues with drivers like these. So my hosts still had access to my iSCSI array trough one of the intel 10Gb nics, and I could access the console via the Dell iDRAC.

Looking at the vSphere Download page I found an Offline Bundle for the Broadcom drivers:

I was able to put in on a VMFS volume using the Datastore Browser on a host I had not upgraded yet, and install it trough the iDRAC on the broken hosts:

After a reboot my host was back online and connectivity was restored.

So what did I learn?

  • Do not trust the ESXi update procedure blindly. I did lots of upgrades in the past, but the process is not flawless and might install older non functioning drivers. If you relay want to be sure, use the vSphere ESXi Image Builder CLI to compare driver versions between the ISO and the update
  • Test the upgrade on a single host (which I usually do, but since this was a new build, I did some hosts in parallel)
  • Make sure you have access to the console or physical system when you do upgrades
  • It does not hurt to have nics from different vendors in a host. I always try to do that in my designs, but this time it saved me a trip to the Datacenter …

8 thoughts on “Connectivity issue when upgrading Dell R620 to ESXi 5.1 build 914609

  1. Nice article Duco. I’ve read about these issues before, specifically be careful with OEM VMware installation cd’s:)…like the Dell one, but same counts for other vendors.

  2. Great timing, I just ran into this issue yesterday on a new server build, and this article saved me a lot of troubleshooting time. In my case, it was the Broadcom driver that did the trick. Thanks for the write-up Duco!

  3. Try following command to update your ESXi. It’s safe.
    # esxcli software vib update -d

    If you enter “esxcli software vib install -d “, customised driver will removed.

Leave a Reply

Your email address will not be published. Required fields are marked *