Monthly Archives: March 2014

Upgrading Cisco 3850 Switch Stack, One Switch at a Time

So last time I upgraded a 3850 switch stack, I used the command:

3850-stack# software install file TFTP://$server/path/to/bin switch 1-3

So with the last part “switch 1-3”, I had the idea that maybe I could update one at a time. I did a little bit of research, but didn’t find anything solid on if it’s possible or not.

The biggest reason for wanting to do this is our virtual environment. We have two port channels, over multiple stack members, going to a UCS blade. My thought was that if I can reboot one switch at a time, I can keep access to the virtuals active.

So I decided that why not? I needed to do the upgrade due to a multicast bug on the IOS-XE 3.3.1 anyway 🙂

Image was downloaded and ready on the TFTP server. I ran a console cable to switch 3 of stack, ssh’d to the stack, and enabled “terminal monitoring”. I then issued the following command:

3850-stack# software install file TFTP://$server/path/to/bin switch 3

After downloading and unpacking the BIN, the switch went through a verification process. This process is to determine if the upgraded firmware will work with the current firmware. Well, that failed 😦

So I issued the same command with the “force” option, to skip the verification process:

3850-stack# software install file TFTP://$server/path/to/bin switch 3 force

Again, the BIN was downloaded, unpacked, and the verification step was skipped. So far so good.

Then came the reboot. The switch went down and came back up … ish. The console showed the normal boot process, but froze after a bit. Hitting enter and closing console terminal (putty for me) and opening again, still no response to hitting the enter key

Looking at the ssh session, I saw a couple things.

First, the stack was apparently in half-duplex (link between Switch 1 and 2 was loose). This means that when Switch 3 was removed from the stack, I also lost switch 2.

Second, as switch 3 was coming back up, I received an error indicating that the firmware was not compatible with the current stack’s firmware.

At this point, I essentially had a one switch stack! To add salt to the wound, the TFTP server was up linked to switch 3. Didn’t think that one through too well 🙂

Luckily, both my PC and the firewall were connected to switch 1. Phew!

I jumped on Cisco’s site and started the download for the new firmware. While this was downloading, I decided to get switch 2 back in the stack. If not, after current stack, just switch 1, was upgraded, switch 2 would still be on the old firmware. This means that switch 1 and 3 would form a stack, but switch 2 would have a non-compatible firmware version to join the stack.

I verified that the stack cable was indeed loose and tightened it down. Back to the ssh session, I see that switch 1 is seeing switch 2. After a couple minutes, switch 1 reported switch 2 as removed.

All this time, console access on switch 3 is still not responding.

After some thinking, I had an idea. Since switch 3 was how switch 1 saw switch 2 (due to the loose cable), maybe the stack is getting confused with how to see switch 2. So I physically removed switch 3 from the stack.

Back to the ssh session, I see messages regarding switch 2. After a couple more minutes, switch 2 is finally added back to the switch.

Right before switch 2 came back, the firmware download completed. I started up a local TFTP server and issued the following command:

3850-stack# software install file TFTP://$server/path/to/bin switch 1-2

Once again, the firmware was downloaded, unpacked, and verified. The switches then went down for a reboot. I hurried to the server room and plugged up the switch 3 stack ports. At this time, I also moved the console port to switch 1.

Back at my desk.

On the console session, I see switch 1 coming back up. After some time, the stack master election starts. Switch 3 becomes the master (longest up switch). After fully booted, I see that switch 3 is the master and all three switches are back in the stack!

So in conclusion, you must upgrade the whole switch stack at the same time. Upgrading less than the whole stack causes members to be removed due to non-compatible firmware versions