Upgrading Cisco 3850 Switch Stack, One Switch at a Time

So last time I upgraded a 3850 switch stack, I used the command:

3850-stack# software install file TFTP://$server/path/to/bin switch 1-3

So with the last part “switch 1-3”, I had the idea that maybe I could update one at a time. I did a little bit of research, but didn’t find anything solid on if it’s possible or not.

The biggest reason for wanting to do this is our virtual environment. We have two port channels, over multiple stack members, going to a UCS blade. My thought was that if I can reboot one switch at a time, I can keep access to the virtuals active.

So I decided that why not? I needed to do the upgrade due to a multicast bug on the IOS-XE 3.3.1 anyway 🙂

Image was downloaded and ready on the TFTP server. I ran a console cable to switch 3 of stack, ssh’d to the stack, and enabled “terminal monitoring”. I then issued the following command:

3850-stack# software install file TFTP://$server/path/to/bin switch 3

After downloading and unpacking the BIN, the switch went through a verification process. This process is to determine if the upgraded firmware will work with the current firmware. Well, that failed 😦

So I issued the same command with the “force” option, to skip the verification process:

3850-stack# software install file TFTP://$server/path/to/bin switch 3 force

Again, the BIN was downloaded, unpacked, and the verification step was skipped. So far so good.

Then came the reboot. The switch went down and came back up … ish. The console showed the normal boot process, but froze after a bit. Hitting enter and closing console terminal (putty for me) and opening again, still no response to hitting the enter key

Looking at the ssh session, I saw a couple things.

First, the stack was apparently in half-duplex (link between Switch 1 and 2 was loose). This means that when Switch 3 was removed from the stack, I also lost switch 2.

Second, as switch 3 was coming back up, I received an error indicating that the firmware was not compatible with the current stack’s firmware.

At this point, I essentially had a one switch stack! To add salt to the wound, the TFTP server was up linked to switch 3. Didn’t think that one through too well 🙂

Luckily, both my PC and the firewall were connected to switch 1. Phew!

I jumped on Cisco’s site and started the download for the new firmware. While this was downloading, I decided to get switch 2 back in the stack. If not, after current stack, just switch 1, was upgraded, switch 2 would still be on the old firmware. This means that switch 1 and 3 would form a stack, but switch 2 would have a non-compatible firmware version to join the stack.

I verified that the stack cable was indeed loose and tightened it down. Back to the ssh session, I see that switch 1 is seeing switch 2. After a couple minutes, switch 1 reported switch 2 as removed.

All this time, console access on switch 3 is still not responding.

After some thinking, I had an idea. Since switch 3 was how switch 1 saw switch 2 (due to the loose cable), maybe the stack is getting confused with how to see switch 2. So I physically removed switch 3 from the stack.

Back to the ssh session, I see messages regarding switch 2. After a couple more minutes, switch 2 is finally added back to the switch.

Right before switch 2 came back, the firmware download completed. I started up a local TFTP server and issued the following command:

3850-stack# software install file TFTP://$server/path/to/bin switch 1-2

Once again, the firmware was downloaded, unpacked, and verified. The switches then went down for a reboot. I hurried to the server room and plugged up the switch 3 stack ports. At this time, I also moved the console port to switch 1.

Back at my desk.

On the console session, I see switch 1 coming back up. After some time, the stack master election starts. Switch 3 becomes the master (longest up switch). After fully booted, I see that switch 3 is the master and all three switches are back in the stack!

So in conclusion, you must upgrade the whole switch stack at the same time. Upgrading less than the whole stack causes members to be removed due to non-compatible firmware versions

Adding Custom Ringtone to Cisco Phones

I received a request today to add a custom ringtone to our call manager. After doing some research and testing, I was finally able to get the custom ring uploaded. The process is fairly straight forward, with the part taking the longest was figuring out the conversion process.

The basic steps are:

  • get mp3
  • get audacity with lame plugin
  • import
  • change 8khz
  • speed up until timing seems right
  • export > other > ulaw > raw (headerless)
  • upload tftp
  • modify Ringline.xml (first letter must be capital)
  • restart tftp service
  • choose new ringtone

I’m not 100% sure on the actual ringtone lengths, but here’s a snippet from Cisco’s site on the PCM file:

  • Raw PCM (no header)
  • 8000 samples per second
  • 8 bits per sample
  • mu-law compression
  • Maximum ring size—16080 samples
  • Minimum ring size—240 samples
  • Number of samples in the ring evenly divisible by 240
  • Ring starts and ends at the zero crossing.
  • To create PCM files for custom phone rings, you can use any standard audio editing packages that support these file format requirements.

Nonetheless, I just used Audacity with the Lame plugin to import the MP3 file. You need to change the sample rate to 8KHz (8000Hz) of the track. To do this, look to the left side of the track in question. You’ll see a small downward facing arrow to the right of the truncated filename. Right-click here > Set Rate > 8000Hz.

Now you’ll notice that the ringtone length may have increased. If so, you’ll want to speed up the track until it’s where you’d like it (I found that if going from 44100Hz to 8KHz, I had to speed up about 500%). Effect > Change Speed…

Once you get it where you want it, you’ll need to export the file as RAW (no headers) and u-law. File > Export. In the filetype drop down, choose “Other uncompressed files”. Next, click on “Options…”. Header needs to be “RAW (header-less)” and Encoding needs to be “U-Law”. Save as the file name you want. Next, you need to upload to the TFTP server in the root directory.

Next, you need to modify the current Ringline.xml file on the TFTP server. The format of this XML file can be found here, and it is:

<DisplayName>Analog Synth 1</DisplayName>
<DisplayName>Analog Synth 2</DisplayName>

The order you place the ringtones in this list, is the order it is displayed on the phones when a user wants to choose their ringtone. Once modified with your settings, you’ll need to upload this to the TFTP server. If you are using the CUCM as your TFTP server, you’ll need to restart the service (Goto Serviceability > Tools > Service Features > Restart TFTP).

*NOTE: At least with CUCM TFTP server, the server filename architecture is case sensitive. Make sure your XML file syntax matches the case in the example and you name the file “Ringline.xml” (notice capital “R”; this got me).

Finally, when you go to choose the ringtone on a phone, it should see the list populated with the naming scheme you chose.

Static IP and Dynamic IP Point-to-Point VPNs

I used to use a router for each P2P VPN tunnel before I realized that these can be kept on one router. So I have move all of our P2P VPNs (besides one) to a Cisco ISR 2811 and any new VPNs are terminated here. So recently, I was tasked with getting a Home user a constant connection to allow their hard phone to connect to our internal 3com NBX. The phones use a network connection to talk to the NBX through the data link layer (layer 2) by default, but can be programmed to communicate through IP (layer 3). So a constant connection that uses Layer 3 and is external pretty much says VPN tunnel all over it, and I want to keep the same design of using one end point to terminate the VPN tunnels.

I’ve setup many a VPN tunnel for clients and a few to our call centers through the web, but never a P2P to a home user. The problem here is that the home user may not always come from the same IP address. So I need to find a way to allow connections from any IP address using a specific key, while allowing connections from specific IPs using their own keys. Searching for this I found a few Cisco sites using ISAKMP profiles. I configured this and got it to work for the home P2P tunnel, but the other client P2P tunnels failed. The reason for this is that profiles take precedence over ISAKMP policies and the profile is setup to accept connections from Therefore, the client P2P VPNs would try to match their key to the key setup for the home P2P, which would obviously fail. Did some more looking and profiles are good for allowing more than one type of authentication, where here I am only using one type of authentication, pre-shared keys (PSK).

Nonetheless, after a lot of trial and error and googling I finally found the one command I was looking for: crypto isakmp key $key address [b][/b]

Apparently, the term for this is called “wildcard keys”, where any end point meeting the Phase1 parameters with the proper key can create a VPN tunnel. Using this allowed both the current P2P VPN sessions and the home session to connect at the same time. If you want some more info and bit more of the walk through of my through process you can have a look see at the networking-forum.com post: . Also, here’s the final code that I used to get this connection working on my side:

crypto isakmp policy 200
encr 3des
authentication pre-share
group 2
crypto isakmp policy 900
encr aes
authentication pre-share
group 2
lifetime 28800
crypto isakmp key $KEY address 65.x.x.x
crypto isakmp key $KEY address 113.x.x.x
crypto isakmp key $KEY address 190.x.x.x
crypto isakmp key $KEY address
crypto isakmp keepalive 20 periodic
crypto ipsec transform-set esp-3des-sha esp-3des esp-sha-hmac
crypto ipsec transform-set esp-3des-md5 esp-3des esp-md5-hmac
crypto ipsec transform-set esp-aes-sha esp-aes esp-sha-hmac
crypto dynamic-map ZachHome 1000
set transform-set esp-3des-sha
crypto map external_ipsec 200 ipsec-isakmp
set peer 65.x.x.x
set transform-set esp-3des-sha esp-3des-md5
match address
crypto map external_ipsec 900 ipsec-isakmp
set peer 113.x.x.x
set transform-set esp-aes-sha
match address
crypto map external_ipsec 65000 ipsec-isakmp
set peer 190.x.x.x
set transform-set esp-3des-sha
match address
crypto map external_ipsec 65500 ipsec-isakmp dynamic ZachHome
interface FastEthernet0/0
ip address
ip access-group inside_allowed_in in
ip access-group outbound_to_inside out
ip nat inside
interface FastEthernet0/1
ip address 64.x.x.x
ip access-group inbound_from_outside in
ip nat outside
crypto map external_ipsec

Digi Etherlite 32 As A Terminal Server

So building my lab at work, I have read many posts on forums about others using a terminal server. Namely they use a Cisco 2500 series router with an async module.

Since we don’t have any spares of this lying around at work and my place of employment will only spend money on *necessary* equipment that will keep production going, I had to look for another solution. So I started doing some looking and found these “network serial concentrators” (http://www.digi.com) that sounded like they would work. From my understandings, we used these at terminals to allow users to connect to our call manager to allow them to make calls.

I started looking into them and they sounded like something that would work for my situation. I jumped on google to see if anyone else has used these to connect to the console ports of Cisco gear previously, but the best I could find was the pinout. When using T568B as the wiring scheme for the Digi side the wiring scheme will look like:

Digi Cisco Pinout

Now that we know the pinout of the cable connections, we now need to find how to administer this box. From looking at the box, the only connections it has are the 32 serial ports and one “10BASE-T” uplink port:

Digi Etherlite 32

So that means no console connections :(. Looking through Digi’s documentation, looks like I need to download two packages: one to re-ip the box and one add virtual com ports to my machine to connect to the Digi’s serial ports.

*NOTE: Prior to downloading and running either of these applications, you will need to find the MAC address of the Digi. Normally it will be labeled on the bottom of the device; however, if it is not (which is my case), just plug it into a managed switch and find the MAC address found on the port.

The first program can be found here. Just unzip it and run dgipserv.exe and follow the on-screen instructions.

*NOTE: It is required to unplug the Digi’s power and *only* resupply the power when the program requires it. This is because the Digi will only request an IP address when it first receives power.

Once that is completed and you can ping the IP of the digi from your machine, you are now ready to install the digi software. I use Windows at work, so here is the link for the windows installer. Digi also has installs for other operating systems. It is pretty straight forward from here, at least for the windows setup.

Once the install completes and you have everything cabled up, you will need to just open up your favorite serial terminal (putty for me) and open the com port associated with your device.


I didn’t see much documentation in ways of using a Digi Etherlite product in the manner that I am, so I thought that writing a blog about it and documenting it would do good for others :).

If the download links for the Digi applications no longer work, I have them on my dropbox account:

New Blog

If you made it here than you are now a witness to my new blog. I will be messing with the themes, categories and widgets, so come back often to see any updates. I will also try to be more active on this site as opposed the the former site.