printable format  e-mail to a friend More Archive Articles A Lab Simulation for CCIEs Building a Design Foundation Understanding WANs: A Technical Primer T1 and The Link Protocols That Use It Prepping for CID CCNA Cisco Certified Network Associate Test Yourself Practice Exams: Exam 640-507 CCNA Cisco Certified Network Associate Study Guide, Second Edition (Osborne/McGraw-Hill) CCNA Certification Kit CCNA 2.0 640-507 Routing and Switching Cheat Sheet Exam Cram Routing and Switching Audio Review

You probably already know that you want a loop-free topology in a bridging network in order for it to function properly. The Spanning Tree Protocol (STP) allows for physical redundancy and dynamically ensuring a loop-free topology. If a problem arises with the operation of this protocol, a bridging loop occurs and, as a result, a broadcast storm takes place. To the end user this causes snafus from intermittent slow response time to complete loss of connectivity.

This article describes my experiences in detecting these problems, shows some ways to prevent them in the first place, and explains what to do when they occur. All the examples are things I've seen on production routers in a large network.

-- advertisement (story continued below) --

I assume you've got a basic knowledge of switches, in general, and STP, in particular. Note: Since all switches are really multiport bridges, I refer to "bridge" when I'm discussing software and STP functions; I use the term "switch" when I refer to a physical device.

User-Visible Symptoms

How do you know when the problem has surfaced? If you're running IPX with a significant number of SAPs (services), you'll completely lose connectivity to hosts on the bridged network. You won't be able to ping anything, and users won't be able to access anything. Users with slower PCs may even have problems with non-network applications like spreadsheets. IPX SAP broadcasts occur every 60 seconds, consist of many packets, and will be issued by IPX servers as well as the routers. If you have dual homed servers this situation gets even worse; the servers will send the SAP they got on network card A out network card B and vice versa. Because IPX networks generate much more broadcast traffic in normal operation, the impact of a bridging loop is magnified.

In the case of IP-only networks, I've seen long response time, intermittent ping performance, and dropped sessions.

Router-based Indicators

I'm going to focus on router-based indicators because you can't assume that you'll be able to reach the switches over the network. While you can attach local consoles to the switches, in most cases you won't have on-site support with the required tools and access, and/or you won't want to wait. Also if you have TACACS or a similar security feature on the switches that require access to a central server, it will require a longer and more complex login process to even start looking at switches. You're not going to be able to reach the security server and will have to use "last resort" procedures.

Let's examine both log messages and the fields in the sho interface command that indicate a problem. Most of us will first check the log (sho log). We find two messages of interest there:

Aug 1 00:31:09: %SCHED-3-THRASHING: Process thrashing on watched queue 'IP Packets' (count 36).
-Process= "IP Input", ipl= 6, pid= 27
-Traceback= 6017D17C 6017D498 6024C1C0 60136C50 60136C3C

This message occurs when the router thinks it's spending too much time on a process. You'll more often se "IP ARP" than "IP Packets".
In the case of IP ARP, what's happening is that a system is broadcasting an ARP, the bridging loop causes it to appear multiple times, and the router is spending lots of time replying to the same ARP request hundreds of times.
On this particular network, Hot Standby Router Protocol (HSRP) is running. I suspect here that since the HSRP routers send a multicast hello every five seconds and the bridge loop causes the router to see its own hellos repeated multiple times, the IP Packets referenced in the message are HSRP hellos.

On IP-only networks with a bridging loop problem, I've always seen the SCHED-3-THRASHING message in the log.

Aug 1 00:31:06: %STANDBY-3-DUPADDR: Duplicate IP address
172.30.90.252 detected on FastEthernet4/1

This message seems to indicate a duplicate IP address, but two things are odd here. The message starts with STANDBY indicating that HSRP has the problem. Also the IP address isn't the HSRP virtual address, but the "real" address of the router interface. Also there weren't normal router duplicate IP address messages, just the ones from HSRP. What's happening here is that the router keeps seeing its own HSRP hello; it's decided that the reason for that is another router is misconfigured (or just confused). Now let's look at the router interface display:

DERFRTR1#sho int faste 4/1
FastEthernet4/1 is up, line protocol is up
Hardware is cyBus FastEthernet Interface, address is
0010.5447.4881 (bia 0010.5447.4881)
Description: 10/100 Switched Lan with Catalyst 5000 and other switches
Internet address is 172.25.20.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, rely 255/255, load 1/255
Encapsulation ARPA, loopback not set, keepalive set (10 sec), fdx, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters 23:10:15
Queuing strategy: fifo
Output queue 0/40, 0 drops; input queue 5/75, 110233 drops
5 minute input rate 1069375 bits/sec, 2000 packets/sec
5 minute output rate 1106000 bits/sec, 1032 packets/sec
870300 packets input, 9240200 bytes, 0 no buffer
Received 490600 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 watchdog, 0 multicast
0 input packets with dribble condition detected
696800 packets output, 91210052 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Notice several things here.

First, you may see an input queue. Normally (at least on my network), you don't see input queues on a Fast Ethernet interface.

Second, you see a very large number of drops on the input queue ("input queue 5/75, 110233 drops").

Third, look at the large number of input broadcast packets relative to the total packet number (490600 broadcasts vs. 870300 packets input). On IP-only networks I see one to two percent of total packets as broadcasts. On an IPX network with a bridging loop, I've seen broadcasts equal to 100 times the number of non-broadcast packets.

So What Do You Do?

Basically break the loop. If you've configured your network with an explicitly defined root and backup root bridge, then you can either telnet to the backup root and shut down all the ports or even turn it off. Or you can have ports physically unplugged in the backup root bridge. The point is to break the loop either by command or physical action.

It should be clear that even though you can use the router to detect the problem, there's nothing you can do at the router console to resolve it. The routers are layer 3 devices that don't take part in STP. If you have time (meaning: your users aren't in "angry mob" mode), you should visit each switch you can and gather data. We've been asked for the following commands. Note that these are Catalyst 5000 formats; if you have other switches, see the documentation for their equivalents.

sh ver
sh spantree
sh cdp neighbor
sh biga and sh mbuf (only available on some switches)
sho spanningtree 1/1 1/2 (the uplink ports on remote switches; the command isn't on all switches
sho cam dyn

Even if you don't have a bridging loop, if you configure your switches appropriately, you won't create any new problems by shutting down your redundant links; they're not forwarding anyway.

Proactive Detection

You have several helpers:

• Syslog server
• SNMP traps
• SNMP polls

Routers can be configured to send their log messages to a syslog server. You then run scripts or use other network management probes to pick up the messages described above and alert your operations folks.

Some switches can send SNMP traps when the spanning tree topology changes. This is a recent development (to us). The issue is that you'll get traps on all ports, even dedicated host ports. However, you can limit what ports you'll get traps on. I recommend you only trap on ports that connect to other switches and routers. You may not know when hosts are rebooted (causing a link down and up trap), so you only want traps on objects you control. However, I've never seen a bridging loop caused by a physical failure that created a trap, so beware of a false sense of security.

Design Considerations

Don't get me wrong; I'm not concerning myself with switched network design in this article. There are many factors to be considered besides those relating to Spanning Tree operation, but those are the only ones I'm considering here.
For the purposes of debugging, simplicity and determinism is good.
Cisco design recommendations encourage mesh routed networks; that goes double for switches. Each switch should connect to at most two in the next higher level.

Never connect peer switches except at the core. It makes for a simple STP topology.

Select one core bridge as the root bridge. Set its bridge priority to 10. Select another core switch to be the backup root bridge and set its bridge priority to 11. Leave the non-core switches at default values.

The reason for explicitly setting a backup root bridge is that depending on the models of the access layer switches, if the root fails, an access switch may become the new root bridge since it would have a higher MAC address. You don't want this. I suggest you set priorities in the core switches and then take defaults for everything else. This has two advantages. First, it's a simpler configuration. Second, since there are several command interfaces (depending on switch model), this allows for simpler configuration standards. I haven't implemented a switched network that had three layers of switching with redundant connections between layers 2 and 3, so I have no real-life experience in these configurations.

In general, you explicitly want to set what ports will be forwarding. If I had redundant connections to layer 2 bridges, I would set bridge priorities to 12, 13, etc.

If you run HSRP and use priority, set as the root bridge the one attached to HSRP "winner." If you don't use HSRP, set the root bridge to the most common default gateway.

Make sure you understand the switch features that modify STP timers or other behaviors before using them. For example, portfast bypasses the listening and learning phases of port startup. You'll want to enable this feature when you connect a system that does Microsoft networking directly to the switch. A Microsoft network machine does certain operations (WINS registration and DHCP requests) when the system first comes up. You don't want the switch to not forward these packets. But if you enable portfast then attach another switch to that port, you can create a spanning tree loop.

You have the option of limiting broadcast traffic. We haven't done this because we have a large number of switched networks, many with IPX, and we haven't arrived at a good number as a standard. Clearly, due to the nature of IPX broadcast traffic, a value for an IPX network would be too high for a pure IP network.

Remember, if you use ISL, you have one STP per virtual LANs (vLANs). If you use 802.1Q, you only have a single STP running for all vLANs. Even though you can have a different root bridge per ISL vLAN, I wouldn't do so due to the extra complexity.

Some people recommend using a separate vLAN for management--that is, configure all your IP address for the switches themselves on their own vLAN. Use vLAN 1 for that purpose. Put the user vLANs on other numbers. It's said that removing the management traffic off the user vLANs reduces the chance that spanning tree loops will occur. Also it's easier to manage a number of switches on vLANs if you have them on the same subnet.
You want to connect the same two ports in each access switch to the core switches. It's a good idea to use the first two ports in the access switch.

Also wise: Have as few access switches as possible; that is, it's better to have a single large access switch than four or five smaller ones. It makes for a simpler configuration. The less STP you have, the less likely you'll have problems. Also network management is easier. One argument against this, of course, is that more eggs are in fewer baskets. But having dual power supplies, supervisor cards, and on-site spare port cards can address those concerns.

The Normal Network

Let's assume you follow these rules and have a configuration like below. Note: There can be many access switches; they're all connected the same way; and since I take the STP defaults in the access switches, they all act the same.

                 !
                 !
           Access
     +---Switch---+
     !                      !
     !(2)                 !(3)
     !                      !
 Root         Backup Root
 Switch--------Switch
     !        (1)         !
     !                      !
 Router           Router      

I connect the core switches, and each access switch is connected to both the core switches. Each link is a LAN as far as STP is concerned, and thus each port on the three LANs will have a state of forwarding or blocking.

According to the STP rules, both ports on the Root Switch will be forwarding. The port on link (2) of the access switch will be forwarding since it's closest to the root; the port on link (1) on the backup root will be forwarding since it's closest to the root. The only question: What happens on links (3)? Since the backup root switch has a higher priority than the access switch, it forwards, and the port on the access switch will be blocked. As indicated before, this will be true on all ports from the access switches to the backup root bridges.

From a troubleshooting perspective a spanning tree loop means that the port from an access switch to a backup root bridge is in a forwarding state and not a blocking state. This is a highly redundant configuration, which means it may not be cost-effective for your environment. The point is to configure your network so you know where traffic is supposed to be flowing. You don't want to try to discover it in a problem situation.

Case Solved

Loops can form in any environment where you have redundant switch connections bridging. A number of indictors in the routers can tell you that you're having this problem. Some of these indicators can be acquired by network management systems to provide proactive warnings. Good network design will make bridging loops less likely to form in the first place, plus make it easier for you to diagnose and repair problems as they arise. Go to it!

Fred P. Baker, CCIE, MCSE+Internet, has worked in networking for over 20 years most with a major telephone company designing deploying and maintaining SNA and IP networks. He has worked with Cisco equipment since the AGS and IOS 8.2 days. He has had his CCIE since 1998. Fred can be reached at fpbaker@anet.com. You can contact Fred about "The Case of the Spanning Tree Problem" at editor@tcpmag.com.