Chapter 04: Maintaining and Troubleshooting Campus Switched Solutions (Part01)

Ethernet LAN switching is a commonly used technology in current enterprise networks. Layer 2 and Layer 3 switching are a major part of campus networks, and they can also be found in data centers and some WAN solutions. It is therefore essential to have a good understanding of campus switching technologies, such as virtual local-area networks (VLANs), trunks, Spanning Tree Protocol (STP), multilayer switching, and first-hop redundancy protocols. It is also important for a network engineer to be able to diagnose and resolve problems associated with these technologies. This chapter reviews prominent campus multilayer switching technologies and focuses on resolving problems related to these technologies.

Troubleshooting VLANs

Switched Ethernet has been the dominant LAN technology for more than a decade. This is why a good understanding of the processes involved in Layer 2 switching is essential to any engineer who is involved in network troubleshooting. VLAN-based switched infrastructures are at the core of every campus network and being able to diagnose and resolve Layer 2 switching problems in these environments is a fundamental skill that any network engineer should have. This section first reviews the Layer 2 switching process and the associated switch data structures. Next, you will learn how to gather information from these data structures using Cisco IOS Software commands. Finally, you will learn how to interpret and analyze the gathered information in order to verify the proper operation of the Layer 2 switching process or to pinpoint and resolve problems.

LAN Switch Operation

As a network engineer, you must have an in-depth knowledge of the core processes performed by hosts and network devices. When things break down and devices do not function as they should, a good understanding of processes helps you determine where exactly a process breaks down. In addition, you will be able to determine which parts of the network are functioning correctly and which parts are not functioning correctly. This section examines the processes that take place when two IP hosts communicate over a switched LAN. The focus here is at the IP and lower layers; this means that the host application is in working condition and matters such as name to IP address resolution are functional. To limit the scope of the discussion to the processes involved in Layer 2 switching, assume that the two hosts reside on a common subnet (VLAN). Because the actual application being used is irrelevant to this discussion, imagine that the user of Host A would like to test connectivity to Host B using ping as per the network shown in Figure 4-1.

Figure 4-1: Host A Tests Connectivity to Host B on the Same VLAN (Subnet)

This process can be broken down into the following steps:

Step 1	Host A will look up the destination (Host B) IP address in its routing table and determine that it is on a directly connected network.
Step 2	Because Host B is directly reachable, Host A will consult its Address Resolution Protocol (ARP) cache to the find the MAC address of Hose B.
Step 3	If the ARP cache on Host A does not contain an entry for the IP address of Host B, it will send out an ARP request as a broadcast to obtain the MAC address of Host B (see Figure 4-2). Figure 4-2: Host A Sends a Broadcast ARP Request to Obtain Host B’s MAC Address
Step 4	Switch C checks the VLAN of the port upon which it receives the frame, records the source MAC address in its MAC address table, and associates it to that port and VLAN. Switch C will perform a lookup in its MAC address table to try to find the port that is associated to the broadcast MAC address. The MAC address table never contains an entry for the broadcast MAC address (FFFF:FFFF:FFFF). Therefore, Switch C will flood the frame on all ports in that VLAN, including all trunks that this VLAN is allowed, that are active, and that are not pruned on (except the port it came in from). Switches D and E repeat this process as they receive the frame (see Figure 4-3). Figure 4-3: Switches Flood Broadcast Frames
Step 5	Host B receives the ARP request, records the IP address and MAC address of Host A in its own ARP cache, and then proceeds to send an ARP reply as a unicast back to Host A (see Figure 4-4). Figure 4-4: Host B Sends a Unicast ARP Reply Back to Host A
Step 6	The switches will check the VLAN of the port they received the frame on, and because all switches now have an entry in their MAC address table for the MAC address of Host A, they will forward the frame containing the ARP reply on the path to Host A only, not flooding it out on any other port. At the same time, they will record Host B’s MAC address and corresponding interface and VLAN in their MAC address table if they did not already have that entry (see Figure 4-5). Figure 4-5: Switches Forward the ARP Reply Unicast Frame Toward Host A
Step 7	Host A receives the ARP reply and records the IP and MAC address of Host B in its ARP cache. Now it is ready to send the original IP packet.
Step 8	Host A encapsulates the IP packet (which encapsulates an ICMP echo request) in a unicast frame destined for Host B and sends it out. Note that the Ethernet type field of 0x0800 indicates that the frame is encapsulating an IP packet.
Step 9	The switches again consult their MAC address tables, find an entry for Host B’s MAC address, and forward it on the path toward Host B (see Figure 4-6). Figure 4-6: Switches Forward ICMP Echo-Request Unicast Frame Toward Host B
Step 10	Host B receives the packet and responds to Host A (by sending an ICMP echo-reply packet).
Step 11	The switches again consult their MAC address tables and forward the frame straight to Host A, without any flooding (see Figure 4-7). Figure 4-7: Switches Forward ICMP Echo-Reply Unicast Frames Toward Host A
Step 12	Host A receives the packet, and this concludes this simple packet exchange (see Figure 4-8). Figure 4-8: Host A Receives ICMP Echo Reply Back from Host B

Although this process might seem trivial, listing the steps clearly shows that even for the simplest communication, an elaborate chain of events take place. If at any point this chain is broken due to faulty cabling, failing devices, or misconfiguration, the communication will fail. Therefore, you must leverage your knowledge of these processes to diagnose and solve problems in a switched environment. Possible issues that could cause the communication to fail include the following:

• Physical problems

• Bad, missing, or miswired cables

• Bad ports

• Power failure

• Device problems

• Software bugs

• Performance problems

• Misconfiguration

• Missing or wrong VLANs

• Misconfigured VLAN Trunking Protocol (VTP) settings

• Wrong VLAN setting on access ports

• Missing or misconfigured trunks

• Native VLAN mismatch

• VLANs not allowed on trunk

Note

Note that this list is not complete, and is focused on Layer 1 and Layer 2 issues. For example, a firewall may stop the Internet Control Message Protocol (ICMP) packets. Sometimes the very first ICMP echo request times out because of the requirement for an ARP request, which is not necessary on the following ICMP echo requests.

To keep the focus specifically on LAN switching, this chapter does not discuss any generic physical problems. However, you will learn how specific commands available on the Cisco Catalyst LAN switches could supplement your troubleshooting toolkit to help troubleshoot data link and physical layer problems.

Verifying Layer 2 Forwarding

A common method to troubleshoot Layer 2 switching problems is to follow the path of the frames through the switches. Following the actual frames themselves in real time takes a lot of time and effort because it requires packet captures at various points in the network; that is neither practical nor feasible. Instead of trying to follow the frame in real time, you must follow the trail of the frame. The objective is to confirm that frames have passed through the switches and to verify how each switch made its forwarding decisions. If you find a point where the trail suddenly stops or if you find that the information that the switch uses to forward frames does not match our expectations, you will gain important clues. These clues will help in reducing the scope of the possible problem areas, and it helps in formulating a hypothesis on the cause of the problem; it may even outright point at the cause of the problem. Consequently, a troubleshooter needs to learn how to follow the trail of the frame, and know which data structure proves whether frames have passed through the switch.

One key data structure that you can consult is the switch’s MAC address table. In this table, the switch registers the source MAC address of each frame that it receives, and records the port and VLAN on which the frame was received. When you see an entry for a particular MAC address in this table, it proves that at some point, usually up to five minutes ago, this switch received frames from that source. This does not necessarily tell you anything about a particular frame or how long ago the last frame was received. Therefore, it might be a good practice to clear the MAC entry from the table by using the clear mac-address-table dynamic command and verify that the MAC address is learned again when you reinitiate the connection. Second, the MAC address table enables you to verify that frames are received on the port and VLAN where they are expected. If the output does not match your expectations and assumptions, use it as a clue in forming a hypothesis about the cause of the problem. Because there are many possible findings and conclusions that you might draw from, the following is a short list of some common findings and plausible conclusions:

• Frames are not received on the correct VLAN: This could point to VLAN or trunk misconfiguration as the cause of the problem.

• Frames are received on a different port than you expected: This could point to a physical problem, spanning-tree issues, or duplicate MAC addresses.

• The MAC address is not registered in the MAC address table: This tells you that the problem is most likely upstream from this switch. You should retrace your steps and investigate between the last point where you know that frames were received and this switch.

The next step is to use your knowledge of the forwarding process combined with the information you can gather using the switch’s diagnostic commands output to determine what the next step in the process must be. Again, you should validate the facts that you gather about the switch’s forwarding behavior against your expectations and assumptions. After you have confirmed that the behavior of the switch matches your expectations, you have successfully reduced the possible scope of the problem—You have confirmed that everything works as expected up to this point.

You must definitely have a good grasp of the switch diagnostic commands and use them effectively. You will use these commands to gather the information from the switch that is needed to validate your assumptions. Some commonly used diagnostic commands that help you obtain information about the Layer 2 switching process, VLANs, and trunks are as follows:

• show mac-address-table: This is the main command to verify Layer 2 forwarding. It shows you the MAC addresses learned by the switch and their corresponding port and VLAN associations. This command gives you an indication if frames sourced by a particular host have succeeded in reaching this switch. It will also help you verify whether these frames were received on the correct inbound interface. Note that if the MAC address table becomes full, no more learning can happen. During troubleshooting, always check to see whether the table is full.

• show vlan: This command enables you to verify VLAN existence and port-to-VLAN associations. This command lists all VLANS that were created on the switch (either manually or through VTP). It will also list the ports that are associated to each of the VLANs. Note that trunks are not listed because they do not belong to any particular VLAN.

• show interfaces trunk: This command displays all interfaces that are configured as trunks. It will also display on each trunk which VLANs are allowed and what the native VLAN is.

• show interfaces switchport: This command combines some of the information found in show vlan and show interfaces trunk commands. It is most useful if you are not looking for a switch-wide overview of trunk or VLAN related information, but if you would rather have a quick summary of all VLAN-related information for a single interface.

• show platform forward interface: You can use many parameters with this command and find out how the hardware would forward a frame that matches the specified parameters, on the specified interface.

• traceroute mac: You specify a source and destination MAC address with this command to see the list of switch hops that a frame from that source MAC address to that destination MAC address passes through. Use this command to discover the Layer 2 path frames take from the specified source MAC address to the specified destination MAC address. This command requires that Cisco Discovery Protocol (CDP) be enabled on all the switches in the network (or at least within the path).

Based on the information they provide, the commands listed can be categorized. To display the MAC address table, use the show mac-address-table command. To display VLAN database and port-to-VLAN mapping, use the show vlan command. To see the trunk port settings and port-to-VLAN associations, use the show interfaces switchport and show interfaces trunk commands. To directly verify frame forwarding, use the show platform forward and the traceroute mac commands.