Cisco Ebook: Chapter 05: Maintaining and Troubleshooting Routing Solutions (Part01)

TCP/IP is the dominant networking protocol suite within modern networks. Internet Protocol (IP) routing takes place in almost all parts of an enterprise network, and it facilitates communication within a campus network, between branch and headquarter offices, to and from the Internet, and among virtual private network (VPN) sites. Border Gateway Protocol (BGP) is the only routing protocol capable of inter-autonomous system (liner-AS) or interdomain routing. Enhanced Interior Gateway Routing Protocol (EIGRP) or Open Shortest Path First (OSPF) Protocol, are the most preferred intra-autonomous system (intra-AS) or interior routing protocols. A good understanding of EIGRP, OSPF, BGP, route redistribution mechanisms, network layer connectivity in general, and the ability to diagnose and resolve problems associated with these technologies is therefore essential to any network engineer.

Troubleshooting Network Layer Connectivity

For most connectivity problems in IP networks, the network layer is the point where troubleshooting processes start. Examining network layer connectivity between two hosts helps determine whether the problem cause is at the same, lower, or higher layer than the network layer of the Open Systems Interconnection (OSI) model. If network layer connectivity between two hosts is functional, the problem is more likely to be caused by application issues or security-related problems. On the other hand, lack of network layer connectivity indicates that the problem is at or below network layer. A network engineer should have the skills and knowledge to diagnose and resolve these problems in an efficient manner. This can be accomplished by comprehending the processes and data structures used by routers to forward IP packets and the Cisco IOS tools that can be used to diagnose those types of problems.

Routing and Routing Data Structures

To troubleshoot Layer 3 connectivity, you need to have a good understanding of the processes involved in routing a packet from a host through multiple routers to the final destination. Figure 5-1 shows an IP network with IP Hosts A and B on the opposite ends of the network, interconnected by two networks and Routers C, D, and E.

Figure 5-1: Which Processes and Data Structures Are Involved in Sending Packets from Host A to Host B and Back?

As a network support and troubleshooting professional, you need to be able to answer the following questions about the data structures used and the processes involved in the packet exchange process between Hosts A and B, on all the devices:

Step 1	Which decisions does host A make, what information does it need, and which actions does it perform to successfully send a packet destined for host B to the first hop Router C?
Step 2	Which decisions does Router C make, what information does it need, and which actions does it perform to successfully send the packet from Host A destined for Host B to the next hop Router D?
Step 3	Which decisions does Router D make, what information does it need, and which actions does it perform to successfully send the packet from Host A destined for Host B to the next hop Router E? Is the answer to this question the same as the answer to the previous question or are there any differences?
Step 4	Which decisions does Router E make, what information does it need, and which actions does it perform to successfully send the packet from Host A destined for Host B to its final destination, Host B?
Step 5	Are there any differences in the processes and information required to successfully transmit return packets from Host B back to Host A?

If the source host or any of the routers in the path are not capable of forwarding packets, because of the lack of proper configuration or the required forwarding information, the packets are dropped, and Layer 3 connectivity is lost.

The sequence of the major processes, decisions, and actions that take place on the end devices and the intermediate devices (routers) is as follows:

Step 1	When Host A starts the process to send a packet to Host B, it first determines whether the destination network is the same or different from its own local subnet. This is done by comparing the destination IP address (10.1.4.2) to its own IP address and subnet mask (10.1.1.1/24). Host A concludes that the destination is not local, and therefore it attempts to forward the packet to its default gateway, which is known through manual configuration or learned through Dynamic Host Configuration Protocol (DHCP). To encapsulate the packet in an Ethernet frame, Host A needs the MAC address of the default gateway. This can be resolved using the Address Resolution Protocol (ARP). Host A will either already have an entry in its ARP cache for the default gateway IP address or, alternatively, it will send out an ARP request to obtain the information and populate the cache.
Step 2	Router C de-encapsulates the IP packet from the received Ethernet frame and examines the destination IP address of the IP packet. Router C decrements the Time To Live (TTL) field in the IP header of the packet by one. If this causes the TTL field to be set to zero, Router C will discard the packet and send an Internet Control Message Protocol (ICMP) “time exceeded” message back to the source, Host A. If the TTL of the packet is not reduced to zero the router performs a forwarding table lookup to find the longest prefix that matches the destination IP address of the packet being processed. In this example, Router C finds the entry 10.1.4.0/24 as the best match for the destination address of the packet (10.1.4.2). Two important parameters are associated with this entry—the next-hop IP address 10.1.2.2 and the egress interface Serial 0. This serial interface uses the High-Level Data Link Control (HDLC) protocol on Layer 2. Because this protocol does not require a MAC address or other form of Layer 2 address for the next-hop IP address, no further lookups are necessary. Router C encapsulates the packet (datagram) in a HDLC frame and transmits it out of interface Serial 0.
Step 3	Router D goes through the same general process as Router C. It also finds an entry for prefix 10.1.4.0/24 in its forwarding table. In this case, the next hop is 10.1.3.2, and the egress interface is FastEthernet 0. The biggest difference with the previous step is the Layer 2 protocol of the egress interface. Because this is a Fast Ethernet interface, the router might have to make use of ARP to resolve the next-hop IP address 10.1.3.2 to a MAC address. Normally, Router D has this address recorded in its Cisco Express Forwarding (CEF) adjacency table and need not use ARP. Router D encapsulates the packet in an Ethernet frame and forwards it to the next hop, Router E.
Step 4	The process on Router E is similar to the process on Router C and Router D. The most important difference is that the entry for prefix 10.1.4.0/24 that Router E finds in the routing table is listed as being directly connected to interface FastEthernet 1. Therefore, instead of forwarding the packet to a next-hop router, Router E forwards the packet directly to the destination Host B on the connected network. Because the Layer 2 protocol in this example is again Ethernet, Router E consults its ARP cache to find the MAC address for Host B (10.1.4.2). Host B’s MAC address might or might not be present in Router E’s ARP cache. Therefore, Router E might have to send an ARP request out and obtain Host B’s MAC address. Router E encapsulates the packet in an Ethernet frame destined for Host B and transmits the frame out of its FastEthernet 1 interface. When Host B receives the packet, this concludes the transmission of a packet from Host A to Host B.
Step 5	The process in sending return packets from Host B to Host A is similar. However, the information used in the lookups in the routing table and Layer 3 to Layer 2 mapping tables, such as the ARP cache, is different. For the return packets, the destination IP address is 10.1.1.1, so instead of entries for subnet 10.1.4.0/24, entries for subnet 10.1.1.0/24 will have to be present in all router forwarding tables. These entries will have different associated egress interfaces and next-hop IP addresses. As a result, the corresponding ARP cache entries for those next hops need to be present, and these differ from the entries that were used on the path from Host A to Host B. Consequently, you cannot conclude that if packets are successfully forwarded from Host A to Host B, return packets from Host B to Host A will automatically be successful, too. This is a wrong assumption made by many people.

When you find that there is no network layer connectivity between two hosts, a good method to troubleshoot the problem is to track the path of the packet from router to router, similar to the method of tracking the path of a frame from switch to switch to diagnose Layer 2 problems. Along the way, you have to verify the availability of a matching route in the forwarding table for the destination of the packet and, subsequently, the availability of a Layer 3 to Layer 2 address mapping for the next hop for those technologies that require a Layer 2 address, such as Ethernet. For any type of application that requires two-way communication, you have to track the packets in both directions. Availability of the correct routing information and Layer 3 to Layer 2 mappings for packets traveling in one direction does not imply that the correct information is available for packets traveling in the opposite direction, too.

Based on the processes that take place in the example presented in Figure 5-1, as packets from Host A are forwarded across multiple router hops (Routers C, D, and E) to Host B, the correct values for the packet and frame header address fields are shown in Table 5-1.

Table 5-1: Packet and Frame Header Address Fields on the Packet in Transit
Packet Position	Source IP Address	Destination IP Address	Source MAC Address	Destination MAC Address
From Host A to Router C	10.1.1.1	10.1.4.2	Host A’s MAC address	MAC address of interface Fa0 on router C
From Router C to Router D	10.1.1.1	10.1.4.2	Not applicable	Not Applicable
From Router D to Router E	10.1.1.1	10.1.4.2	MAC address of Router D’s Fa0 interface	MAC address of router E’s Fa0 interface
From Router E to Host B	10.1.1.1	10.1.4.2	MAC address of router E’s Fa1 interface	Host B’s MAC Address

To forward packets, a router combines information from various control plane data structures. The most important of these data structures is the routing table. Unlike switches, which flood a frame to all ports if its destination MAC address is not known by the MAC address table, routers drop any packet for which they cannot find a matching entry in the routing table. When a packet has to be forwarded, the routing table is searched for the longest possible prefix that matches the destination IP address of the packet. Associated with this entry is an egress interface and, in most cases, a next-hop IP address.

For point-to-point egress interfaces, such as a serial interface running PPP or HDLC, a point-to-point Frame Relay or point-to-point Asynchronous Transfer Mode (ATM) subinterface, a next-hop IP address is not mandatory, because all the information that is necessary to construct the frame and encapsulate the packet can be derived from the egress interface itself. For example, in the case of Frame Relay, a point-to-point subinterface has a single associated data-link connection identifier (DLCI), so there is no need to map the next-hop IP address to a DLCI to be able to construct the Frame Relay frame header and encapsulate the packet. For multipoint egress interfaces, such as Ethernet interfaces or multipoint Frame Relay or ATM subinterfaces, the next-hop IP address is a mandatory element because it is used to find the correct Layer 2 destination address or other Layer 2 identifier to construct the frame and encapsulate the packet. The mapping between the next hop-IP address and the Layer 2 address or identifier is stored in a data structure specific for that Layer 2 protocol. For example, in the case of Ethernet, this information is stored in the ARP cache, and in the case of Frame Relay, the information is stored in the Frame Relay map table. Consequently, a routing table lookup may need to be followed up by a lookup in a Layer 3 to Layer 2 mapping table to gather all the necessary information required to construct a frame, encapsulate the packet, and transmit it.

Executing different table lookups and combining the information to construct a frame every time a packet needs to be routed is an inefficient approach to forwarding IP packets. To improve this process and increase the performance of IP packet-switching operations on routers, Cisco has developed Cisco Express Forwarding (CEF). This advanced Layer 3 IP switching mechanism can be used on all routers, and is at the core of the Layer 3 switching technology used in Cisco Catalyst multilayer switches. On most platforms, the CEF switching method is enabled by default.

CEF combines the information from the routing table and other data structures, such as Layer 3 to Layer 2 mapping tables, into two new data structures: the Forwarding Information Base (FIB) and the CEF adjacency table. The FIB mostly reflects the routing table with all the recursive lookups resolved. A lookup in the FIB results in a pointer to an adjacency entry in the CEF adjacency table. Similar to an entry from the routing table, an adjacency table entry can consist of an egress interface only for a point-to-point interface or an egress interface and next-hop IP address for a multipoint interface.

Using IOS Commands to Verify Routing Functions

To determine the information that is used to forward packets, you can verify the availability of specific routing entry (prefix) in the routing table or the CEF FIB table.

Note

Since the release of RFC 3222 in 2001, the routing table is also called Routing Information Base (RIB) by many. Note that during discussions about protocols such as BGP and OSPF, the collection of the best paths to different destinations are referred to as BGP RIB or OSPF RIB, too. However, you must remember that BGP, OSPF, or any protocol’s best path to any destination might or might not be installed in the IP routing table (or the generic IP RIB). The reason is that when there are multiple alternatives, the IP process installs paths with the smallest administrative distance in the IP routing table (or the generic IP RIB).

The choice of whether to check the IP routing table or the FIB depends on what you are exactly trying to diagnose. To diagnose control plane problems, such as the exchange of routing information by routing protocols, the show ip route command is a clear choice because it contains all the control plane details for a route, such as the advertising routing protocol, routing source, administrative distance, and routing protocol metrics. To diagnose problems more closely related to the data plane, for example by tracking the exact traffic flow between two hosts through the network, the FIB is often the best choice because it contains all the details that are necessary to make packet-switching decisions.

To display the content of the IP routing table, you can use the following commands:

show ip route ip-address: Supplying the destination IP address as an option to the show ip route command causes the router to perform a routing table lookup for that IP address and display the best route that matches the address and all associated control plane details. (Note that the default route will never be displayed as a match for an IP address.)
show ip route network mask: The provided network and mask as options to this command request the routing table to be searched for an exact match (for that network and mask). If an exact match is found, this entry is displayed with all of its associated control plane details.
show ip route network mask longer-prefixes: The longer-prefixes option causes the router to display all prefixes in the routing table that fall within the prefix specified by the network and mask parameters. This command can prove useful to diagnose problems related to route summarization.

To display the content of the CEF FIB table, you can use the following commands:

show ip cef ip-address: This command is similar to the show ip route ip-address command, but it searches the FIB rather than the routing table. Therefore, the displayed results do not include any routing protocol-related information, but only the information necessary to forward packets. (Note that this command will display the default route if it is the best match for a particular IP address.)
show ip cef network mask: This is similar to the show ip route network mask command, but it displays information from the FIB rather than the routing table (RIB).
show ip cef exact-route source destination: This command displays the exact adjacency that will be used to forward a packet with source and destination IP addresses, as specified by the source and destination parameters. The main reason to use this command is in a situation when you are tracking a packet flow across the routed network but the routing table and FIB contain two or more equal routes for a particular prefix. In this case, the CEF mechanisms will balance the traffic load across the multiple adjacencies associated with that prefix. By use of this command, you can determine which of the possible adjacencies is used to forward packets for a specific source and destination IP address pair.

After the egress interface and, in case of multipoint interfaces, a next-hop IP address for the destination of a packet has been determined by the routing table or FIB, the router needs to construct a frame for the data link layer protocol associated with the egress interface. Depending on the data link layer used on the egress interface, the header of this frame will require some connection specific parameters, such as source and destination MAC addresses for Ethernet, DLCI for Frame Relay, or virtual path identifier/virtual channel identifier (VPI/VCI) for ATM. These data link layer parameters are stored in various different data structures. For point-to-point (sub-) interfaces, the relation between the interface and data link identifier or address is usually statically configured. For multipoint (sub-) interfaces, the relation between the next-hop IP address and the data link identifier and address can be manually configured or dynamically resolved through some form of an address resolution protocol. The commands to display the statically configured or dynamically obtained mappings are unique for each data link layer technology. Research the command references for the data link layer protocol that you are troubleshooting to find the appropriate commands. Commands that enable you to verify the Layer 3 to Layer 2 mappings include the following:

show ip arp: You can use this command to verify the dynamic IP address to Ethernet MAC address mappings that were resolved by ARP. Routers cache this information for four hours by default. If you need to refresh the content of the ARP cache, you can enter the clear ip arp command to clear all or a particular entry from the ARP cache.
show frame-relay map: This command lists all the mappings of next-hop IP addresses on multipoint (sub-) interfaces to the DLCI of the corresponding permanent virtual circuit (PVC). These mappings may have been manually configured or dynamically resolved using Frame Relay inverse ARP. In addition, this command lists any DLCIs that were manually associated to specific point-to-point subinterfaces.

When CEF is used as the switching method, the information from the various Layer 2 data structures is used to construct a frame header for each adjacency that is listed in the adjacency table. You can display the full frame header that will be used to encapsulate the packet via the show adjacency detail command. In addition, this command displays packet and byte counters for all traffic that was forwarded using this particular adjacency entry. Certain troubleshooting cases require verifying the Layer 3 to Layer 2 mappings. If the routing table or the FIB lists the correct next-hop IP address and egress interface for a particular destination, but packets do not arrive at that next hop, you should verify the Layer 3 to Layer 2 mappings for the data link protocol that is used on the egress interface. Specifically, verify that a correct frame header is constructed to encapsulate the packets and forward them to the next hop.

Troubleshooting EIGRP

The Enhanced Interior Gateway Routing Protocol (EIGRP) is a Cisco Systems proprietary interior gateway or intra-autonomous system routing protocol. This means that EIGRP is used inside an enterprise network for computing best paths and exchange of routing information among the routers. EIGRP is one of the routing protocols commonly used in any size, even large, enterprise networks. Troubleshooting problems related to the exchange of routing information is one of the most essential skills for a network engineer who is involved in the implementation and maintenance of enterprise networks. To diagnose and resolve problems related to EIGRP, you must be able to do the following:

Apply your knowledge of EIGRP data structures to plan the gathering of necessary information as part of a structured approach to troubleshooting EIGRP routing problems.
Apply your knowledge of the processes that EIGRP uses to exchange routing information to interpret and analyze the information that is gathered during an EIGRP troubleshooting process.
Use Cisco IOS commands to gather information from the EIGRP data structures and track the flow of EIGRP routing information to troubleshoot EIGRP operation.

EIGRP Routing Review

Network layer connectivity depends on having correct, consistent, and up-to-date routing information available in all routers. To forward packets correctly between hosts across a routed network, all routing tables in the path between the source and the destination need to have an entry for the destination network in their routing tables. The information used to select the best path needs to be consistent, to ensure that no routing loops occur. The information also needs to be up-to-date, to ensure that traffic will not be forwarded toward a broken link or a failed router. Although static routing can be efficiently deployed in relatively small environments or in networks that do not incorporate a lot of redundant elements at the network layer, many large routed enterprise networks deploy a routing protocol to exchange routing information in a highly scalable manner and to ensure that routing information accurately reflects changes in the state of the network. At a high level, each routing protocol consists of the following elements and processes:

Reception of routing information from neighbors: Through routing protocol updates, each router learns about the availability of paths to other (nonconnected) subnets. Interior gateway protocol (IGP) updates are exchanged among routers that are Layer 2 adjacent, commonly called neighbors. Some protocols, such as EIGRP, OSPF, or Intermediate System-to-Intermediate System (IS-IS), have a process to establish a neighbor relationship before exchanging routing information. Other protocols, such as Routing Information Protocol (RIPv2) simply accept routes that they receive without building a neighbor session first.
Routing protocol data structures: Most modern routing protocols have their own data structures where they store the routing updates that they receive from their neighbors, the locally connected or injected routes (static), or the routes redistributed from other routing protocols. Some older routing protocols, such as RIP (Version 1), do not have data structures of their own, but install and store their routing information directly in the routing table.
Route injection or redistribution: All routing information originates at a certain device in the network. All IGPs inject directly connected routes, which are associated with the interfaces that the routing protocol is activated for, into their data structures, and advertise them out to their neighbors. Routes from other sources can also be specifically redistributed into the protocol’s data structures and advertised out to the neighbors.
Route selection and installation: For each prefix stored in its data structures, a routing protocol selects one or more paths as the best for that particular prefix from all the available alternatives. Different routing protocols use different algorithms to calculate the best path. Best path selection is based on the routing protocol’s metric. Multiple paths for the same prefix will only be selected (as best) if their metrics are identical (unless the protocol, such as EIGRP, supports unequal cost load balancing). The routing protocol offers its best paths to each prefix to the IP routing table. If multiple routing protocols offer paths for the same prefix to the IP routing table, the administrative distance of the routing protocols involved will determine which routing protocol succeeds in installing its path. The path with lower administrative distance is preferred and will be installed in the IP routing table.
Transmission of routing information to neighbors: Routing information learned from other routers, plus those injected or redistributed into the routing protocol data structures, are forwarded to neighboring routers that run the same protocol. As stated about the reception of routing information, some routing protocols, such as EIGRP, OSPF, or IS-IS, require a neighbor relationship to be established before any exchange of routing information, whereas other protocols such as RIP just broadcast or multicast their information to any adjacent router that is listening.

These generic elements, behaviors, and processes help us understand the flow of information among neighboring routers, inside the router, and between the routing protocol data structures and the routing table. A main trigger that makes us start troubleshooting routing protocol operation is discovering a router that is missing routing information or has incorrect information in the routing table. That router will drop packets or send traffic out of the wrong egress interface or to the wrong next hop. Understanding these generic processes helps us establish a troubleshooting plan and track the flow of routing information as it is propagated by the routing protocol. Most of the principles used to troubleshoot one routing protocol apply to other routing protocols, too.

EIGRP stores its operational data, configured parameters, and statistics in three main data structures:

Interface table: This table lists all interfaces that have been enabled for the processing of EIGRP packets, such as hellos, updates, queries, replies, and acknowledgments. Passive interfaces are not listed in this table.
Neighbor table: This table is used to keep track of all active EIGRP neighbors. Neighbors are added to this table based on the reception of hello packets, and they are removed when a neighbors hold-time expires or when the associated interface goes down or is removed from the interface table. This table is also used to keep track of the status of any routing information exchanges between the router and its neighbors.
Topology table: This table holds all the routes that were received from neighboring routers, locally injected, or redistributed into EIGRP. For each prefix, EIGRP selects the best path from among the available possible paths stored in this table to be offered to the IP routing table. EIGRP’s best path selection is based on the diffusing update algorithm (DUAL). If multiple paths have the exact same metric, all entries that share that same metric are selected for installation in the routing table. Routes with a higher metric are not selected for installation in the routing table, unless unequal-cost load balancing has been enabled.

EIGRP uses an incremental update process. When the adjacency is first established, each router sends a full update to its neighbor (adjacent) router, sending all prefixes that it has a successor for in the topology table (Note that only the successor route entries will be sent, not feasible successors or possible successors). After the initial updates have been exchanged, routing updates will be exchanged only as a result of changes on the networks. Changes can be caused by changes in connectivity, such as loss of a link or neighbor, or they can be caused by configuration events, such as the addition of new interfaces to EIGRP, implementation of route summarization, or implementation of route filtering. All routes received from neighbors are stored in the topology table. From the routes in the topology table, the router selects the best path (or paths) for each prefix and attempts to install these entries in the routing table. Entries marked as a successor in the topology table are offered to the IP routing table and advertised to neighbors.

Monitoring EIGRP

You can use the following commands to gather information from the EIGRP data structures:

show ip eigrp interfaces: This command is used to display the list of interfaces that have been activated for EIGRP processing. This list contains all the interfaces that have an IP address that is covered by one of the network statements under the EIGRP configuration, and have not been marked as a passive interface within the EIGRP process.
show ip eigrp neighbors: This command lists all neighbors that have been discovered by this router on its active EIGRP interfaces.
show ip eigrp topology: This command displays the content of the EIGRP topology table. To select a specific entry from the table, the network and mask of the selected prefix can be provided as an option to the command.

The following debug commands enable you to observe the transmission and reception of packets and the exchange of routing information:

debug ip routing: This command is not specific to EIGRP, but displays any changes that are made to the routing table, such as installation or removal of routes. This can be useful to diagnose routing protocol instabilities.
debug eigrp packets: This command displays the transmission and reception of EIGRP packets. Either all packets can be displayed, or packets of a particular type, such as hellos, updates, queries, and replies, can be selected.
debug ip eigrp: This command displays EIGRP routing events, such as updates, queries, and replies, sent to or received from neighbors. Whereas the debug eigrp packets command shows the transmission and reception of the packets themselves, debug ip eigrp focuses on the routing information contained in the packets and the actions that EIGRP takes as a result of the information received.

debug commands can generate a large amount of output, so you must take proper care to prevent the execution of debug commands from affecting the router’s performance. Logging debug output to buffers on the router instead of to the console can limit the impact of these commands. The output of the debug eigrp neighbor and debug ip eigrp commands can be further limited by use of two additional debug commands:

debug ip eigrp neighbor as-number ip-address: By imposing this extra condition, the output of the debug eigrp packets and debug ip eigrp commands will be limited to information associated with the specified neighbor.
debug ip eigrp as-number network mask: By imposing this extra condition, the output of the debug eigrp packets and debug ip eigrp commands will be limited to information associated with the network specified by the network and mask options.

Troubleshooting Example: Routing Problem in an EIGRP Network

This example illustrates how you can leverage knowledge of EIGRP data structures, update processes, and Cisco IOS commands to troubleshoot a routing problem. The network depicted in Figure 5-2 is using EIGRP as its routing protocol.

Figure 5-2: Network Diagram for the EIGRP Troubleshooting Example

When a traceroute command is executed on the BRO1 router to determine the path used to reach the loopback of router CRO1, as shown in Example 5-1, it turns out that the traffic path goes through BRO2.

Example 5-1: Result of traceroute from BRO1 to CRO1’s Loopback Interface

BRO1# traceroute 10.1.220.1

Type escape sequence to abort.
Tracing the route to cro1.mgmt.tshoot.local (10.1.220.1)

1 10.1.163.130 0 msec 0 msec 0 msec
2 10.1.194.5 12 msec 12 msec *

This is not the expected behavior because the path across the WAN link is the better path (both WAN links have the same bandwidth). To verify whether this problem is caused by an underlying Layer 1, Layer 2, or Layer 3 problem, as shown in Example 5-2, a ping command is executed to test the Layer 3 connectivity to the other side of the WAN link.

Example 5-2: Result of ping from BRO1 to CRO1’s WAN (Serial) Interface

BRO1# ping 10.1.194.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.194.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/29/32 ms

From the fact that the ping from BRO1 to CRO1 succeeds, you can conclude that the WAN link is operational at Layers 3 and below. Given that the link is functional, but is not used to route the packets across, makes this a routing problem and calls for investigation of the operation of EIGRP. Note that you can troubleshoot this problem in various ways. The objective is to find out why the direct route across the WAN is not used to route packets to the loopback interface of router CRO1, but the packets are routed through router BRO2 instead. There are two likely reasons why the traffic is routed through router BRO2 instead of directly across the WAN to router CRO1. The first reason could be that router BRO1 has not learned about the direct route to router CRO1. The second reason could be that the router has learned about the route but incorrectly selects the route through router BRO2 as the best route. To find out which of these two possible scenarios fits the situation, you should look at the EIGRP topology table. The topology table lists all routes that were received, so you can see whether the direct route to CRO1 is missing. On the other hand, if both routes were learned, but the wrong route was selected, you will see both routes listed in the topology table, together with their metrics and this might yield a clue as to why the wrong path was selected. Example 5-3 shows the output of the show ip eigrp topology for the 10.1.220.1 prefix.

Example 5-3: How Many Paths to 10.1.220.1 in BRO1’s EIGRP Topology Table?

BRO1# show ip eigrp topology 10.1.220.1 255.255.255.255
IP-EIGRP (AS 1): Topology entry for 10.1.220.1/32
State is Passive, Query origin flag is 1, 1 Successor(s), FD is 40642560
Routing Descriptor Blocks:
10.1.163.130 (FastEthernet0/1.30), from 10.1.163.130, Send flag is 0x0
    Composite metric is (40642560/40640000), Route is Internal
    Vector metric:
      Minimum bandwidth is 64 Kbit
      Total delay is 25100 microseconds
      Reliability is 255/255
      Load is 1/255
      Minimum MTU is 1500
      Hop count is 2

The output of Example 5-3 shows that only one entry is listed in BRO1’s topology table. Therefore, you can conclude that the direct route through router CRO1 was not learned by BRO1. Because EIGRP learns all its routes from neighbors that were first discovered using hello packets, you must now determine whether the route was not learned because a neighbor relationship with CRO1 was never established or if the relationship was established but the specific route was not exchanged. Therefore, a good next step is to display the neighbor table to find out if a neighbor relationship was established between router BRO1 and router CRO1, and if so, determine what the status is. Example 5-4 displays the output of the show ip eigrp neighbors command on router BRO1.

Example 5-4: Have BRO1 and CRO1 Built an EIGRP Neighbor Relation?

BRO1# show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q   Seq
                                          (sec)         (ms)       Cnt  Num
0   10.1.163.130            Fa0/1.30        12 00:09:56   4      200  0   585

As you can see from the output of Example 5-4, router CRO1 is not listed as a neighbor on router BRO1. There can be two different reasons why router BRO1 does not list router CRO1 in its neighbor table. The trigger to add a neighbor to the neighbor list is the reception of hello packets from that neighbor. So one reason that could explain why router CRO1 is not listed is that router BRO1 has not received any hello packets from router CRO1. Another explanation could be that the packets were received, but ignored, because router BRO1 is not processing EIGRP packets on that interface. The interface table will allow you to see which interfaces are enabled for EIGRP processing. Example 5-5 shows the output of the show ip eigrp interfaces command on router BRO1.

Example 5-5: Is EIGRP Enabled on the WAN (Serial) Interface of the BRO1 Router?

BRO1# show ip eigrp interfaces
IP-EIGRP interfaces for process 1

                        Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable   SRTT  Un/Reliable   Flow Timer   Routes
Fa0/1.30           1         0/0         4        0/1            50            0

Output from the show ip eigrp interfaces command shown in Example 5-5 does not list the serial interface of router BRO1. Consequently, even if hello packets are received on the serial interface, router BRO1 does not process them. Two conditions need to be met for an interface to be added to the EIGRP interface table:

The interface has to be up, and its IP address must match one of the configured network statements.
The interface should not be configured as a passive interface.

These two conditions are both controlled by configuration commands, so at this point the configuration needs to be checked to see whether one of these two conditions is not met. Example 5-6 shows the EIGRP section of BRO1’s running configuration.

Example 5-6: On BRO1, Is EIGRP Configured to Operate over the WAN (Serial) Interface?

BRO1# show running-config | section router eigrp
router eigrp 1
network 10.1.163.129 0.0.0.0
network 10.1.194.1 0.0.0.0
no auto-summary

As you can see in the output of Example 5-6, a problem exists with one of the network statements. The statement network 10.1.194.1 0.0.0.0 matches IP address 10.1.194.1, which is not one of router BRO1’s IP addresses, but an IP address of router CRO1. This is clearly a configuration mistake, and the network statement needs to be replaced with the statement network 10.1.194.2 0.0.0.0 or some other network statement that matches IP address 10.1.194.2, which is the IP address of router BRO1 for the WAN link. The network statement must be replaced to test the hypothesis that the misconfigured network statement was the cause of the problem. Next, the results of the change need to be verified to confirm that the problem was solved. Strictly speaking, the only test that needs to be performed to confirm that the problem was solved is to execute the traceroute command again to confirm that packets traveling to the loopback IP address of router CRO1 are now sent directly across the WAN instead of making a detour through router BRO2. However, to show the impact of the change on all the EIGRP data structures, the changes in the output of the show commands that were earlier used to diagnose the problem can be tested next. Example 5-7 shows the output of show ip eigrp interfaces and the show ip eigrp neighbors commands on router BRO1 after the erroneous EIGRP network statement was replaced by the correct one. Example 5-7 shows that the interface table now lists subinterface Serial 0/0/0.111 in addition to subinterface FastEthernet 0/1.30. This means that EIGRP packets are now processed on interface Serial 0/0/0.111. Also, the IP address of router CRO1 (10.1.194.2) is now listed in the neighbor table on the WAN interface.

Example 5-7: Results of show Commands After Correcting the EIGRP network Statement

BRO1# show ip eigrp interfaces
IP-EIGRP interfaces for process 1

                        Xmit Queue   Mean   Pacing Time    Multicast    Pending
Interface         Peers  Un/Reliable   SRTT  Un/Reliable   Flow Timer    Routes
Fa0/1.30            1         0/0         1        0/1           50             0
Se0/0/0.111         1         0/0       707       10/380      4592              0

BRO1# show ip eigrp neighbors
IP-EIGRP neighbors for process 1
H   Address                   Interface       Hold Uptime   SRTT   RTO   Q  Seq
                                            (sec)         (ms)   Cnt      Num
1   10.1.194.1                Se0/0/0.111       14 00:10:10  707  4242   0  783
0   10.1.163.130              Fa0/1.30          12 01:34:49    1   200   0  587

Next, the EIGRP topology table must be examined. Example 5-8 shows the output from the show ip eigrp topology command with 10.1.220.1 as the parameter. The EIGRP topology table now lists two entries for network 10.1.220.1/32. It lists the entry through router BRO2 (10.1.163.130), which was present before the change was made, but it also lists an entry through router CRO1 (10.1.194.1). The entry through router CRO1 was selected as the best route because its metric of 40640000 is better than the metric through router BRO1, which equals 40642560. Based on the metric difference, the successor for network 10.1.220.1 is CRO1, and this path must be offered to the IP routing table.

Example 5-8: The New EIGRP Topology Table After Corrections Were Made

BRO1# show ip eigrp topology 10.1.220.1 255.255.255.255
IP-EIGRP (AS 1): Topology entry for 10.1.220.1/32
State is Passive, Query origin flag is 1, 1 Successor(s), FD is 40640000
Routing Descriptor Blocks:
10.1.194.1 (Serial0/0/0.111), from 10.1.194.1, Send flag is 0x0
    Composite metric is (40640000/128256), Route is Internal
    Vector metric:
      Minimum bandwidth is 64 Kbit
      Total delay is 25000 microseconds
      Reliability is 255/255
      Load is 1/255
      Minimum MTU is 1500
      Hop count is 1
10.1.163.130 (FastEthernet0/1.30), from 10.1.163.130, Send flag is 0x0
    Composite metric is (40642560/40640000), Route is Internal
    Vector metric:
      Minimum bandwidth is 64 Kbit
      Total delay is 25100 microseconds
      Reliability is 255/255
      Load is 1/255
      Minimum MTU is 1500
      Hop count is 2

Example 5-9 shows the output of the show ip route command with network 10.1.220.1 as the parameter. The output confirms that the path through router CRO1 has been installed in the routing table, and the result of the traceroute command, also shown in Example 5-9, confirms that this path is now used to forward packets to the loopback of CRO1.

Example 5-9: The IP Routing Table After the EIGRP Corrections Were Made

BRO1# show ip route 10.1.220.1 255.255.255.255
Routing entry for 10.1.220.1/32
Known via "eigrp 1", distance 90, metric 40640000, type internal
Redistributing via eigrp 1
Last update from 10.1.194.1 on Serial0/0/0.111, 00:20:55 ago
Routing Descriptor Blocks:
* 10.1.194.1, from 10.1.194.1, 00:20:55 ago, via Serial0/0/0.111
    Route metric is 40640000, traffic share count is 1
    Total delay is 25000 microseconds, minimum bandwidth is 64 Kbit
    Reliability 255/255, minimum MTU 1500 bytes
    Loading 1/255, Hops 1

BRO1# traceroute 10.1.220.1

Type escape sequence to abort.
Tracing the route to cro1.mgmt.tshoot.local (10.1.220.1)
1 10.1.194.1 16 msec 12 msec *

This troubleshooting example illustrates the use of the Cisco IOS show commands to display the content of the EIGRP data structures. It also illustrates how you can leverage knowledge of these data structures and the flow of EIGRP routing information to diagnose and resolve EIGRP routing problems.

Cisco Ebook

Thursday, May 26, 2011

Chapter 05: Maintaining and Troubleshooting Routing Solutions (Part01)