Chapter 10: Review and Preparation for Troubleshooting Complex Enterprise Networks

ddd

Today’s networks incorporate many different technologies, protocols, devices, and features. To maintain and troubleshoot these complex, integrated enterprise networks efficiently and effectively, you need to acquire broad knowledge and a wide range of skills. In addition to detailed knowledge of individual technologies, protocols, devices, and features, you need to understand how these components interact and depend on each other.

You need to have a sound understanding of the methods that you could use to isolate problems to a particular device, protocol, or feature, and have the ability to plan a troubleshooting process and adjust your plan based on the results of the process. You also need to know how to find the documentation for technologies, protocols, or features that you are not familiar with. In addition, you must be familiar with the tools that are available to support network maintenance and troubleshooting processes and know how to use those tools. Finally, you need to be able to plan, communicate, document, and execute structured network maintenance and troubleshooting processes.

Practicing these skills in a noncritical environment is an important tool to assess whether you have understood and incorporated your acquired knowledge, and to further improve your troubleshooting expertise.

Review of Key Maintenance and Troubleshooting Concepts and Tools

In Chapter 1, “Planning Maintenance for Complex Networks,” you learned about the advantages of structured network maintenance over interrupt-driven network maintenance, and you were introduced to some examples of structured network maintenance methodologies. Network maintenance planning includes scheduling maintenance, change-control procedures, network documentation, effective communication, defining templates/procedures/conventions, and disaster recovery.

In Chapter 2, “Troubleshooting Processes for Complex Enterprise Networks,” you learned that the fundamental elements of a troubleshooting process are gathering information and symptoms, analyzing information, eliminating possible causes, formulating a hypothesis, and testing the hypothesis. You also learned that the commonly used troubleshooting approaches are top down, bottom up, divide and conquer, follow the path, spot the differences, and move the problem. Communication is an essential part of the troubleshooting process, and it happens in all the following stages of troubleshooting:

Step 1	Reporting the problem
Step 2	Gathering information
Step 3	Analyzing and eliminate
Step 4	Proposing and test a hypothesis
Step 5	Solving the problem.

Chapter 3, “Using Maintenance and Troubleshooting Tools and Applications,” showed you how to use the Cisco IOS commands to selectively gather information in support of basic diagnostic processes. It also introduced you to the tools commonly used for specific maintenance and troubleshooting processes and preparing the infrastructure for their use.

Chapter 4, “Maintaining and Troubleshooting Campus Switched Solutions,” reviewed important LAN switching topics and presented you with useful skills and commands to troubleshoot VLANs, Spanning Tree Protocol, inter-VLAN routing, and first-hop redundancy protocols.

Chapter 5, “Maintaining and Troubleshooting Routing Solutions,” discussed network layer connectivity and in-depth troubleshooting of Enhanced Interior Gateway Protocol (EIGRP), Open Shortest Path First (OSPF) Protocol, Border Gateway Protocol (BGP), and route redistribution.

Chapter 6, “Troubleshooting Addressing Services,” analyzed common IPv4 addressing issues related to address translation (NAT), Dynamic Host Configuration Protocol (DHCP) and common IPv6 routing issues.

In Chapter 7, “Troubleshooting Network Performance Issues,” you learned that the main categories of application services are as follows:

• Network classification
• Application scalability
• Application networking
• Application acceleration
• WAN acceleration
• Application optimization

You also learned that the recipe to application optimization is a four-step cycle that incrementally increases your understanding of network applications and allows you to progressively deploy measurable improvements and adjustments as required. Those steps are as following:

Step 1	Baseline application traffic.
Step 2	Optimize the network.
Step 3	Measure, adjust, and verify.
Step 4	Deploy new applications.

Chapter 7 also provided you with some knowledge and skills to troubleshoot network applications services, switch performance issues, and router performance issues.

In Chapter 8, “Troubleshooting Converged Networks,” you learned that common wireless integration issues include the following:

• Problems at the wireless to wired boundary
• Filters that might be blocking traffic
• Wireless quality of service (QoS) and wired QoS mapping might be incorrect
• Power over Ethernet (PoE) issues
• Trunk issues

You also learned that the design and troubleshooting considerations of integrating unified communications (VoIP and IP Telephony, for example) into a campus LAN are as follows:

• QoS (bandwidth, delay, jitter, packet loss, network QoS readiness, trust boundaries)
• High availability (Spanning Tree Protocol / Rapid Spanning Tree Protocol (STP/RSTP), Hot Standby Router Protocol / Gateway Load Balancing Protocol / Virtual Router Redundancy Protocol (HSRP/GLBP/VRRP)
• Security (traffic segregation, firewalling/filtering)
• Availability and correct provisioning of other services (Power over Ethernet (PoE), Dynamic Host Control Protocol (DHCP), Trivial File Transfer Protocol (TFTP), Network Time Protocol (NTP), Cisco Discovery Protocol (CDP), trunking, VLANs)

In the final part of Chapter 8 you learned some of the common video integration issues:

• Excessive bandwidth utilization
• Lack of control
• Poor quality (lack of QoS)
• Security issues (filtering of key protocols, and stateful requirements)
• Multicast issues

In Chapter 9, “Maintaining and Troubleshooting Network Security Implementations,” you learned about the three main functional planes:

• Management plane
• Control plane
• Data plane

The management functions of a router or switch are commonly accessed using three different methods:

• Cisco IOS command-line interface (CLI)
• Web-based device management
• A network management platform that is based on Simple Network Management Protocol (SNMP)

All of these methods must be used in the most secure way, based on the device type, its operating system (IOS) capabilities, and the security policies of the organization. There are two common techniques to secure management access to network devices. First, access to the management plane can be restricted using packet or session filters—Access is allowed only from specific source IP addresses. Second, it is necessary to authenticate the user that is attempting to access the device. Misconfiguration of any of the following can lead to control plane failures:

• Routing protocol or first-hop redundancy protocol authentication
• STP options such as BPDU Guard, BPDU filter, Root Guard, and Loop Guard
• DHCP snooping
• Dynamic ARP inspection
• Control plane policing or protection

Data plane security is accomplished using a variety of router and switch options such as Unicast Reverse Path Forwarding (Unicast RPF), IPsec, Network Admission Control, and 802.1x port authentication.

Applying Maintenance and Troubleshooting Concepts and Tools

In complex integrated networks, a combination of many different protocols and technologies deliver network services that support the enterprise applications. These protocols and technologies interact with each other: Some processes merely interact with each other, some depend on other processes, and yet other processes perform their functions in parallel (see Figure 10-1). If one of the elements fails or does not perform as expected, it impacts the system as a whole. At some point, symptoms will become noticeable at the application level, and a user will report a problem. A diagnostic process starts to find the elements that are causing the problem and eliminates the problem by implementing a solution or workaround.

Figure 10-1: Network Elements Are Interrelated, or Dependent on Each Other, or Run in Parallel

To be able to troubleshoot a complex network in an effective and efficient manner, the following aspects are important:

• You should have a good understanding of each of the individual technologies, protocols, and features used in your network. At the minimum, you should have a high-level overview of the functionality offered by each of the technologies, protocols, and features. You might not need to have specialist knowledge about each particular technology, but if this is not the case, you should know how to obtain the required background knowledge, documentation, or expert support in your organization.
• You should have a good understanding of the interactions and dependencies between the different technologies, protocols, and features in your network. If a problem symptom is noticed in one of the elements, this does not necessarily mean that the root cause of the problem exists in that element itself. The root cause might be in another element, upon which the element that you are investigating, depends or interacts with. For example, a routing protocol failure might be caused by an underlying Layer 2 problem.
• You should have a good understanding of the way control plane processes influence data plane processes. Problem symptoms are usually noticed and reported on the data plane, while the root cause in many cases is in the control plane, where the solutions or workarounds must be implemented. Knowing how control plane data structures are used in data plane forwarding, and knowing how these data structures are populated, is essential for successful diagnosis and resolution of network problems.
• You should have a good working knowledge of and experience with the tools available to gather information about the operation of various technologies, protocols, features, and processes. This includes both specialized troubleshooting tools and tools used in support of regular maintenance processes. Examples of such tools are those that support change processes, baseline collection, communication, and documentation.

Before you start troubleshooting, you must perform the following preliminary tasks:

• Document the network (devices, connections, protocols, addresses, routing protocols, VLANs). Use IOS commands and other tools to support this effort. Create physical and logical topologies.
• Back up all current configurations.
• Review the security policy. Document the security implementations in the network.

As you troubleshoot different cases, keep in mind that different approaches might suit different cases. Therefore, depending on the case, the approach you take might be top down, bottom up, follow the path, or a hybrid method. The key is to have an approach and follow through, documenting all discoveries or changes. No matter which method is used, the basic elements should include the following, the workflow for which is illustrated in Figure 10-2:

 







Figure 10-2: Troubleshooting Approach Workflow

Step 1	Define the problem.
Step 2	Gather information. Document any issues.
Step 3	Analyze the information.
Step 4	Eliminate possible causes.
Step 5	Formulate and propose a hypothesis. Document possible solutions.
Step 6	Implement and test hypothesis. Always circle back to the security policy to ensure any changes meet the policy requirements.
Step 7	Solve problem and document changes.

Summary

Previous chapters reviewed many of the individual technologies commonly used in complex, integrated enterprise networks. Furthermore, these chapters demonstrated the use of tools and structured methods with many troubleshooting examples, and important lessons and best practices derived from these experiences.

If you need to master network troubleshooting, you must get a lot of hands-on experience with the networking devices. This can be done using the in-house lab of your employer, by using rental remote labs, setting up your own small lab, or by using network simulators. Also, the Cisco Networking Academy and Cisco Learning Partners offer Cisco official curriculum courses, most of which include extensive hands-on labs.

Finally, if you are interested in becoming a Cisco Certified Network Professional, and need other books to prepare you for the troubleshooting examination, you should consider the Cisco Press CCNP Troubleshooting Exam Certification Guide.