Using Specialized Maintenance and Troubleshooting Tools
Information gathering is essential to both troubleshooting and maintenance. Information is either gathered on a need basis, such as during a troubleshooting effort, or continuously as part of baseline creation. Sometimes network events trigger information gathering. In addition to the tools available in the Cisco IOS CLI, many specialized network maintenance and troubleshooting tools support information gathering processes. These tools and applications typically require communication with the network devices, and several different underlying technologies can govern this communication. Network support professionals must familiarize themselves with the commonly used network management platforms and troubleshooting tools and learn to perform the following tasks:
- Identify the tools and their underlying technologies to support the troubleshooting process.
- Enable Switched Port Analyzer (SPAN) and Remote SPAN (RSPAN) to facilitate the use of packet sniffers.
- Configure routers and switches for communication with Simple Network Management Protocol (SNMP) or NetFlow-based network management systems to facilitate the collection of device and traffic statistics that are part of a network baseline.
- Configure routers and switches to send SNMP traps to provide fault notification to SNMP-based network management systems.
Categories of Troubleshooting Tools
A generic troubleshooting process consists of several phases or subprocesses. Some of these are primarily mental; an example is the elimination process. Some of these processes are administrative in nature, such as documenting and reporting the changes and solutions. Finally, some of these are more technical in nature, such as the gathering and analysis of information. The processes that benefit the most from the deployment of network maintenance and troubleshooting tools are the processes that are technical in nature:
- Defining the problem: One of the main objectives of deploying a proactive network management strategy is to learn about potential problems before users report that they are experiencing outages or performance degradation. Network monitoring and event reporting systems can notify the network support team of events as they happen, giving them lead time to battle the problem before the users notice and report them.
- Gathering information: This is one of the most essential steps in the troubleshooting process, and it is incident driven and targeted. It is beneficial to be able to leverage any tool to obtain detailed information about events in an effective way.
- Analyzing: A major aspect of interpreting and analyzing the gathered information is comparing them against the network baseline. The ability to differentiate between normal and abnormal behavior can yield important clues about the potential problem cause. Collecting statistics about network behavior and network traffic is therefore a key process to support troubleshooting data analysis.
- Testing the hypothesis: Testing a hypothesis commonly involves making changes to the network. This might entail the need to roll back these changes if they do not resolve the problem or cause new ones. Tools that enable easy rollback of changes are therefore important to an efficient troubleshooting process.
With the exception of configuration rollback, which is more a generic change management tool than a specific troubleshooting tool, the mechanisms mentioned fall into one of the following three categories:
- Collection of information on demand, driven by incidents: This is the typical information gathering that you do during the troubleshooting process itself. Information is gathered, interpreted, and analyzed, and based on the outcome of this process, more information is gathered. Examples of this are capturing of network traffic or debugging output of device processes.
- Continuous collection of information to establish a baseline: This operation entails establishing a set of key network performance indicators. Based on these indicators, statistics about the network behavior over a long period of time are collected. These statistics form a baseline that you can use to judge whether the network behavior that you observe is normal. This process also provides historical data that you can correlate to events. Examples of this are collection of statistics through use of the SNMP and traffic accounting by use of NetFlow technology.
- Notification of network events: This method, instead of continuously collecting information, is based on events triggering devices to report specific information. Examples of this are the reporting of events through syslog messages or SNMP traps and the definition and the reporting of specific events through the use of the Embedded Event Manager (EEM) that is part of Cisco IOS Software.
What these categories all have in common is that their functionality depends on interaction between a tool or an application running on a host and the network devices. In the first two categories, the information is pulled from the network elements to the application or tool, whereas in the last category the information is pushed to the application or tool by the network devices. A broad spectrum of tools and applications can perform the processes mentioned, but it is unfeasible to mention them all, let alone compare and contrast them. However, many of these tools depend on the same underlying technologies and protocols for the communication between the application and the network devices. Examples of such technologies and protocols are syslog, SNMP, and NetFlow, plus network event notification technologies. Apart from understanding the main benefits that a particular tool or application brings to the network troubleshooting process, it is therefore also important for an engineer to know how to enable the necessary communication between the network devices and the tools and applications.
Using Traffic-Capturing Tools
Packet sniffers, also called network or protocol analyzers, are important and useful tools for network engineers. Using these tools, you can look for and observe protocol errors like retransmissions or session resets. Captured traffic can also be helpful when diagnosing communication problems between two hosts. If you can spot where packets start to go missing, for example, this will help in pinpointing the problem. Packet sniffers are powerful tools because they capture large amounts of very detailed data, but that can also be a drawback. Unless you know exactly what you are looking for and you know how to set up a filter so that only the traffic you are interested in is displayed, it can be very hard to analyze packet captures. Figure 3-2 displays a sample screen from a protocol analyzer. The first four packets shown on the screen are the four-way DHCP exchange resulting in an IP address lease from a DHCP server to a DHCP client. The next three packets are gratuitous ARP from an IP device.
Various tools on the market (some free and some for a fee) enable packet capturing and packet analysis. These tools can be either software based and installed on a regular computer or they can be appliance-style devices (with specialized hardware) that can capture vast amounts of data in real time. Whichever tool you select, it is always important to learn the filtering capabilities of the product so that can select just the information that you are interested in. Furthermore, one of the issues that you generally run into is that it is not always practical or even possible to install the software on the device that is the subject of your troubleshooting. On servers, and even on certain clients, the installation of software is often tightly controlled (and in many cases, prohibited). Sometimes it is the capturing of large amount of data that is not possible or allowed on the server or client. Fortunately, a solution to this problem exists. If you cannot perform packet capturing on a particular device itself, then from the switch that the device is connected to, you can transport the traffic that you want to capture to another device that has the software installed.
SPAN and RSPAN
The Switched Port Analyzer (SPAN) feature of Cisco Catalyst switches allows copying the traffic from one or more switch interfaces or VLANs to another interface on the same switch. You connect the system with the protocol analyzer capability to an interface on the switch; this will be the destination interface of SPAN. Next, you configure the switch to send a copy of the traffic from one or more interfaces or VLANs to the SPAN destination interface, where the protocol analyzer can capture and analyze the traffic. The traffic that is copied and sent to the SPAN destination interface can be the incoming traffic, outgoing traffic, or both, from the source interfaces. The source and destination interfaces (or VLANs) all reside on the same switch.
Figure 3-3 shows a switch that is configured to send traffic from the source interface Fa0/7 to the destination interface Fa0/8 using the SPAN feature. The objective is to capture all the traffic sent or received by the server connected to interface Fa0/7, to troubleshoot a problem with that server. A packet sniffer is connected to interface Fa0/8. The switch is instructed to copy all the traffic that it sends and receives on interface Fa0/7 to interface Fa0/8. This is done using the monitor session session# commands shown on the top of Figure 3-3. Each SPAN session has a unique session identifier; in this particular case, the configured SPAN session number is 1. The source ports or VLANs are identified by use of the monitor session session# source command, and the destination port is identified by use of the monitor session session# destination command. The session number is what binds the commands together to form a single session. On the bottom of Figure 3-3, the configuration of the SPAN session is verified using the show monitor command. The output of this command shows that both incoming and outgoing traffic are sent from Fa0/7 to Fa0/8. Also, the frame type of native indicates Ethernet frames rather than 802.1Q frames. The last line indicating ingress as disabled means that on the destination interface Fa0/8 (where the sniffer is connected) ingress traffic is not accepted as per the current configuration.
Using the Remote Switched Port Analyzer (RSPAN) feature, however, you can copy traffic from ports or VLANs on one switch (let’s call it the source switch) to a port on a different switch (destination switch). A VLAN must be designated as the RSPAN VLAN and not be used for any other purposes. The RSPAN VLAN receives traffic from the ports or VLANs on the source switch. The RSPAN VLAN then transports the traffic through one or more switches all the way to the destination switch. On the destination switch, the traffic is then copied from the RSPAN VLAN to the destination port. Be aware that each switching platform has certain capabilities and imposes certain restrictions on the usage of RSPAN/SPAN. You can discover these limitations and capabilities of such in the corresponding device documentation. Figure 3-4 shows an example of RSPAN configuration on two LAN switches connected by an 802.1Q trunk.
The configuration of RSPAN is similar to the configuration of SPAN in the sense that it uses the monitor session session# source and monitor session session# destination commands to define the interface that traffic is captured from and the interface that traffic is copied to. However, because the source and destination interface are now on two different switches, a medium is needed to transport the traffic from one switch to the other. This is done using a special RSPAN VLAN. As you can see in Figure 3-4, the SPAN VLAN number 100 is used for this purpose, and it is created similarly to any other VLAN. However, the configuration of RSPAN VLAN requires the remote-span command within the VLAN configuration mode, and this VLAN needs to be defined on all switches and allowed on all the trunks within the path between the source and destination switches. On the source switch, the RSPAN VLAN is configured as the destination for the SPAN session through use of the monitor session session# destination remote vlan vlan# command, and in a similar way the destination switch is configured to use the RSPAN VLAN as the source of the SPAN session through use of the monitor session session# source remote vlan vlan# command. The RSPAN VLAN needs to match on the source and destination switches, but the session numbers do not need to match. The session numbers are local identifiers that define the relationship between sources and destinations for a session on a single switch. The session numbers are not communicated between switches. In Figure 3-5, the show monitor command is used to verify the configuration of RSPAN on the source and destination switches.
From the output of the show monitor command in Figure 3-5, notice that on the source switch (SW1), the session is identified as a Remote Source Session, whereas on the destination switch (SW2), it is marked as a Remote Destination Session. In addition to verifying the correct configuration of the RSPAN session, it is important that you verify the fact that the VLAN is configured correctly as a RSPAN VLAN on both switches. The show vlan remote-span command enables you to verify this. Finally, if pruning is enabled on the trunks within the path between source and destination switches, you should verify that the RSPAN VLAN is allowed on those trunks.
Gathering Information with SNMP
Simple Network Management Protocol (SNMP) and NetFlow are two main technologies that are used to gather statistics from Cisco switches and routers. Although there is a certain amount of overlap between the types of data they can collect, SNMP and NetFlow each have a different focus. SNMP’s focus is primarily on the collection of various statistics from network devices. Routers and switches (and other network devices) keep statistics about the operation of their processes and interfaces locally. These statistics can be viewed through the CLI or graphical user interface (GUI), which is enough if all you need is a snapshot view of particular statistics or parameters at a single moment of time. However, if you want to collect and analyze these statistics over time, you can take advantage of SNMP.
SNMP makes use of an SNMP network management station (NMS) and one or more SNMP agents. Agents are special processes running on the devices we would like to monitor and collect information about. You can query the SNMP Agent by use of the SNMP protocol and obtain the values for the parameters or counters of interest. By periodically querying (polling) the SNMP agent, the SNMP NMS can collect valuable ongoing information and statistics and store them. This data can then be processed and analyzed in various ways. Averages, minimums, maximums, and so on can be calculated, the data can be graphed, and thresholds can be set to trigger a notification process when they are exceeded. Statistics gathering with SNMP is considered a pull-based system because the NMS polls devices periodically to obtain the values of the objects that it is set up to collect. An NMS can query about numerous objects. These objects are organized and identified in a hierarchical model called a Management Information Base (MIB).
To configure a router for SNMP-based access is fairly simple. Although SNMP Version 3 is the official current standard, Version 2c is still the most widely used version. SNMP Version 3 offers enhanced security, through authentication and encryption. In SNMP Version 2c, access to the SNMP agent is granted based on an SNMP community string. An SNMP community string is comparable to a shared password; it must match between the NMS and the agent. Two different SNMP community strings are usually defined, one for read-only access and another one for read-write access. For statistics gathering, only read access is required, and therefore a read-write community is optional and does not need to be defined. Although it is not strictly necessary, it is also beneficial to define the SNMP contact and location. Because these parameters can be collected using SNMP, the support contact and physical location of a device can be retrieved. Another useful configuration, especially when creating a baseline or graphing interface-related variables, is the snmp-server ifindex persist command. This command guarantees that the SNMP interface index for each interface will stay the same, even if the device is rebooted. Without this command, you could run into a situation where the interface’s ifindex changes after a reboot and counters for that interface are no longer correctly graphed. Figure 3-6 displays a simple set of SNMP configuration commands on a router.
In the example shown in Figure 3-6, the read-only community string is set to cisco, and the read-write community is set to san-fran. Furthermore, the location is set to TSHOOT Lab Facility, the contact is set to support@mgmt.tshoot.local, and ifindex is set to persistent.
For increased security, you can define access lists to allow only SNMP access from certain subnets. Finally, in scenarios where access needs to be granted for only a small collection of MIB objects, an SNMP view can be defined together with a specific community string. Then, access to those MIBs will be granted only if the requestor has a matching community.
Gathering Information with NetFlow
NetFlow has a different focus and uses different underlying mechanisms. A NetFlow-enabled device, such as a router or Layer 3 switch, will collect information about the IP traffic that is flowing through (transit through) the device. The NetFlow feature classifies traffic by flow. A flow is identified as a collection of packets that have the same essential header fields, such as source IP address, destination IP address, protocol number, Type of Service (TOS) field, port number (if applicable), plus the ingress interface. For each individual flow, the number of packets and bytes is tracked and accounted. This information is kept in a flow cache. Flows are expired from the cache when the flows are terminated or time out.
Among Cisco LAN switches, NetFlow is currently supported on most router platforms. Regarding the Catalyst switches, currently only 4500 and 6500 series support the NetFlow. You can enable this feature as a standalone feature on a router (interfaces) and examine the NetFlow cache using the proper CLI commands. This can be a useful tool during troubleshooting, because it enables you to see the flow entries being created as packets enter the router. In that sense, you could utilize NetFlow as a diagnostic tool. However, the biggest strength of the NetFlow technology is that in addition to keeping a local cache and temporary accounting of the flows on the device itself, the flow information can be exported to a NetFlow collector. Before entries are expired from the cache, the flow information, consisting of the key packet headers and additional information such as packet counts, byte counts, egress interface, flow start, and flow duration, is sent to the flow collector. The collector receives the flow information and records it in a database. Although the content of the packet payloads is not recorded, the flow information transferred to the collector by the router essentially contains a full view of all the traffic that has transited through the router. Enabling NetFlow and exporting the flows from a number of key routers can yield a fairly complete view of all the traffic on the network. After collection, the NetFlow data can be processed and graphed.
To export NetFlow information to a collector, you must first enable NetFlow accounting (collection) on the desired router interfaces. This is done using the ip flow ingress interface configuration mode IOS command. You also need to configure three more items:
|
|
|
|
|
|
|
|
|
Figure 3-7 shows a router with its Fa0/0 and Fa0/1 interfaces enabled to collect NetFlow information for ingress traffic. The definition of a flow is unidirectional, so if you want to account for both inbound and outbound traffic, the feature needs to be turned on for both interfaces. The Cisco IOS router command ip flow ingress replaces the old ip route-cache flow command. The NetFlow information is exported to a collector with the IP address 10.1.152.1 (at UDP port 9996), the packet format version is 5, and interface loopback 0’s IP address is used as the source of the outgoing IP packet.
After the router starts caching and accounting flow information locally in its memory, you can display the NetFlow cache content by issuing the show ip cache flow command. This command can prove very useful when troubleshooting connection problems because it shows the active flows that are sending packets through the router. Example 3-15 shows partial output of the show ip cache flow command on a router.
Example 3-15: show ip cache flow Command Output
<...output omitted>
SrcIf SrcIPaddress DstIF DstIPaddress Pr SrcP DstP Pkts
Se0/0/0.121 10.1.194.10 Null 224.0.0.10 58 0000 0000 27
Se0/0/0.121 10.1.194.14 Null 224.0.0.10 58 0000 0000 28
Fa0/0 10.1.192.5 Null 224.0.0.10 58 0000 0000 28
Fa0/1 10.1.192.13 Null 224.0.0.10 58 0000 0000 27
Fa0/1 10.1.152.1 Local 10.1.220.2 01 0000 0303 1
Se0/0/1 10.1.193.6 Null 224.0.0.10 58 0000 0000 28
Fa0/1 10.1.152.1 Se0/0/1 10.1.163.193 11 0666 E75E 1906
Se0/0/1 10.1.163.193 Fa0/0 10.1.152.1 11 E75E 0666 1905
The output filtering options for show commands can be used to select only those IP addresses that you are interested in. For example, for the sample output in Example 3-15, the command show ip cache flow | include 10.1.163.193 could have been used to limit the output to only those flows that have 10.1.163.193 as the source or destination IP address.
In contrast to SNMP, NetFlow uses a “push”-based model. The collector will simply be listening to NetFlow traffic, and the routers will be in charge of sending NetFlow data to the collector, based on changes in their flow cache. Another difference between NetFlow and SNMP is that NetFlow only gathers traffic statistics, whereas SNMP can also collect many other performance indicators, such as interface errors, CPU usage, and memory usage. On the other hand, the traffic statistics collected using NetFlow have a lot more granularity than the traffic statistics that can be collected using SNMP.
Enabling Network Event Notification
A key element of a proactive network management strategy is fault notification. When a significant event such as a failure or intrusion happens on a network, the support group should not be notified of it through user reports or complaints. It is best if network devices report that event to a central system and the support group becomes aware of the issue before problems associated with the event are noticed and reported by users. In addition to learning about the event earlier, the support group will also have the advantage of getting a report of the underlying event rather than a mere description of symptoms. Two popular protocols that are used for this purpose are syslog and SNMP. In addition, the EEM feature in Cisco IOS provides an advanced method to create custom events and define actions to be taken in response to those events.
Syslog is a simple protocol used by an IP device (syslog client) to send text-based log messages to another IP device (syslog server). These messages are the same messages that are displayed on the console of Cisco routers and switches. The syslog protocol allows these messages to be forwarded across the network to a central log server that collects and stores the messages from all the devices. By itself, this constitutes only a very basic form of event notification and collection: The network device notifies the log server, and the log message is stored. However, the network support team must be notified of significant events. Fortunately, syslog capabilities are included as a component of many network management systems, and these systems often include advanced mechanisms to notify network support engineers of significant events.
SNMP allows an agent running on a network device to be queried by an SNMP manager for various matters, including configuration settings, counters, and statistics. In addition to responding to polling, the agent can be configured to send messages to the SNMP manager based on the occurrence of events, such as an interface going down or device configuration change. These messages are called traps and do not contain user-readable text; instead, they include SNMP MIB objects and the associated variables. Consequently, traps must always be processed by an SNMP-based network management system that can interpret and process the MIB object information contained in the trap.
Both syslog messages and SNMP traps use predefined messages that are embedded in Cisco IOS Software. These messages can be triggered on predefined conditions, and the content of each message is fixed. The number and variety of defined syslog messages and SNMP traps is extensive, and as a result, they will fulfill the fault-notification needs of most organizations. However, special cases arise at times when you would like to be notified of a particular condition or event that is not part of the standard set of log messages and events included in Cisco IOS. For these special cases, Cisco IOS Software has a feature called Embedded Event Manager (EEM), which enables you to define custom events and corresponding actions.
Figure 3-8 shows a router configured with the commands that enable SNMP trap notification. This task is performed in two steps. In the first step, one or more trap receivers are defined, and in the second step, sending traps is enabled.
Trap receivers are configured using the snmp-server host host traps [version {1 | 2c | 3}] community-string command. By default, Cisco routers send SNMP Version 1 traps, but higher versions can be configured explicitly. If you want a router to send specific traps, each desired trap must be encoded with the snmp-server enable traps notification-type command. On the other hand, to enable all trap types with a single command, you must use the snmp-server enable traps command. This command will not appear in the configuration as a command because it executes a macro that enables all available categories of traps. This effect is visible in the output of the show running-config | include traps command in Figure 3-8. SNMP and syslog both act on predefined triggers and send predefined messages. Both protocols allow for a limited amount of filtering, but it is not possible to define entirely new event triggers or messages.
The EEM framework enables the creation of custom policies that trigger actions based on events. Events can be triggered based on various Cisco IOS subsystems such as syslog messages, Cisco IOS counter changes, SNMP MIB object changes, SNMP traps, CLI command execution, timers, and many other options. Actions can consist of sending SNMP traps or syslog messages, executing CLI commands, sending e-mail, or even running tool command language (TCL) scripts. Thus, EEM allows you to create very powerful and complex policies.
Example 3-16 shows how to implement a policy using the Cisco IOS EEM feature. Assume that all network engineers within an organization are granted privileged access to the routers and switches and can make changes if necessary. The rule is that only Level 3 support engineers are allowed to make emergency changes if required, but Level 1 and 2 engineers always need to obtain authorization before making any change to the system. Whenever an engineer configures a router or switch, a %SYS-5-CONFIG_I message is logged to the syslog server. However, this message is logged as a syslog level five “notification” message and does not show up in the logs as a high-priority item. There is a requirement that a message must be logged as soon as anybody enters configuration mode; that is in addition to the %SYS-5-CONFIG_I message that is logged after configuration mode is exited. This message should be logged as a critical message and an informational message should be logged reminding the engineer of the existing change-control policies.
Example 3-16: A Sample EEM Configuration
Router(config-applet)#event cli pattern "configure terminal" sync no skip no
occurs 1
Router(config-applet)#action 1.0 syslog priority critical msg "Configuration mode
was entered"
Router(config-applet)#action 2.0 syslog priority informational msg "Change control
policies apply. Authorized access only."
The simple EEM applet shown in Example 3-16 accomplishes the policy requirements using the following four command lines:
- The event manager applet CONFIG-STARTED command creates an applet called CONFIG-STARTED.
- The event that should trigger this applet is defined on the second line using the command event cli pattern “configure terminal” sync no skip no occurs 1. This line effectively says that the policy should be triggered if a command that includes “configure terminal” is entered. The occurs 1 option forces the event to be triggered on a single occurrence of the CLI pattern.
- The action 1.0 syslog priority critical msg “Configuration mode was entered” command defines an action named 1.0 (actions are sorted in alphabetic order) and instructs the router to log a critical message containing the text “Configuration mode was entered.”
- The action 2.0 syslog priority informational msg “Change control policies apply. Authorized access only.” command defines an action named 2.0 and instructs the router to log an informational message containing the text “Change control policies apply. Authorized access only.”
Example 3-17 shows the effect and result of the EEM policy discussed in Example 3-16. As soon as a user enters the IOS global configuration mode, two messages appear. The first message is a critical message (syslog level 2) stating “%HA_EM-2-LOG: CONFIG-STARTED: Configuration mode was entered,” and the second message is an informational message (syslog level 6) stating “%HA_EM-6-LOG: CONFIG-STARTED: Change control policies apply. Authorized access only.”
Example 3-17: A Sample EEM Policy Result
Enter configuration commands, one per line. End with CNTL/Z.
RO1(config)#
Mar 13 03:24:41.473 PDT: %HA_EM-2-LOG: CONFIG-STARTED: Configuration mode was
entered
Mar 13 03:24:41.473 PDT: %HA_EM-6-LOG: CONFIG-STARTED: Change control policies
apply. Authorized access only.
Examples 3-16 and 3-17 demonstrated a simple implementation of EEM and its policy results. Bear in mind, however, that the EEM is a very powerful tool, and by incorporating the use of the TCL scripting language tool, you can implement a complete distributed notification system.
| Note |
|
Summary
You can apply filtering to the Cisco IOS show ip route command. For example:
- To limit the output of the show ip route command, you can enter a specific IP address on the command line as an option. Doing so causes the router to execute a routing table lookup for that specific IP address and see whether it finds a match. If it finds a match in the routing table, it displays the corresponding entry with all its details. If it does not find a match in the routing table, it displays a message saying % Subnet not in table.
- Another way to limit the output from the show ip route command to a particular subset of routing information that you are interested in is to specify a prefix followed by the longer-prefixes keyword. The router will then list all subnets that fall within the prefix that you have specified (including that prefix itself, if it is listed in the routing table).
The output of Cisco IOS show commands can be filtered by appending a pipe character (|) to the show command followed by one of the keywords include, exclude, or begin and then a regular expression.
For selecting pieces of the configuration file, there is an option that allows you to select lines from the configuration that match a particular regular expression and any following associated lines. For example, the command show running-config | section router will select all lines that include the expression router and the configuration section that follows that line.
Instead of filtering the output of a show command, the output can also be redirected, copied or appended to a file by using the pipe character followed by the options redirect, tee, or append and a URL that denotes the file.
When you use the | redirect option on a show command, the output will not be displayed on the screen, but redirected to a text file instead. This file can be stored locally on the device’s flash memory or it can be stored on a network server such as a TFTP or FTP server. The | tee option is similar to the | redirect option, but the difference is that this command both displays the output on your screen and copies it to a text file. Finally, the | append option is analogous to the | redirect option, but it allows you to append the output to a file instead of replacing that file.
Ping is an excellent connectivity-testing tool, especially when used with its options such as repeat, size, source, and df-bit. The repeat option enables you to specify the number of echo requests sent out. The size option enables you to specify the size of the packets sent out, and the source option enables you to specify which local interface IP address should be used as the source IP address on the packets sent out. The Don’t Fragment option (df-bit) instructs the routers on the path not to fragment the packet sent; this option proves useful for path MTU testing. If you type ping with no IP address and press Enter, the extended ping dialog allows you to choose more options such as sweep sizes.
The Telnet application on Cisco IOS enables you to do transport layer testing by attempting to build a Telnet session to any TCP port of an IP device. If the port is active, you will get a response such as Open. A no response or refusal indicates that either the port is not active on the target IP device or that security features do not allow you to connect to that port.
Cisco IOS Software includes many commands to diagnose hardware operation. Due to their nature, many of those commands and features are product and platform specific. Essential commands common to both routers and switches include the following:
- show processes cpu
- show memory
- show interfaces
The show processes cpu command gives you an overview of all processes currently running on the router, the total amount of CPU time that they have consumed over their lifetime, and their CPU usage over the last 5 seconds. This command also displays a 1-minute and a 5-minute weighted average of CPU utilization for all processes
Both routers and switches have an amount of generic RAM memory, used by processes and for temporary packet buffering. Not having sufficient free memory can cause memory-allocation problems. Establishing a baseline can help discover these issues before they cause disruption.
The output of the show interfaces command include statistics such as input and output errors, CRC errors, collisions, and queue drops. Error statistics should be related to total packet statistics. Use the clear counters command to reset the interface counters and ensure that you are observing recent data. Use output filtering to limit the output to the fields that you are interested in viewing.
Other commands that can be useful in troubleshooting hardware related problems include the following:
- show controllers
- show platform
- show inventory
- show diag
You can use the following features, among others, to diagnose interface hardware or cabling issues:
- Generic Online Diagnostics (GOLD)
- Time Domain Reflectometer (TDR)
Different phases of the troubleshooting process can benefit from specific types of troubleshooting tools:
- Defining the problem: Network monitoring and event reporting tools
- Gathering information: Incident driven, targeted information-gathering tools
- Analyzing: Baseline creation and traffic-accounting tools
- Testing the hypothesis: Configuration rollback tools
Examples of network monitoring and event reporting tools include the following:
- Logging system messages to syslog
- Event notification using SNMP
- Event notification using the Embedded Event Manager (EEM)
Examples of incident-related information gathering are SPAN and RSPAN (used for traffic capturing).
The following are examples of baseline creation and traffic-accounting tools:
- Statistics gathering using SNMP
- Traffic accounting using NetFlow
Packet sniffers can be used to capture packets to allow detailed analysis of packet flows. Taking packet captures at various points in the network allows you to spot potential differences.
The SPAN feature allows traffic to be copied from one or more source ports or source VLANs to a port on the same switch for capture and analysis.
The RSPAN feature allows traffic to be copied from one or more source ports or source VLANs on one switch to a port on another switch by use of a special RSPAN VLAN. The RSPAN VLAN needs to be carried between the source and destination switches by use of trunks. RSPAN cannot cross Layer 3 boundaries.
Two main technologies that can be used to create a baseline of network usage and performance are SNMP and NetFlow.
SNMP is a standard protocol and is a pull model where the NMS polls devices for specific information. Cisco IOS NetFlow is based on the collection of detailed traffic profiles, and it is considered a push model. The NetFlow-enabled device pushes flow information to a collector, as traffic flows through the device. NetFlow is a Cisco router-specific feature.
Routers and switches can notify network management stations of significant events using two common methods: syslog and SNMP. In addition, you can use the EEM feature in Cisco IOS to create custom events and define actions to be taken in response to the event.
0 comments
Post a Comment