Network Maintenance Tools, Applications, and Resources
After you determine and define the network maintenance methods, processes, and procedures to be implemented in your organization, you can choose the tools, applications, and resources for executing your network maintenance tasks in an efficient manner. These tools must be adequate and affordable. Ideally, all the tasks that are part of your maintenance plan should be supported by the products and applications that you choose, with the initial and ongoing costs within your budget. To determine the suitability of a network maintenance toolkit, you must learn to perform the following tasks:
- Identify, evaluate, and implement the elements of a basic network maintenance toolkit.
- Evaluate tools that support the documentation process and select the tools appropriate to your organization.
- Describe how configuration, software, and hardware resource management can improve disaster recovery procedures.
- Describe how network monitoring software benefits the maintenance process.
- Analyze the metrics that could be used to measure network performance and the key elements of the performance measurement process to create a performance measurement plan appropriate to your organization.
Fundamental Tools, Applications, and Resources
You have many tools, applications, and resources from which to choose to support network maintenance processes. These tools and applications vary based on price, complexity, capability, and scalability. Figure 1-2 shows the fundamental tools and applications that belong to network maintenance toolkits.
The basic components of a network maintenance toolkit are as follows:
- Command-line device management: Cisco IOS Software includes a powerful command-line interface (CLI) that you can use to configure and monitor individual routers and switches. This includes commands such as the show commands, the debug commands, Embedded Event Manager (EEM) commands, and IP SLA commands. After an initial configuration through the serial console of the device, the CLI is commonly accessed remotely through use of the Telnet or Secure Shell (SSH) protocols. To be able to manage the devices during network outages, an out-of-band management solution can be implemented to allow access to the CLI using the serial console at all times.
- Graphical user interface (GUI)-based device management: Cisco provides free GUI-based device management tools for many Cisco routers and switches. Examples of such tools include Cisco Configuration Professional (CCP), Secure Device Manager (SDM), Cisco Configuration Assistant (CCA), and Cisco Network Assistant (CNA).
- Backup server: To create backups of the software and configurations of your routers and switches, you need to provide a TFTP, FTP, HTTP, or Secure Copy Protocol (SCP) server. Many operating systems include these services as optional add-ons, and many software packages offer those services, too.
- Log server: Basic logging functionality can be provided by sending the router’s or switch’s log messages to a syslog server using the syslog protocol. Syslog is a standard service on most UNIX-based operating systems or could be provided by installing additional software on the operating system of your choice.
- Time server: To synchronize clocks on all your network devices, it is useful to have a Network Time Protocol (NTP) server on your network. You could even synchronize your router and switch clocks to one of the many public time servers available on the Internet.
To ensure correct time stamps on logging and debug output and to support other time-based features such as the use of certificates or time-based access, it is vital that the clocks of the network devices be properly set and synchronized. The Network Time Protocol (NTP) can be used to synchronize the clock of a device to the clock of an NTP server, which in turn is synchronized to another server higher up the NTP hierarchy. The position of a device in the NTP hierarchy is determined by its stratum, which serves as an NTP hop count. A stratum 1 server is a server that is directly connected to an authoritative time source such as a radio or atomic clock. A server that synchronizes its clock to a stratum 1 server will become a stratum 2 time source, and so on.
It is common to have a redundant set of servers in the core of the network that are synchronized to an authoritative source or a service provider server, and to configure other devices to synchronize their clocks to these central sources. In large networks, this hierarchy could even consist of multiple levels. You configure time servers using the ntp server command. If multiple time servers are configured for redundancy, the NTP protocol decides which server is most reliable and will synchronize to that server. Alternatively, a preferred server can be appointed by use of the prefer command option on the ntp server command. In addition to defining timeservers, you can define your local time zone and configure the devices to adapt to daylight savings time. Finally, after you have the time synchronized and the correct time zone has been configured, you configure the router or switch to time stamp its log and debug entries.
Example 1-1 shows the clock of a device that is synchronized to a single time server with IP address 10.1.220.3. The time zone is configured to Pacific standard time (PST), which has an −8 hour offset to universal time coordinated (UTC). The clock is configured to change to daylight savings time on the second Sunday in March at 2:00 a.m. and back to standard time on the first Sunday in November at 2:00 a.m. The logging for system logging is configured to use the local date and time in the time stamps and to include the time zone in the time stamp. For log entries generated by debugs, the settings are similar, but milliseconds are included in the time stamps for greater accuracy.
Example 1-1: NTP Example
service time stamps log datetime localtime show-timezone
!
clock timezone PST -8
clock summer-time PDT recurring 2 Sun Mar 2:00 1 Sun Nov 2:00
!
ntp server 10.1.220.3
For more details about the configuration of NTP, consult the “Configuring NTP” section of the Cisco IOS Network Management Configuration Guide:
- http://tinyurl.com/yezfkr3
Configuration and Documentation Tools
Many web-based (online) maintenance tools and resources can prove helpful during the planning and implementation of network maintenance procedures. Cisco.com provides numerous tools to support your network maintenance processes. Many of the tools are available only to registered users with a valid Cisco service contract or to Cisco Channel Partners, however. Some useful and freely available tools are as follows:
- Dynamic Configuration Tool: This tool aids you in creating hardware configurations. It verifies compatibility of the hardware and software you select, and it gives you a complete bill of materials (BoM) that lists all the necessary part numbers. For more information, see https://apps.cisco.com/qtc/config/html/configureHomeGuest.html.
- Cisco Feature Navigator: This tool enables you to quickly find the right Cisco IOS Software release for the features you want to run on your network. For more information, see http://tools.cisco.com/ITDIT/CFN/
- SNMP Object Navigator: The Simple Network Management Protocol (SNMP) Navigator translates SNMP object identifiers (OIDs) into object names. This tool also enables you to download SNMP Management Information Base (MIB) files and to verify the supported MIBs in a particular Cisco IOS Software version. For more information, see http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en.
- Cisco Power Calculator: This tool calculates the power-supply requirements for a particular Power over Ethernet (PoE) hardware configuration. Access to this tool requires a Cisco.com account.
Good documentation is mostly the result of a good process. Good tools can prove extremely helpful in supporting the documentation process, but it is most important that creating and updating documentation be an integral part of your maintenance processes. The value of documentation heavily depends on how accessible it is and how up-to-date it is. If you cannot find or cannot access the documentation when you need it or if you cannot trust its information, the documentation has almost no value. Therefore, you must ensure that any tool or application you use to support creation, retrieval, and updating of documentation must be easy to access and easy to use. Examples of tools that can be used to access, create, and maintain documentation include the following:
- Wiki: A wiki combines easy web-based access with intuitive editing capabilities. It is suitable as a base documentation system. You can use a wiki as a framework to link various other existing documentation systems.
- Issue tracking system: Other names for issue tracking systems include trouble ticket, support ticket, or incident ticket system. An issue tracking system allows incoming support requests, problems, or other incidents to be logged, tracked, and documented. By documenting progress, communication, and escalation of incidents, an issue tracking system enables a team of people to work on the same incidents in an efficient manner. You can also build a historical database of problems, their treatments, and the resolutions.
Logging Services
Events that happen on a networking device such as router can be logged. There are different types of events, and different events have different levels of significance or severity. Examples of events include interfaces going up and down, configuration events, and routing protocol adjacencies being established. By default, events are usually logged only to the device’s console; however, because the console is usually not easily accessible, let alone monitored, it is worthwhile to collect and store the logs on a server, or at least in a separate piece of memory of the router, to facilitate access to them during troubleshooting procedures.
Logging the messages to buffers on the router or switch is a minimal step that guarantees that logs are available on the device, as long as it is not rebooted. On some devices and Cisco IOS Software versions, logging to buffers is turned on by default. To enable buffer logging manually, you can use the logging buffered command to specify that messages should be logged to a buffer in the device’s RAM. As an option, you can specify the amount of RAM that should be allocated to this buffer. The buffer is circular, meaning that when the buffer has reached its maximum capacity, the oldest messages will be discarded to allow the logging of new messages. You can display the content of this logging buffer via the show logging command. Logging severity levels on Cisco Systems devices are as follows:
- (0) Emergencies
- (1) Alerts
- (2) Critical
- (3) Errors
- (4) Warnings
- (5) Notifications
- (6) Informational
- (7) Debugging
When logging is enabled, the severity level can be specified as an option. This causes the device to only log messages with a severity at the configured level or lower numeric value. By default, messages of all severity levels are logged to buffer.
You can also adjust the logging severity level of the console. By default, all messages from level 0 to 7 are logged to the console; however, similar to buffer logging, you can configure the severity level as an optional parameter on the logging console command.
Logging messages are best to be sent to a syslog server. Doing so allows you to store the logs of all your network devices centrally. When logging messages are sent to a syslog server, logs are no longer lost when the networking device crashes or reboots. You can configure one or more syslog servers by entering the logging host command. By default, only messages of severity level 6 or lower are logged to the syslog server. This can be changed, similar to buffer or console logging, but unlike these other commands, the severity is configured by entering the logging trap level command. This command applies to all configured syslog hosts. Figure 1-3 shows three logging commands configured on a device and an explanation for each command.
For more detail about the configuration of logging and syslog services, consult the “Logging System Messages” section of the Cisco IOS Network Management Configuration Guide:
- http://tinyurl.com/yc8ftr6
Network Monitoring and Performance Measurement Tools
GUI- and CLI-based device management tools are used to examine individual devices, for example, at a time you suspect a problem exists. However, the problem might not be noticed until a user complains. At that point, when users have already noticed the failure, the problem has most likely had some impact on the business. A network monitoring system continuously checks your network devices’ availability and status. This enables you to detect possible problems as soon as they occur, and it might even allow you to diagnose and resolve these problems before they even become apparent to end users. Most network monitoring software incorporates various protocols such as SNMP, ICMP, and syslog so that you can monitor devices and network events. In addition, Cisco IOS NetFlow technology can be leveraged, not just to monitor the devices, but also to monitor the actual traffic on the network. Some network monitoring tools also incorporate performance monitoring capabilities. There is a gray area between network monitoring and performance measurement. While you monitor the network’s performance and look for faults and problems, you can also use your data as input for capacity planning, SLA compliance measurements, or accounting purposes.
The three main motivations for measuring network performance are as follows:
- Capacity planning: By measuring average and peak loads on the network, you can create a baseline of the traffic on your network and know the utilization of your network. By repeating the measurements over time, you also recognize trends in the growth of the traffic and can predict when you need to upgrade links or equipment before the growth starts causing congestion and performance problems.
- Diagnosing performance problems: Performance problems are difficult to troubleshoot because they are hard to quantify and are often intermittent in nature. A user might say, “Application x is really slow lately.” But what does that mean? When exactly is the application slow? What is causing this problem? Is it the client software faulty, or is it the server, or the network in between? Having a good insight into the load on the network, specifically on the path between the client and the server, helps you to determine whether network congestion might be causing the problems.
- SLA compliance: Whether you are guaranteeing a level of service to others through an SLA or whether you have been promised a certain level of service by a provider, you need to have a way to measure whether the service guarantees defined in the SLA are met.
You can measure network performance by first gathering statistics from the routers and switches. The gathered statistics can then be stored in a database, graphed over time, and analyzed. This information can be very useful for capacity planning and performance troubleshooting. Typical statistics to gather are the packet and byte counters on interfaces and device CPU and memory utilization. For SLA compliance, on the other hand, it is most useful to measure key indicators such as round-trip time (RTT), jitter, and packet loss. The IP SLA feature that is available on many Cisco routers enables you to set up probes that measure these key indicators along particular paths through your network. The statistics can then be read using Cisco IOS commands or collected using SNMP. The raw data is stored in a database and can then be analyzed or graphed. Cisco Internetwork Performance Monitor (IPM), which is part of the CiscoWorks LAN Management Solution, can leverage the IP SLA functionality in the routers to provide detailed performance graphs. In addition, various other network management software have the capability to collect statistics using SNMP and graph the results. The open source Multi Router Traffic Grapher (MRTG) and other products based on it are examples of this type of software.
Implementing Backup and Restore Services
An essential element of any network maintenance toolkit is a backup server from which device configurations and Cisco IOS Software can be copied to and restored. The simplest and most commonly implemented service is TFTP, which does not require any configuration on the network devices. The server is set up to serve and receive files without any need for authentication or identification, other than specifying the name of the configuration or software file itself. The fact that the protocol does not require any authentication and that all content is sent across the network in clear text makes it a relatively unsecure mechanism. More secure protocols such as FTP, SCP, and HTTP or HTTPS can also be used as a means of transferring configurations and software. To use any of these more-secure protocols, you must specify the username and password that are used to authenticate to the server. For all of these protocols, the credentials can be specified as part of the Uniform Resource Locator (URL) that is used with the copy command. The username and password are specified by placing the username and password as username:password@ before the server name or IP address in the URL. Example 1-3 shows how to copy the startup configuration using FTP to a server with the IP address 10.1.152.1 and a file named RO1-test.cfg using the user name backup and password san-fran.
Example 1-3: Performing Backup Using FTP with Username and Password
Address or name of remote host [10.1.152.1]?
Destination filename [RO1-test.cfg]?
Writing RO1-test.cfg !
2323 bytes copied in 0.268 secs (8668 bytes/sec)
For SCP, HTTP and HTTPS you would use a similar syntax, replacing the URL prefix ftp:// with scp://, http:// or https://, respectively. Specifying the username and password on the command line is somewhat cumbersome and suffers from the fact that the password is displayed in clear text on the screen, which is less desirable from a security standpoint. To circumvent this issue, the username and password can be specified in the configuration, instead of on the command line, for the FTP, HTTP, and HTTPS protocols. Example 1-4 shows how to store FTP and HTTP username and passwords in the configuration and perform an FTP backup without having to type the username and password.
Example 1-4: Storing FTP, HTTP Usernames and Passwords in Configuration
RO1(config)# ip ftp password san-fran
RO1(config)# ip http client username backup
RO1(config)# ip http client password 0 san-fran
RO1(config)# exit
RO1# copy startup-config ftp://10.1.152.1/RO1-test.cfg
Address or name of remote host [10.1.152.1]?
Destination filename [RO1-test.cfg]?
Writing RO1-test.cfg !
2323 bytes copied in 0.304 secs (7641 bytes/sec)
The same configuration commands are used for both HTTP and HTTPS. The only difference is the protocol identifier in the URL. It is important to know that even though FTP and HTTP require authentication, these protocols send credentials in clear text. HTTPS and SCP use encryption to ensure confidentiality of both the transmitted credentials and the content of the transferred file. When possible, use secure protocols such as HTTPS and SCP.
Creating backups of configurations should be an integral part of your network maintenance routines. After any change, you should create backups, copying the configuration file to NVRAM on the device and to a network server. If you have sufficient flash storage space on the device, you might find it useful to not only build a configuration archive on the server, but in the flash memory of the device, too. A feature that can be helpful in the creation of configuration archives, either locally on the device or remotely on a server, is the configuration archiving feature that is part of the Configuration Replace and Configuration Rollback feature that was introduced in Cisco IOS Software Release 12.3(7)T. Example 1-5 shows how to set up the configuration archive.
Example 1-5: Setting Up the Configuration Archive
Router(config-archive)# path flash:/config-archive/$h-config
Router(config-archive)# write-memory
Router(config-archive)# time-period 10080
The configuration archive is set up by entering the archive command in global configuration mode, which gets you into the config-archive configuration mode. In this configuration submode, you can specify the parameters for the archive. The only mandatory parameter is the base file path. This path will be used as the base filename and is appended with a number for each subsequent archived configuration. The path is specified in URL notation and can either be a local or a networked path supported by the Cisco IOS file system. Not all types of local flash storage are supported, so check your device’s flash type for support of this feature if you want to store your configuration archive locally on a device instead of on a server. The configuration path can include the variables $h for the device’s hostname and $t to include a time and date stamp in the filename.
After you specify the location of the archive, it is ready to be used, and archive copies of the configuration can be created manually by issuing the archive config command. However, the biggest advantage of this feature is the way you can use it to create and update a configuration archive automatically. By adding the write-memory option to the archive configuration section, you can trigger an archive copy of the running configuration to be created any time the running configuration is copied to NVRAM. It is also possible to generate archive copies of the configuration periodically by specifying the time-period option followed by a time period, specified in minutes. Each time the configured time period elapses, a copy of the running configuration will be archived.
You can verify the presence of the archived configuration files by using the show archive command. In addition to the files themselves, this command displays the most recent archived file and the filename for the next archive to be created, as demonstrated in Example 1-6.
Example 1-6: show archive Command Output
There are currently 5 archive configurations saved.
The next archive file will be named flash:/config-archive/RO1-config-6
Archive # Name
0
1 flash:/config-archive/RO1-config-1
2 flash:/config-archive/RO1-config-2
3 flash:/config-archive/RO1-config-3
4 flash:/config-archive/RO1-config-4
5 flash:/config-archive/RO1-config-5 <- Most Recent
By creating backups, either by manually copying the files or through use of configuration archiving, you have something to fall back to when disaster strikes. If the configuration of a device is lost through human error, hardware failure, or when a device needs to be replaced, you can copy the last archived configuration to the NVRAM of the device and boot it to restore it to the exact same configuration that you had stored in your archive. Another common event when you might want to restore the device to its last archived configuration is when you have made a change or a series of changes and they did not work out as expected. If these changes were made during a regularly scheduled maintenance window, you can often perform the same procedure as if you have lost a configuration entirely. You copy the last archived known-good configuration to the NVRAM of the device and reload it. However, if you made these changes during normal network operation (for example, while troubleshooting a problem), reloading the device could be a disruptive operation and not acceptable unless you have no other option.
This situation is what the configuration replace feature was designed to deal with. The configure replace command enables you to replace the currently running configuration on the router with a saved configuration. It does so by comparing the running configuration with the configuration file appointed by the configure replace command and then creates a list of differences between the files. Based on the discovered differences, various Cisco IOS configuration commands are generated that will change the existing running configuration to the replacement configuration. Example 1-7 makes a simple demonstration of this command. The advantage of this method is that only parts of the configuration that differ will be changed. The device does not need to be reloaded, and existing commands are not reapplied. This manner of rolling back to an existing archived configuration is the least-disruptive method you can use. Within the documentation at Cisco.com, the configure replace command is sometimes referred to as configuration rollback, although the command itself does not include rollback as a keyword.
Example 1-7: A Simple Demonstration of the configure replace Command
Enter configuration commands, one per line. End with CNTL/Z.
RO1(config)# hostname TEST
TEST(config)# ^Z
TEST# configure replace flash:config-archive/RO1-config-5 list
This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ? [no]: yes
!Pass 1
!List of Commands:
no hostname TEST
hostname RO1
end
Total number of passes: 1
Rollback Done
RO1#
Example 1-7 shows that the hostname of a device is changed and then the configuration is rolled back to the most current archived configuration. The command option list is added to the configure replace command to show the configuration commands that are being applied by the configuration replacement. As you can see from the example, the change that was made is undone, without affecting any other parts of the configuration. Although this command was designed to complement the configuration archiving feature, you can use the configure replace command with any complete Cisco IOS configuration file.
Disaster Recovery Tools
To successfully recover from a disaster, you need the following:
- Up-to-date configuration backups
- Up-to-date software backups
- Up-to-date hardware inventories
- Configuration and software provisioning tools
As parts of the fundamental network maintenance toolkit, TFTP, FTP, SCP, HTTP, and HTTPS servers are useful for creating backups of the configuration and operating system of a router or switch. Additional software can be used to extend the functionality of the toolkit and perform the following tasks:
- Automatic backup scheduling
- Configuration file comparison and change tracking
- Template creation and editing
- Pushing configurations to multiple devices
- Hardware inventory tracking
CiscoWorks Resource Manager Essentials (RME), which is part of the CiscoWorks LAN Management Solutions (LMS), is a prime example of software that provides these functions. For more on LMS and other network management utilities offered by Cisco, see http://www.cisco.com/en/US/products/sw/netmgtsw/index.html.
Summary
There are two network maintenance models: interrupt driven and structured. The structured network maintenance model has the following advantages over its interrupt-driven counterpart:
- Reduced network downtime
- More cost-effectiveness
- Better alignment with business objectives
- Higher network security
Examples of structured network maintenance methodologies include the following:
- IT Infrastructure Library (ITIL)
- FCAPS (Fault, Configuration, Accounting, Performance, and Security Management; by ISO)
- TMN (Telecommunications Management Network; By ITU-T)
- Cisco Lifecycle Services
Upon selection of a network maintenance model/methodology, you must translate the theoretical model to practical procedures that structure the network maintenance processes for your network. After you have defined your processes and procedures, it becomes much easier to see what functionalities and tools you need to have in your network management toolkit to support these processes. As a result, you can select an efficient and cost-effective network management and support toolkit that offers those tools and hopefully meets your budgetary constraints.
All network maintenance plans need to include procedures to perform the following tasks:
- Accommodating adds, moves, and changes
- Installing and configuring new devices
- Replacing failed devices
- Backing up device configurations and software
- Troubleshooting link and device failures
- Upgrading or patching software
- Monitoring the network
- Performance measurement and capacity planning
- Writing and updating documentation
Network maintenance planning includes the following:
- Scheduling maintenance
- Change-control procedures
- Network documentation
- Effective communication
- Defining templates/procedures/conventions
- Disaster recovery.
Benefits of scheduled maintenance include the following:
- You reduce network downtime.
- Long-term maintenance tasks will not be neglected or forgotten.
- You have predictable lead times for change requests.
- Disruptive maintenance tasks can be scheduled during assigned maintenance windows, reducing downtime during production hours.
Typical elements of network documentation include the following:
- Network drawings
- Connection documentation
- Equipment lists
- IP address administration
- Configurations
- Design documentation
To perform successful disaster recovery when a device fails, you need to have the following saved and available:
- Replacement hardware
- The current software version for the device
- The current configuration for the device
- The tools to transfer the software and configuration to the device
- Licenses (if applicable)
- Knowledge of the procedures to install software, configurations, and licenses
To determine the suitability of a network maintenance toolkit, you must learn to perform the following tasks:
- Identify, evaluate, and implement the elements of a basic network maintenance toolkit.
- Evaluate tools that support the documentation process and select the tools appropriate to your organization.
- Describe how configuration, software, and hardware resource management can improve disaster recovery procedures.
- Describe how network monitoring software benefits the maintenance process.
- Analyze the metrics that could be used to measure network performance and the key elements of the performance measurement process to create a performance measurement plan that is appropriate to your organization.
The basic components of a network maintenance toolkit are as follows:
- CLI-based device management
- GUI-based device management
- Backup server
- Log server
- Time server
Examples of Cisco Systems web-based (online) maintenance tools and resources that can prove helpful during the planning and implementation of network maintenance procedures are as follows:
- Dynamic Configuration Tool: This tool aids you in creating hardware configurations. It verifies compatibility of the hardware and software you select, and it gives you a complete bill of materials (BoM) that lists all the necessary part numbers.
- Cisco Feature Navigator: This tool enables you to quickly find the right Cisco IOS Software release for the features you want to run on your network. At the time of this writing, Cisco Feature Navigator does not support Cisco IOS Software Release 15.0.
- SNMP Object Navigator: The Simple Network Management Protocol (SNMP) Navigator translates SNMP object identifiers (OIDs) into object names. This tool also enables you to download SNMP Management Information Base (MIB) files and to verify the supported MIBs in a particular Cisco IOS Software version.
- Cisco Power Calculator: This tool calculates the power supply requirements for a particular Power over Ethernet (PoE) hardware configuration.
Examples of tools that can be used to access, create, and maintain documentation are as follows:
- Wiki
- Issue tracking system
The three main motivations for measuring network performance are as follows:
- Capacity planning
- Diagnosing performance problems
- SLA compliance
TFTP, FTP, SCP, HTTP, and HTTPS can all be used to transfer files between your network and backup devices. FTP, SCP, HTTP, and HTTPS are more secure than TFTP because they require authentication. SCP and HTTPS are most secure because they also incorporate encryption.
A feature that can prove helpful in the creation of configuration archives, either locally on the device or remotely on a server, is the configuration archiving feature that is part of the Configuration Replace and Configuration Rollback feature that was introduced in Cisco IOS Software Release 12.3(7)T.
To successfully recover from a disaster, you need the following:
- Up-to-date configuration backups
- Up-to-date software backups
- Up-to-date hardware inventories
- Configuration and software provisioning tools
0 comments
Post a Comment