Chapter 13: Deploying High Availability Features (Part02)

8-3: Firewall Failover Configuration Examples

This section presents several examples of firewall failover configuration, each with a different set of platforms or failover modes. In each example, two firewalls are configured as a failover pair. Each interface from the failover pair connects to a separate switch or VLAN so that the failover feature can exchange hello messages and detect failures.

Active-Standby Failover Example with PIX Firewalls

Figure 8-7 shows the IP addresses of each interface. The addresses of the standby unit interfaces are also given. Stateful failover is used so that connection state information is passed to the standby unit in real time. An example of failover using the serial failover cable is shown first, followed by a LAN-based failover scenario. Following the failover guidelines, a separate VLAN or switch is used for stateful failover (the “stateful” interface) and for LAN-based failover (the “lanfo” interface) information exchange.

Figure 8-7: Network Diagram for the Active-Standby Example

The failover pair is configured to use the most aggressive peer monitoring policy possible, using a peer hello interval of 3 seconds on PIX 6.3 and 200 milliseconds on ASA.

Preliminary configuration for stateful failover using is as follows:

Open table as spreadsheet

PIX 6.3 Configuration

ASA Configuration

Firewall(config)# nameif gb- ethernet0 outside security0 Firewall(config)# ip address outside 192.168.1.1 255.255.255.0 Firewall(config)# failover ip address outside 192.168.1.2 Firewall(config)# nameif gb- ethernet1 inside security100 Firewall(config)# ip address inside 192.168.254.1 255.255.255.0 Firewall(config)# failover ip address inside 192.168.254.2 Firewall(config)# nameif gb- ethernet2 stateful security20 Firewall(config)# ip address stateful 192.168.199.1 255.255.255.0 Firewall(config)# failover ip address stateful 192.168.199.2 Firewall(config)# failover link stateful Firewall(config)# failover replicate http Firewall(config)# failover poll 3 Firewall(config)# failover

Firewall(config)# interface gigabitethernet0 Firewall(config-if)# description Outside public network Firewall(config-if)# nameif outside Firewall(config-if)# security-level 0 Firewall(config-if)# ip address 192.168.1.1 255.255.255.0 standby 192.168.1.2 Firewall(config-if)# exit Firewall(config)# interface gigabitethernet1 Firewall(config-if)# description Inside private network Firewall(config-if)# nameif inside Firewall(config-if)# security-level 100 Firewall(config-if)# ip address 192.168.254.1 255.255.255.0 standby 192.168.254.2 Firewall(config-if)# exit Firewall(config)# interface gigabitethernet2 Firewall(config-if)# description Stateful Failover link Firewall(config-if)# failover interface ip stateful 192.168.199.1 255.255.255.0 standby 192.168.199.2 Firewall(config-if)# exit Firewall(config)# failover link stateful ethernet0 Firewall(config)# failover replication http Firewall(config)# failover polltime unit msec 200 holdtime msec 800 Firewall(config)# failover

Additional configuration for the primary LAN-based failover unit is as follows:

Open table as spreadsheet

PIX 6.3 Configuration

ASA Configuration

Firewall(config)# nameif gb- ethernet3 lanfo security30 Firewall(config)# ip address lanfo 192.168.200.1 255.255.255.0 Firewall(config)# failover ip address lanfo 192.168.200.2 Firewall(config)# failover lan unit primary Firewall(config)# failover lan interface lanfo Firewall(config)# failover lan key myprivatekey Firewall(config)# failover lan enable

Firewall(config)# interface gigabitethernet3 Firewall(config-if)# description LAN-based failover Firewall(config-if)# no shutdown Firewall(config-if)# exit Firewall(config)# failover interface ip lanfo 192.168.200.1 255.255.255.0 standby 192.168.200.2 Firewall(config)# failover lan unit primary Firewall(config)# failover lan interface lanfo gigabitethernet3 Firewall(config)# failover lan myprivatekey Firewall(config)# failover lan enable

Now, a session is opened to the secondary failover unit, and the following additional configuration commands are added to it:

Open table as spreadsheet

PIX 6.3Configuration

ASA Configuration

Firewall(config)# nameif gb-ethernet3 lanfo security30 Firewall(config)# ip address lanfo 192.168.200.1 255.255.255.0 Firewall(config)# failover ip address lanfo 192.168.200.2 Firewall(config)# failover lan unit secondary Firewall(config)# failover lan interfac lanfo Firewall(config)# failover lan key myprivatekey Firewall(config)# failover lan enable

Firewall(config)# interface gigabitethernet3 Firewall(config-if)# description LAN-based failover Firewall(config-if)# no shutdown Firewall(config-if)# exit Firewall(config)# failover interface ip lanfo 192.168.200.1 255.255.255.0 standby 192.168.200.2 Firewall(config)# failover lan unit secondary Firewall(config)# failover lan interface lanfo gigabitethernet3 Firewall(config)# failover key myprivatekey Firewall(config)# failover lan enable

Active-Standby Failover Example with FWSM

Now, suppose these firewalls are actually FWSMs. Suppose the inside interface uses VLAN 100, outside uses VLAN 200, stateful uses 300, and lanfo uses 400. The configuration for the primary FWSM in slot 3 would look like this, beginning with the necessary Catalyst 6500 commands:

Switch(config)# vlan 100,200,300,400
Switch(config)# firewall vlan-group 1 100,200,300,400
Switch(config)# firewall module 3 vlan-group 1
Switch(config)# exit
! Now open a session to the FWSM itself
Switch# session slot 3 processor 1

Next are the FWSM commands:

fwsm(config)# nameif vlan100 inside security100
fwsm(config)# ip address inside 192.168.254.1 255.255.255.0 standby 192.168.254.2
fwsm(config)# nameif vlan200 outside security0
fwsm(config)# ip address outside 192.168.1.1 255.255.255.0 standby 192.168.1.2
fwsm(config)# failover interface ip stateful 192.168.199.1 255.255.255.0 standby
 192.168.199.2
fwsm(config)# failover link stateful vlan 300
fwsm(config)# failover interface ip lanfo 192.168.200.1 255.255.255.0 standby
 192.168.200.2
fwsm(config)# failover lan unit primary
fwsm(config)# failover lan interface lanfo vlan 400
fwsm(config)# failover replication http
fwsm(config)# failover polltime unit msec 500 holdtime 3
fwsm(config)# failover

Then, on the secondary FWSM in the module 4 slot, the following configuration commands are entered, beginning with the Catalyst 6500 session. Note that the first two switch commands are entered again only if the two FWSMs are located in separate switch chassis. If the FWSMs are housed in the same switch chassis, the VLANs and the firewall VLAN group are already configured:

Switch(config)# vlan 100,200,300,400
Switch(config)# firewall vlan-group 1 100,200,300,400
Switch(config)# firewall module 4 vlan-group 1
Switch(config)# exit
! Now open a session to the FWSM itself
Switch# session slot 4 processor 1
[output omitted]
fwsm(config)# failover interface ip stateful 192.168.199.1 255.255.255.0 standby
 192.168.199.2
fwsm(config)# failover link stateful vlan 300
fwsm(config)# failover interface ip lanfo 192.168.200.1 255.255.255.0 standby
 192.168.200.2
fwsm(config)# failover lan unit secondary
fwsm(config)# failover lan interface lanfo vlan 400
fwsm(config)# failover

Active-Active Failover Example

Suppose a Cisco firewall is to be configured with multiple security contexts so that it can provide firewall services to two customers of a service provider. A total of three contexts are needed:

• admin

• CustomerA

• CustomerB

To enhance the availability of the firewall contexts, a second firewall is added to form a failover pair. Active-active failover is used so that one firewall has the active role for some contexts and the other firewall is active for a different set of contexts.

Figure 8-8 shows a basic diagram for this arrangement. The primary and secondary firewalls use LAN-based failover communication over their Ethernet0/0 interfaces. The firewalls send failover hello messages once every second and wait for 3 seconds before declaring their peer failed. Stateful failover carries connection state information between the firewalls over their GigabitEthernet1/2 interfaces.

Figure 8-8: Network Diagram for the Active-Active Example

Tip

This example uses interface names that are relevant to ASA platforms with a four-port GigabitEthernet module. The same example can apply to FWSM platforms, by substituting VLANs for physical interface names.

One path to the public network is provided over the firewalls’ GigabitEthernet1/0 interfaces. This connection becomes the outside interface for each of the security contexts so that they share a common pipe to the public Internet.

Each security context has its own separate inside interface, which is carried as a VLAN over the GigabitEthernet1/1 trunking firewall interfaces. VLAN 10 is assigned to the admin context, VLAN 20 to the CustomerA context, and VLAN 30 to the CustomerB context.

Finally, two failover groups are used to allow the two firewalls to have differing roles in the contexts. Failover group 1 gives preference to the primary unit having the active role. The admin and CustomerA contexts are members of group 1. Failover group 2 gives preference to the secondary unit having the active role, where the CustomerB context is a member.

Remember that each interface of the primary firewall is “connected” to the corresponding interface of the secondary firewall. In other words, the two firewalls have their corresponding interfaces assigned to the same VLANs so that failover messages can be exchanged between the firewall units over each of their interfaces. This is required for failover operation.

Active-active failover can be difficult to visualize and configure. The configuration for this example is broken into several steps, as described in the following sections.

Primary Firewall Configuration

First, you need to configure the primary firewall. Remember that failover is configured in the system execution space of a multiple-context firewall.

1. Begin with only the interfaces needed for failover.

   The GigabitEthernet1/2 (stateful failover) and Ethernet0 (LAN-based failover) interfaces are needed. Their configuration is not necessary, because the actual failover commands remove any IP addressing or other parameters that might be assigned in interface configuration mode. They are shown here only for clarity:

   Firewall(config)# mode multiple
   [output omitted]
   ! Here, the system execution space is being configured
   !
   Firewall(config)# interface gigabitethernet1/2
   Firewall(config-if)# description Stateful failover interface
   Firewall(config-if)# exit
   Firewall(config)# interface ethernet0/0
   Firewall(config-if)# description LAN-based failover interface
   Firewall(config-if)# exit
   

2. Configure LAN-based failover:

   Firewall(config)# failover lan unit primary
   Firewall(config)# failover lan interface LAN-fo ethernet0/0
   Firewall(config)# failover interface ip LAN-fo 192.168.1.1 255.255.255.0
    standby 192.168.1.2
   Firewall(config)# failover polltime unit 1 holdtime 3
   Firewall(config)# failover key *****
   Firewall(config)# failover lan enable
   

3. Configure stateful failover:

   Firewall(config)# failover interface ip Stateful 192.168.2.1 255.255.255.0
    standby 192.168.2.2
   Firewall(config)# failover link Stateful gigabitethernet1/2
   

4. Define the failover groups:

   Firewall(config)# failover group 1
   Firewall(config-fover-group)# primary
   Firewall(config-fover-group)# preempt
   Firewall(config-fover-group)# exit
   Firewall(config)# failover group 2
   Firewall(config-fover-group)# secondary
   Firewall(config-fover-group)# preempt
   Firewall(config-fover-group)# exit
   

5. Enable failover:

   Firewall(config)# failover
   

Secondary Firewall Configuration

1. Begin with only the interfaces needed for failover.

   The GigabitEthernet1/2 (stateful failover) and Ethernet0/0 (LAN-based failover) interfaces are paired with identical interfaces on the primary firewall. Their configuration is not necessary; it is shown here only for clarity:

   Firewall(config)# mode multiple
   [output omitted]
   ! Here, the system execution space is being configured
   !
   Firewall(config)# interface gigabitethernet1/2
   Firewall(config-if)# description Stateful failover interface
   Firewall(config-if)# exit
   Firewall(config)# interface ethernet0/0
   Firewall(config-if)# description LAN-based failover interface
   Firewall(config-if)# exit
   

2. Configure LAN-based failover.

   Here, the secondary unit begins its life in standby mode for all failover groups. Therefore, it knows to pick up the standby IP address for the LAN-based and stateful failover interfaces in the following commands:

   Firewall(config)# failover lan unit secondary
   Firewall(config)# failover lan interface LAN-fo ethernet0/0
   Firewall(config)# failover interface ip LAN-fo 192.168.1.1 255.255.255.0
    standby 192.168.1.2
   Firewall(config)# failover polltime unit 1 holdtime 3
   Firewall(config)# failover key *****
   Firewall(config)# failover lan enable
   

3. Configure stateful failover:

   Firewall(config)# failover interface ip Stateful 192.168.2.1 255.255.255.0
    standby 192.168.2.2
   Firewall(config)# failover link Stateful gigabitethernet1/2
   

4. Define the failover groups.

   Although the failover groups could be explicitly configured here, as they were on the primary firewall unit, that is not really necessary. The failover group configuration is replicated to the secondary unit as soon as failover is enabled on each. The following commands become a part of the configuration:

   Firewall(config)# failover group 1
   Firewall(config-fover-group)# primary
   Firewall(config-fover-group)# preempt
   Firewall(config)# exit
   Firewall(config)# failover group 2
   Firewall(config-fover-group)# secondary
   Firewall(config-fover-group)# preempt
   Firewall(config)# exit
   

5. Enable failover:

   Firewall(config)# failover
   

Allocating Interfaces to the Contexts

Configuration should continue on the primary firewall in the system execution space. As soon as failover is enabled on both firewalls and is operational, any remaining configuration commands are replicated to the secondary unit automatically. This saves time and effort over entering the same commands in the two firewalls manually.

1. Define physical interfaces in the system execution space.

   Here, you identify the interfaces. You also configure the subinterfaces of GigabitEthernet1/1 with their VLAN numbers on the trunk. No IP addresses are assigned in the system execution space. Rather, the addressing is left up to the administrator of the context where each interface will be allocated:

   Firewall(config)# interface gigabitethernet1/0
   Firewall(config-if)# description Public Network for all contexts
   Firewall(config-if)# exit
   !
   Firewall(config)# interface gigabitethernet1/1
   Firewall(config-if)# description Trunk for non-public networks
   Firewall(config-if)# exit
   !
   Firewall(config)# interface gigabitethernet1/1.10
   Firewall(config-if)# description Private network for admin context
   Firewall(config-if)# vlan 10
   Firewall(config-if)# exit
   !
   Firewall(config)# interface gigabitethernet1/1.20
   Firewall(config-if)# description Private network for CustomerA context
   Firewall(config-if)# vlan 20
   Firewall(config-if)# exit
   !
   Firewall(config)# interface gigabitethernet1/1.30
   Firewall(config-if)# description Private network for CustomerB context
   Firewall(config-if)# vlan 30
   Firewall(config-if)# exit
   

2. Allocate interfaces to the contexts and failover groups.

   For the admin context, you can allocate interfaces only with their physical interface names. For the CustomerA and CustomerB user contexts, however, you can assign logical names (intf0 and intf1) so that the physical identity remains hidden:

   Firewall(config)# context admin
   Firewall(config-ctx)# allocate-interface gigabitethernet1/0
   Firewall(config-ctx)# allocate-interface gigabitethernet1/1.10
   Firewall(config-ctx)# config-url flash:/admin.cfg
   Firewall(config-ctx)# join-failover-group 1
   Firewall(config-ctx)# exit
   !
   Firewall(config)# context CustomerA
   Firewall(config-ctx)# description Virtual firewall for CustomerA
   Firewall(config-ctx)# allocate-interface gigabitethernet1/0 intf0
   Firewall(config-ctx)# allocate-interface gigabitethernet1/1.20 intf1
   Firewall(config-ctx)# config-url flash:/CustomerA.cfg
   Firewall(config-ctx)# join-failover-group 1
   Firewall(config-ctx)# exit
   !
   Firewall(config)# context CustomerB
   Firewall(config-ctx)# description Virtual firewall for CustomerB
   Firewall(config-ctx)# allocate-interface gigabitethernet1/0 intf0
   Firewall(config-ctx)# allocate-interface gigabitethernet1/1.30 intf1
   Firewall(config-ctx)# config-url flash:/CustomerB.cfg
   Firewall(config-ctx)# join-failover-group 2
   Firewall(config-ctx)# exit
   

Configuring Interfaces in Each Context

After you have allocated the physical firewall interfaces to the security contexts, you must configure them for use by the contexts. You do this by opening a session to each context in turn. At this point, remember that the context is a virtual firewall, so each interface needs an IP address, a security level, and a name. Also remember that each context has its own concept of failover. Each interface needs a standby IP address, too.

1. Configure the admin context interfaces:

   Firewall# change to context admin
   Firewall/admin# configure terminal
   Firewall/admin(config)# interface gigabitethernet1/0
   Firewall/admin(config-if)# nameif outside
   Firewall/admin(config-if)# security-level 0
   Firewall/admin(config-if)# ip address 192.168.93.1 255.255.255.0 standby
    192.168.93.2
   Firewall/admin(config-if)# exit
   !
   Firewall/admin(config)# interface gigabitethernet1/1.10
   Firewall/admin(config-if)# nameif inside
   Firewall/admin(config-if)# security-level 100
   Firewall/admin(config-if)# ip address 192.168.1.1 255.255.255.0 standby
    192.168.1.2
   Firewall/admin(config-if)# exit
   Firewall/admin(config)# exit
   

2. Configure the CustomerA context interfaces:

   Firewall/admin# changeto context CustomerA
   Firewall/CustomerA# configure terminal
   Firewall/CustomerA(config)# interface intf0
   Firewall/CustomerA(config-if)# nameif outside
   Firewall/CustomerA(config-if)# security-level 0
   Firewall/CustomerA(config-if)# ip address 192.168.93.140 255.255.255.0
    standby 192.168.93.141
   Firewall/CustomerA(config-if)# exit
   !
   Firewall/CustomerA(config)# interface intf1
   Firewall/CustomerA(config-if)# nameif inside
   Firewall/CustomerA(config-if)# security-level 100
   Firewall/CustomerA(config-if)# ip address 192.168.200.10 255.255.255.0
    standby 192.168.200.11
   Firewall/CustomerA(config-if)# exit
   Firewall/CustomerA(config)# exit
   

3. Configure the CustomerB context interfaces:

   Firewall/CustomerA# changeto context CustomerB
   Firewall/CustomerB# configure terminal
   Firewall/CustomerB(config)# interface intf0
   Firewall/CustomerB(config-if)# nameif outside
   Firewall/CustomerB(config-if)# security-level 0
   Firewall/CustomerB(config-if)# ip address 192.168.93.150 255.255.255.0
    standby 192.168.93.151
   Firewall/CustomerB(config-if)# exit
   !
   Firewall/CustomerB(config)# interface intf1
   Firewall/CustomerB(config-if)# nameif inside
   Firewall/CustomerB(config-if)# security-level 100
   Firewall/CustomerB(config-if)# ip address 192.168.220.10 255.255.255.0
    standby 192.168.220.11
   Firewall/CustomerB(config-if)# exit
   Firewall/CustomerB(config)# exit
   

8-4: Managing Firewall Failover

By nature, firewall failover is a feature that can take action automatically, based on whether two firewalls are operational and connected. You might want to monitor or troubleshoot the failover mechanism on a failover pair so that you can verify its operation. As well, there might be occasions when you need to manually force the failover action between the peers. The following sections cover these topics.

Displaying Information About Failover

When you connect to a firewall remotely, it is not always apparent which unit is the active one. Because the active unit configuration is replicated to the standby unit, the command-line prompt (and the underlying host name) is identical on both units. This can make interacting with the correct firewall very difficult.

After you connect to a firewall, use the show failover command to determine the state of that unit, as shown in the following example:

Firewall# show failover
Failover On
Cable status: Normal
Reconnect timeout 0:00:00
Poll frequency 15 seconds
       This host: Primary - Active
               Active time: 2421015 (sec)
               Interface stateful (192.168.199.1): Normal
               Interface dmz2 (127.0.0.1): Link Down (Shutdown)
               Interface outside (192.168.1.1): Normal
               Interface inside (192.168.254.1): Normal
       Other host: Secondary - Standby
               Active time: 0 (sec)
               Interface stateful (192.168.199.2): Normal
               Interface dmz2 (0.0.0.0): Link Down (Shutdown)
               Interface outside (192.168.1.2): Normal
               Interface inside (192.168.254.2): Normal

Remember that you should make configuration changes to only the active unit, because those changes are replicated in only one direction—active to standby. Active-active failover takes this one step further—configuration changes to the system execution space or the admin context must be made on the firewall unit that is active for failover group 1. If you attempt to configure the standby unit, the standby firewall displays a warning that the configurations are no longer synchronized.

In the case of active-active failover, this gets a little more complicated. Now, a firewall can be either the primary or secondary unit, but it can be active in some contexts while being standby in others. You can find out which failover group the firewall is active in by using the show failover command in the system execution space, as shown in the following example:

Firewall# show failover
Failover On
Cable status: N/A - LAN-based failover enabled
Failover unit Primary
Failover LAN Interface: Failover Ethernet0/2 (up)
Unit Poll frequency 3 seconds, holdtime 9 seconds
Interface Poll frequency 15 seconds
Interface Policy 1
Monitored Interfaces 3 of 250 maximum
Group 1 last failover at: 13:10:46 EST Dec 9 2004
Group 2 last failover at: 13:10:04 EST Dec 9 2004
 This host:    Primary
 Group 1       State:                Active
               Active time:          149706 (sec)
 Group 2       State:                Standby Ready
               Active time:          121650 (sec)
[output omitted]
 Other host:   Secondary
 Group 1       State:                Standby Ready
               Active time:          120936 (sec)
 Group 2       State:                Active
               Active time:          148995 (sec)

If you cannot enable failover, check the status of your firewall license with the show activation-key or show version command. The following example shows the results for an ASA firewall:

Firewall# show activation-key
Serial Number:  JHX1114L04Z
Running Activation Key: 0x7411c36d 0x639a94fa 0xa3f0b034 0x913c0374 0x3f3632ba
License Features for this Platform:
Maximum Physical Interfaces : Unlimited
Maximum VLANs               : 100
Inside Hosts                : Unlimited
Failover                    : Active/Active
VPN-DES                     : Enabled
VPN-3DES-AES                : Enabled
Security Contexts           : 5
GTP/GPRS                    : Enabled
VPN Peers                   : Unlimited
WebVPN Peers                : 2
Advanced Endpoint Assessment: Disabled
This platform has an ASA 5510 Security Plus license.
The flash activation key is the SAME as the running key.
Firewall#

Displaying the Current Failover Status

You can use the following command to display a summary of the current failover status:

Firewall# show failover

The output from this command displays the configured failover state (on or off), along with failover cable status, the last failover date and time, the failover roles (primary or secondary) for both units, the firewall role (active or standby) for both units, the status of each configured interface, and the statistics for the stateful failover link (if configured).

An ASA or FWSM also presents this information for each failover group (1 and 2). Within each group, the status of each of the security contexts and its allocated interfaces are shown. For example, the system execution space on the primary firewall has the following output. Notice that at a glance, the shaded text gives a snapshot of every state and role involved in failover:

Firewall# show failover
Failover On
Cable status: N/A - LAN-based failover enabled

Failover unit Primary
Failover LAN Interface: Failover Ethernet2 (up)
Unit Poll frequency 3 seconds, holdtime 9 seconds
Interface Poll frequency 15 seconds
Interface Policy 1
Monitored Interfaces 3 of 250 maximum
Group 1 last failover at: 13:11:02 EST Dec 7 2004
Group 2 last failover at: 15:01:04 EST Dec 7 2004

 This host:    Primary
 Group 1       State:          Active
               Active time:    7536 (sec)

 Group 2       State:          Standby Ready
               Active time:    663 (sec)

               admin Interface outside (192.168.93.138): Normal
               CustomerA Interface outside (192.168.93.139): Normal
               CustomerA Interface inside (192.168.200.10): Normal
 (Not-Monitored)
               CustomerB Interface outside (192.168.93.143): Normal
               CustomerB Interface inside (192.168.220.11): Normal
 (Not-Monitored)

 Other host:   Secondary
 Group 1       State:          Standby Ready
               Active time:    0 (sec)
 Group 2       State:          Active
               Active time:    6879 (sec)

               admin Interface outside (128.163.93.141): Normal
               CustomerA Interface outside (128.163.93.142): Normal
               CustomerA Interface inside (192.168.200.11): Normal
 (Not-Monitored)
                CustomerB Interface outside (128.163.93.140): Normal
               CustomerB Interface inside (192.168.220.10): Normal
 (Not-Monitored)

Stateful Failover Logical Update Statistics
       Link : Failover   Ethernet2 (up)
       Stateful Obj      xmit         xerr       rcv        rerr
       General           135508407      7        53412868   0
       sys cmd           266210         0        266207     0
       up time           14             0        0          0
       RPC services      0              0        0          0
       TCP conn          123228648      0        47758798   0
       UDP conn          663934         0        448445     0
       ARP tbl           6              0        0          0
       Xlate_Timeout     617643         0        556745     0

Logical Update Queue Information
                         Cur            Max      Total
       Recv Q:           0              35       7519538
       Xmit Q:           0              1        18562497
Firewall#

The Stateful Failover Logical Update Statistics represent the number of connection or table synchronization update messages that the firewall has transmitted and received. The Logical Update Queue Information shows the number of stateful update messages that have been queued as they have been transmitted to or received from the failover peer. Nonzero values mean that more updates have been queued than could be processed. A large value might indicate that the stateful failover bandwidth needs to be increased, usually by choosing a faster interface.

To see the failover status information for just one failover group, you can use the following command:

Firewall# show failover group {1 | 2}

On an ASA or FWSM, you can also get a quick summary of the failover status with the following command:

Firewall# show failover state

In the following example, the firewall is shown to be the primary unit with the active role, and the other peer is the secondary in standby. The configurations are synchronized, and the interface MAC addresses have been set according to the primary and secondary burned-in addresses. If one of the units had failed, a reason would be shown:

Firewall# show failover state
====My State===
Primary | Active |
====Other State===
Secondary | Standby |
====Configuration State===
       Sync Done
====Communication State===
       Mac set
=========Failed Reason==============
My Fail Reason:
Other Fail Reason:
Firewall#

Displaying the LAN-Based Failover Interface Status

An FWSM or an ASA cannot display LAN-based failover interface statistics. However, a firewall running PIX 6.3 displays this information if you use the following command:

Firewall# show failover lan [detail]

For example, in the following output, the LAN-based failover interface is called lan-fo. It uses 192.168.1.1 and 192.168.1.2 on the two peers:

Firewall# show failover lan
LAN-based Failover is Active
       interface lan-fo (192.168.1.1): Normal, peer (192.168.1.2): Normal
Firewall#

You could see much more detail about the interface activity by adding the detail keyword, as shown in the following example. Notice that statistics are kept for the number of failover messages sent, received, dropped, and so on, as well as the response times for message exchanges with the failover peer (the shaded text):

Firewall# show failover lan detail
LAN-based Failover is Active
This PIX is Primary
Command Interface is lan-fo
My Command Interface IP is 192.168.198.1
Peer Command Interface IP is 192.168.198.2
My interface status is Normal
Peer interface status is Normal
Peer interface down time is 0x0
Total cmd msgs sent: 107856, rcvd: 107845, dropped: 1, retrans: 8, send_err: 0
Total secure msgs sent: 147375, rcvd: 147301
bad_signature: 0, bad_authen: 0, bad_hdr: 0, bad_osversion: 0, bad_length: 0
Total failed retx lck cnt: 0
Total/Cur/Max of 52719:0:3 msgs on retransQ, 52718 ack msgs
Cur/Max of 0:7 msgs on txq
Cur/Max of 0:34 msgs on rxq
Number of blk allocation failure: 0, cmd failure: 0, Flapping: 0
Current cmd window: 3, Slow cmd Ifc cnt: 0
Cmd Link down: 17, down and up: 0, Window Limit: 17266
Number of fmsg allocation failure: 0, duplicate msgs: 0
Cmd Response Time History stat:
< 100ms:         52681
100 - 250ms:     12
250 - 500ms:     13
500 - 750ms:     12
750 - 1000ms:    0
1000 - 2000ms:   4
2000 - 4000ms:   1
> 4000ms:        3
Cmd Response Retry History stat:
Retry 0 = 52719, 1 = 4, 2 = 1, 3 = 1, 4 = 1
[output truncated]

Displaying a History of Failover State Changes

A firewall keeps a running history of each time its failover state changes. Although the history events are not recorded with a timestamp, the sequence of events can still be useful information. For example, if failover did not come up correctly, you could trace through the history to see the sequence of state changes and the cause for each. You can see the history with the following command:

Firewall# show failover history

For example, the following output shows the failover state change history for a firewall running in multiple-context mode. Failover groups 0 (for system execution space failover), 1, and 2 are listed, because failover operates independently in each group. This sequence of state changes occurred as failover was configured for the first time. During the No Active unit found changes, the secondary peer had not yet been configured for failover.

Firewall# show failover history
==========================================================================
Group     From State               To State                 Reason
==========================================================================
0         Active Applying Config   Active Config Applied    No Active unit found
0         Active Config Applied    Active                   No Active unit found
1         Disabled                 Negotiation              Failover state check
2         Disabled                 Negotiation              Failover state check
2         Negotiation              Cold Standby             Detected an Active mate
1         Negotiation              Just Active              No Active unit found
1         Just Active              Active Drain             No Active unit found
1         Active Drain             Active Applying Config   No Active unit found
1         Active Applying Config   Active Config Applied    No Active unit found
1         Active Config Applied    Active                   No Active unit found
2         Cold Standby             Sync Config              Detected an Active mate
2         Sync Config              Sync File System         Detected an Active mate
2         Sync File System         Bulk Sync                Detected an Active mate
2         Bulk Sync                Standby Ready            Detected an Active mate
2         Standby Ready            Just Active              Set by the CI config cmd
2         Just Active              Active Drain             Set by the CI config cmd
2         Active Drain             Active Applying Config   Set by the CI config cmd
2         Active Applying Config   Active Config Applied    Set by the CI config cmd
2         Active Config Applied    Active                   Set by the CI config cmd
2         Active                   Standby Ready            Set by the CI config cmd
==========================================================================
Firewall#

Debugging Failover Activity

Table 8-1 summarizes some of the commands you can use to generate debugging information about firewall failover operation.

Table 8-1: debug Commands Relevant to Firewall Failover Operation
Open table as spreadsheet

Command	Display Function
debug fover cable	Failover cable status
debug fover {rx | tx}	Failover messages parsed or sent (serial cable only)
debug fover {rxip | txip}	Failover hello messages received or sent on all interfaces
debug fover fmsg	Stateful failover memory activity
debug fover {get | put}	Stateful failover packets received from or sent to the other unit (not available in PIX 7.x)
debug fover sync	Configuration command replication
debug fover switch	Health monitoring activity
debug fover ifc	Interface health polling

Tip

Commands using the debug keyword produce real-time output for troubleshooting purposes. To see these messages, you must first enable logging output to the firewall console (logging console), to a Telnet or SSH session (logging monitor), to a logging buffer (logging buffered), or to a Syslog server (logging host). The debug output also must be sent to the Syslog destination with the logging debug-trace configuration command. See Chapter 10, “Firewall Logging,” for more information.

Monitoring Stateful Failover

As soon as stateful failover is enabled, you should make sure your stateful failover interface is not being overrun with stateful information packets. In other words, verify that the stateful interface bandwidth is sufficient for the load. Otherwise, information about some active connections will not be passed from the active to the standby firewall. If a failover occurs, these unknown connections are terminated.

In single-context mode, you can make a quick manual estimate by using the show traffic command. Unfortunately, this command shows only cumulative values collected since the traffic counters were last cleared. For the packets-per-second and bytes-per-second values, a running average is computed since the counters were last cleared.

However, you can issue the clear traffic command on the active firewall to clear the counters, wait 10 seconds, and issue the show traffic command. You should do this during a peak load time so that you see a snapshot of the busiest stateful information exchange. The following example shows how this is done:

Firewall# clear traffic
[wait 10 seconds]
Firewall# show traffic
stateful:
       received (in 9.050 secs):
               3 packets       395 bytes
               0 pkts/sec      43 bytes/sec
       transmitted (in 9.050 secs):
               84 packets      98682 bytes
               9 pkts/sec      10904 bytes/sec
[output deleted]

In multiple-context mode (active-active failover), things get a little more difficult. The interface used for stateful failover is defined and configured only in the system execution space, where there is no show traffic command. (That command is available in each security context; however, the stateful failover interface is not!)

To gauge the stateful failover interface usage, you can use the show interface command instead. Issue that command and note the number of bytes shown. (This is a cumulative total, not a bytes-per-second rate.) Then, wait 10 seconds and issue the command again. Note the new byte count, subtract the two, and divide by 10. This gives you an estimate of the bytes per second being sent and received over the stateful interface.

You can also use ASDM to generate statistics or a utilization graph of a stateful LAN interface. Running the graph over a period of time shows you the maximum bit rate that has been used to transfer stateful information. Figure 8-9 shows a sample ASDM graph.

Figure 8-9: Using ASDM to Gauge Stateful Failover Traffic

Finally, the firewall performance itself affects the stateful failover operation. As stateful messages are generated, they are put into 256-byte memory blocks and placed in a queue before being sent to the failover peer. If the firewall cannot generate and send the stateful messages as fast as they are needed, more memory blocks are used. Although the firewall can allocate more 256-byte blocks as needed, the supply of these blocks can be exhausted in an extreme case.

You can use the show blocks command as a gauge of the stateful failover performance. Over time, the 256-byte block “CNT” value should remain above 0. If it continues to hover around 0, the active firewall cannot keep the connection state information synchronized with the standby firewall. Most likely, a higher-performance firewall is needed.

Manually Intervening in Failover

When the firewalls in a failover pair detect a failure and take action, they do not automatically revert to their original failover roles. For example, if the primary firewall is active and then fails, it is marked as failed, and the secondary firewall takes over the active role. After the primary unit is repaired and returned to service, it does not automatically reclaim the active role (unless it has been configured to preempt active control).

You might occasionally find that you need to manually intervene in the failover process to force a role change or to reset a failover condition. The commands discussed in the following sections should be used from configuration mode in single-context mode and in the system execution space in multiple-context mode.

Forcing a Role Change

Ordinarily, the firewalls fail over to each other automatically, without any intervention. However, they do not automatically fail back to their original roles. If for some reason you need to force one unit to become active again, you can use the following privileged EXEC command:

Firewall# [no] failover active [group {1 | 2}]

You can also force a unit into standby mode with the no failover active command.

With active-active failover, you can specify the failover group (1 or 2) that will become active. For example, suppose the secondary firewall should be standby for failover group 1 and active for failover group 2. After a failure, it ends up in standby mode for both failover groups, as shown in the following output:

Firewall# show failover
Failover On
Cable status: N/A - LAN-based failover enabled
Failover unit Primary
Failover LAN Interface: Failover Ethernet0/2 (up)
Unit Poll frequency 3 seconds, holdtime 9 seconds
Interface Poll frequency 15 seconds
Interface Policy 2
Monitored Interfaces 3 of 250 maximum
Group 1 last failover at: 10:29:18 EST Jan 30 2005
Group 2 last failover at: 16:18:28 EST Mar 9 2005
 This host:    Secondary
 Group 1       State:          Standby Ready
               Active time:    3311601 (sec)
 Group 2       State:          Standby Ready
               Active time:    3304092 (sec)

To restore the secondary unit to the active role for failover group 2, you could take two different approaches:

• Force the primary unit (currently active) into the standby role by using the no failover active group 2 command

• Force the secondary unit (currently standby) into the active role by using the failover active group 2 command

Resetting a Failed Firewall Unit

If a firewall has been marked as failed but has been repaired or its connectivity restored, you might have to manually “unfail” it or reset its failover role. You can use the following privileged EXEC command:

Firewall# failover reset [group {1 | 2}]

You can use this command on either the active or failed unit. If it is issued on the active unit, the command is replicated to the failed unit, and only that unit’s state is reset. In active-active failover, you can add the group keyword and failover group number for the firewall role to be reset.

Reloading a Hung Standby Unit

Sometimes, an active and standby firewall can communicate over a failover connection but cannot synchronize their failover operation. In this case, you can manually force the standby unit to reload and reinitialize its failover role with the following command:

Firewall# failover reload-standby

After the reload, it should resynchronize with the active unit.

Executing Commands on a Failover Peer

Although two firewalls can be configured as a failover pair, they still support their own administrative sessions independently. For example, you can connect to the active unit and enter commands, make configuration changes, and so on. The same is true of the standby unit, where you can connect and do everything except make configuration changes.

Sometimes you might find yourself connected to one unit when you would like to do something on the other unit. Normally you would have to open up an administrative connection to the other unit and enter your commands there. Beginning with ASA 8.0(1), you can open a single session and enter commands that are passed to the failover peer and evaluated there.

Use the following EXEC command to send a command string to the appropriate failover unit:

Firewall# failover exec {active | standby | mate} cmd_string

Regardless of the unit to which you are connected, you can send a command line, cmd_string, to any of the following failover pair units:

• active— The command is sent to the current active unit, where it is executed and also replicated to the standby unit.

• standby— The command is sent to the current standby unit, where it is executed; it is not replicated to the active unit.

• mate— The command is sent to the mate or peer of the unit, where the command is entered.

For example, suppose you are connected to the active unit, where you can display its activation key with the show activation-key command. Then, without opening a second connection to the standby unit, you can see the standby unit’s flash file system by sending it the same command, too, as in the following example:

Firewall# show activation-key
Serial Number:  848020184
Running Activation Key: 0x7111c56d 0x689a94fa 0xa4f0b064 0x910c0474 0xcf36c2ba
Licensed features for this platform:
Maximum Physical Interfaces   : 6
Maximum VLANs                 : 25
Inside Hosts                  : Unlimited
Failover                      : Active/Active
VPN-DES                       : Enabled
VPN-3DES-AES                  : Enabled
Cut-through Proxy             : Enabled
Guards                        : Enabled
URL Filtering                 : Enabled
Security Contexts             : 5
GTP/GPRS                      : Enabled
VPN Peers                     : Unlimited
This platform has an Unrestricted (UR) license.
The flash activation key is the SAME as the running key.
Firewall#
Firewall# failover exec standby show activation-key
Serial Number:  857206105
Running Activation Key: 0xb116c169 0xd0f16030 0x44600c98 0xb138e8b0 0x8322dc9f
Licensed features for this platform:
Maximum Physical Interfaces   : 6
Maximum VLANs                 : 25
Inside Hosts                  : Unlimited
Failover                      : Active/Active
VPN-DES                       : Enabled
VPN-3DES-AES                  : Enabled
Cut-through Proxy             : Enabled
Guards                        : Enabled
URL Filtering                 : Enabled
Security Contexts             : 5
GTP/GPRS                      : Enabled
VPN Peers                     : Unlimited
This platform has an Unrestricted (UR) license.
The flash activation key is the SAME as the running key.
Firewall#

You should keep a couple things in mind about remotely executing commands:

• The command strings are sent over the LAN-based failover connection.

• The command strings are sent to the target failover unit, but the command output is always returned to the unit where the failover exec command was entered.

You can also use the failover exec command to remotely execute configuration commands. This can be handy if you need to make a small change, but are not currently connected to the appropriate unit. However, making configuration changes in this way is somewhat cumbersome.

You should never try to execute configuration commands remotely on a standby unit. Configuration changes must always be made on the active unit, while the failover feature takes care of replicating the changes to the standby unit automatically.

Tip

If you do decide to enter configuration commands remotely, be aware that the source and target firewalls each maintain their own command mode. For example, the firewall you enter the failover exec command into might be in configuration mode, but the target firewall is not necessarily in the same mode—it might be at the normal EXEC level prompt. In that case, the command you try to execute remotely might not work. To see what mode the target firewall is currently at, use the show failover exec {active | standby | mate} command.

In multiple context mode, remote command execution takes place only in the context to which you are currently connected. In other words, the changeto command cannot be sent to a target firewall. To send a command to the target in a specific context, you have to first change to that context locally and then send the command.

8-5: Upgrading Firewalls in Failover Mode

Upgrading the operating system on a single standalone firewall is straightforward. You download a new image to the firewall, save the running configuration, and reload the firewall. The Auto Update feature can also be used to automate the upgrade process. Obviously, this should all be done during a scheduled maintenance time in your network, because the reload interrupts network connectivity.

A failover pair of firewalls is slightly more complicated, because both firewalls must be running exactly the same release of code at all times. If the code release differs between the two firewalls, failover is disabled. This causes each firewall to run independently, each thinking that the other has either failed or is incompatible for failover.

Firewall platforms running releases at or greater than FWSM 2.0(1) or PIX 7.0(1) are exceptions. These versions offer a “hitless upgrade” or “zero downtime upgrade” feature that allows failover to continue operating even if the two firewalls are running different releases of the software image. However, you should make sure the failover peers are upgrading in one of the following scenarios to achieve a zero downtime upgrade:

• Upgrade from one maintenance release to another (8.0[1] to 8.0[4], for example).

• Upgrade from one minor release to the next minor release increment (8.0[1] to 8.1[1], for example); don’t skip minor releases.

• Upgrade from the last minor release of one major release to the first minor release of the next major release (7.2[4] to 8.0[1], for example).

Manually Upgrading a Failover Pair

In an active-standby pair, only one firewall is active, and both units must be running identical software releases. The only exception is during a zero downtime upgrade, where the firewalls might be running images with incrementally different release numbers.

In a nutshell, as long as one of the two firewalls is operating in active mode, it continues inspecting traffic and updating state information (connections, translations, and so on) to the standby unit. The idea is to juggle the active and standby roles so that the standby unit is always the one being upgraded. Whichever unit has the active role at any time will always have the newer, upgraded image and will continue to forward traffic and maintain the state information.

An active-active failover pair is only slightly different, because either unit might be running a failover group of security contexts in active mode. Here again, the idea is to move all active roles to one of the units—the primary unit—while upgrading the other.

You should carefully follow these basic steps for a zero downtime upgrade:

Step 1	Download the new image to both active and standby units.
Step 2	(Active-Active) Force all active roles onto the primary unit.
Step 3	Reload the standby unit to run the new image.
Step 4	Swap active and standby roles.
Step 5	Reload the new standby unit to run the new image.
Step 6	Swap active and standby roles again, to return the originally active unit to the active role.

You can use the following steps to upgrade the operating system on a failover pair:

1. Download the new OS image into both firewall units.

   You can use any supported image transfer method, such as TFTP or FTP. Be certain to monitor the image file download to be certain that each firewall has written a complete new image into its flash memory. As soon as the OS image is stored in flash, you can see it in the flash file system directory, but you cannot view or verify it. You can either watch as it is being downloaded and written or watch as it is being run after a reload.

   The OS image file is not automatically replicated from the active unit to the standby unit, or from the primary unit to the secondary unit in an active-active scenario. Instead, you must connect to each firewall and download the file.

   Make sure the boot system command reflects the new image filename in flash memory so that the firewall boots the correct file after it reloads. This command is replicated to the standby unit as well, so both units boot from the same image filename and location.

2. Open a session to the active firewall unit.

   While the two firewalls are operating as a failover pair, you cannot make configuration changes to each one separately. Configuration is allowed only on the active unit. However, you can download a new OS image without disrupting the failover status.

   By keeping a session open to the active unit, you can keep any eye on firewall operation during the upgrade process. As well, you can control the failover roles and reload the standby unit remotely.

   Tip

   For the remainder of the steps in this upgrade process, you should stay connected to the unit that is currently active. From that unit, you can do all of the functions required during a zero downtime upgrade. For an active-active failover pair, you should connect to the primary unit and stay connected there. In Step 3, the primary unit takes over the active role for all contexts, effectively becoming the active unit.

   Remember that both firewalls have the same host name, because it is replicated between them. Therefore, you cannot recognize the primary unit by its host name in the prompt. To determine which failover unit you are connected to (primary or secondary), use the show failover command on each. For example, if the active unit happens to be the primary unit, the firewall shows something like this:

   Firewall# show failover
   Failover On
   Cable status: Normal
   Reconnect timeout 0:00:00
   Poll frequency 15 seconds
   Last Failover at: 04:57:40 EST Sun Oct 26 2003
           This host: Primary - Active
                  Active time: 245010 (sec)
                  Interface stateful (192.168.199.1): Normal
                  Interface dmz2 (127.0.0.1): Link Down (Shutdown)
                  Interface outside (192.168.110.65): Normal
                  Interface inside (192.168.254.1): Normal
          Other host: Secondary - Standby
                  Active time: 0 (sec)
                  Interface stateful (192.168.199.2): Normal
                  Interface dmz2 (0.0.0.0): Link Down (Shutdown)
                  Interface outside (192.168.110.71): Normal
                  Interface inside (192.168.254.9): Normal
   [output omitted]
   Firewall#

   It does not really matter whether the primary unit is in active or standby mode. You should only be concerned about connecting to the unit that is currently active, so you can upgrade the standy unit.

   By looking at the show failover command output now, you can also verify that the failover pair is working properly. Failover must be working on both units to replicate enough dynamic information to have a zero downtime upgrade.

3. Save the running configuration on the primary unit:

   Firewall# write memory
   

   or

   Firewall# copy running-config startup-config
   

   Always make sure you have saved the current running configuration before a reload. The running configuration is dynamically updated from the active unit to the standby unit as commands are entered. However, you must manually save the running configuration to Flash with either of the preceding commands. Doing this causes the same command to be run on the standby unit as well.

   On an active-active failover pair, where you have multiple security contexts, you should use the following command to save all context configurations:

   Firewall# write memory all
   

   Copying the running configuration to an external location is also a good idea. For example, you can copy it to a TFTP server with the copy running-config tftp:[[//location][/pathname]] command. In case the firewall pair has a catastrophic failure, you will have a copy of the configuration to load into a replacement unit.

4. (Active-Active only) Force all active roles onto the primary unit:

   Firewall# failover active
   

   The primary unit might currently have the active role for only some of the contexts; if so, the secondary unit has the active role for the remaining contexts. To perform a zero downtime upgrade, the primary unit must have the active role for all contexts.

   When the failover active command is entered on the primary unit, it immediately assumes the active role for all configured contexts.

   For the remainder of the upgrade process, the primary unit is referred to as the “active” unit and the secondary unit as the “standby” unit.

5. Force the standby unit to reload:

   Firewall# failover reload-standby
   

   From the active unit, you can reload the standby unit remotely without having to connect to it. As soon as the standby unit begins the reload procedure, you will not be able to watch it boot up unless you are connected to its console port.

6. Wait until the standby unit reloads.

   You should wait until you see that the standby unit is back in full operation and failover is working properly before moving on to the next step. You can do this by periodically checking the failover status with the show failover command.

7. Force the active unit to take the standby role:

   Firewall# no failover active
   

   On the active unit, this command forces it to immediately assume the standby failover role. The previous standby unit immediately assumes the active role.

   Now the two failover peers have exchanged roles—the unit with the upgraded image is now the active unit and carries on all firewall operations.

8. Reload the new standby unit:

   Firewall# reload
   

   On the new standby unit (the one that was originally active in Step 2), force a reload. Now that the other peer is running as the active unit, the new standby unit is free to be reloaded without impacting firewall operation. As soon as the standby unit is reloaded, it boots up with the upgraded image.

9. (Optional) Return the active role to the original firewall:

   Firewall# failover active
   

   At this point, the original failover roles have been reversed—the original active unit is now the standby unit. If you want to return the active role to the firewall that was originally active in Step 2, you can use the failover active command to force the roles to change immediately.

   For an active-active failover pair, you can use the failover active group group_id command to keep the active role for failover group group_id on the primary unit.

   You can also use the no failover active group group_id to move the active role for the failover group to the secondary unit.

Tip

You should always upgrade OS images in an incremental fashion so that the zero downtime upgrade feature can work properly. The previous procedure is meant to upgrade an increment from one maintenance release to another or from one minor release to another. However, you can repeat the same procedure over and over if you need to upgrade to a code level that is several releases later. Make sure you use each pass through the upgrade procedure for each increment of the image release number.

Automatically Upgrading a Failover Pair

In Chapter 4, “Firewall Management,” in Section “4-4: Firewall Management,” in Section “Automatic Updates with an Auto Update Server,” firewalls can be configured to automatically poll and download updated image files from an Auto Updates Server (AUS). Normally, these are standalone firewalls, ones not operating as part of a failover pair.

Beginning with ASA 8.0(1), you can configure a failover pair of firewalls to work with AUS so that they both receive an updated image automatically. The firewalls must be operating in single-context mode to work with AUS.

To use AUS with failover, you should configure the Auto Update feature on the primary failover unit only. The primary unit handles all communication with the AUS and manages how the failover units are reloaded after they get their new image files. If the primary unit is not active at the time of an AUS poll, it automatically assumes the active role.

Normally, image files are not replicated as part of the failover process; image files must be downloaded to each failover unit separately. When AUS is used, however, the primary unit downloads an updated file to itself and then replicates it to the secondary unit automatically.

After an updated image file is downloaded to both of the failover units, the firewalls are reloaded under the following conditions:

• If the updated image file is an incremental upgrade, where a zero downtime upgrade is possible, then the standby unit is reloaded first. After it is back up, the failover roles are reversed, and the previously active unit is reloaded automatically.

• If a zero downtime upgrade is not possible, both units are reloaded at the same time. This results in an abrupt stop to traffic passing through the firewall pair.

Refer to Section 4-4 in Chapter 4 for complete information about configuring the Auto Update feature. Make sure you force the primary failover unit to take the active role if it is not already taking it. Then apply the configuration steps to the primary failover unit only; the configuration commands are then replicated to the secondary unit.