Chapter 4: Advanced WAN Services Design Considerations (Part03)

VPLS Overview

VPLS is a multipoint architecture that connects two or more customer devices using Ethernet bridging techniques over an MPLS network. In VPLS, the P-network emulates an IEEE 802.1 Ethernet bridge, with each EMS being analogous to a VLAN.

VPLS is an architecture that is still being defined. There are two RFCs that are distinct and incompatible with one another:

• RFC 4761: Virtual Private LAN Service (VPLS). Using BGP for Auto-Discovery and Signaling is a standard proposed by Juniper.

• RFC 4762: Virtual Private LAN Service (VPLS). Using Label Distribution Protocol (LDP) Signaling is a standard proposed by Cisco.

One of the major differences in the standard is in the VPLS provider edge (PE) discovery process, or how VPLS PE devices find each other, communicate capabilities, and perform tasks such as preprovisioning EVCs.

VPLS Architecture Model

In the VPLS architecture model, UPE devices act as IEEE 802.1D standard bridges or switches. They are interconnected in a full mesh of pseudowires (PW). In Figure 4-10, the PWs cross a routed MPLS and IP provider backbone. From the point of view of the UPEs, these PWs are just Ethernet connections to another switch.

Figure 4-10: VPLS Architecture Model

VPLS will self-learn source MAC address to port associations, and frames are forwarded based on the destination MAC address. If the destination address is unknown, or is a broadcast or multicast address, the frame is flooded to all ports associated with the virtual bridge.

In the event of a provider outage, IP rerouting rapidly restores PW connectivity. In such a case, no MAC aging and relearning is needed.

To simplify processing, the VPLS core does not use STP. Instead, it uses split-horizon forwarding, so that Ethernet frames are not sent back out on the same PW on which they were received.

Broadcast and multicast traffic would always be flooded in VPLS.

VPLS in the Enterprise

VPLS is used as an enterprise WAN connectivity service.

VPLS looks like an Ethernet switch to the customer with the same inherent Layer 2 core issues:

• Stability of the network as it grows: Experience shows that purely STP-based VPLS does not scale gracefully. VPLS based on MPLS scales much better.

• Impact of network outages: A customer should ask about what happens to their traffic in the event of an outage, and receive enough details to justify the quoted speed of convergence.

• Multicast and broadcast radiation between sites: Because the VPLS network acts like a switch, customer multicasts and broadcasts sent into the VPLS cloud radiate to all sites of that customer. This can be somewhat controlled by using routers to connect to the VPLS network, but multicasts sent by the router flood to all the customer’s sites.

• Interior Gateway Protocol (IGP) peering scalability: The VPLS network is one broadcast domain, so all attached routers would typically be routing peers. As the number of routing peers increases, the full mesh of connections becomes a scaling issue. Designs using VPLS in a hierarchical fashion should be more scalable.

• Impact of an STP loop of another customer: Because VPLS uses stat muxing, all customers share bandwidth. It is reasonable to ask what the impact of a customer with a spanning-tree loop would be on other customers. If the customer is attached by a Layer 2 switch, all the packets from the loop would necessarily be flooded within the links interconnecting their VPLS sites. If they connect at 1 Gb/s and the provider trunks are 20 Gb/s, this may not be so bad. If the provider links are 2 Gb/s, the impact might be far greater, particularly if EtherChannel is in use. (Deterministic assignment of traffic to channels would cause the traffic of other selected customers to share the problem customer’s channels.)

A VPLS customer might also want to conduct due diligence to verify that the provider is aware of and has implemented adequate Layer 2 security measures.

Hierarchical VPLS Overview

Hierarchical VPLS (H-VPLS) provides scaling by only interconnecting the core MPLS NPE routers with a full mesh of PWs. The many UPE VPLS devices are then connected hierarchically by PWs to the NPE devices, not to each other. When there is redundancy, as shown in Figure 4-11, the software in the UPE blocks the PWs to all but the highest NPE IP address. H-VPLS partitions the network into several edge domains that are interconnected using an MPLS core.

Figure 4-11: Hierarchical VPLS Overview

One advantage of the H-VPLS approach for the service provider is that the core of the network is an MPLS network, which may also be used for the transport of Layer 3 MPLS VPNs and other traffic. The MPLS core also serves to limit any edge spanning-tree domains, speeding up STP convergence and reducing any potential instability.

The physical topology of Ethernet Edge H-VPLS (EE H-VPLS) can comprise point-to-point Ethernet connections, or Ethernet rings using an STP to provide redundancy and loop avoidance. Other edge architectures use an aggregation layer between the UPE and NPE, or use Ethernet over SONET/SDH (EoS), or RPR as a transport between the UPE and NPE.

H-VPLS provides an extremely flexible architectural model that also enables multipoint Ethernet services (VPLS), and Ethernet point-to-point Layer 2 VPN services and Ethernet access to Layer 3 VPN services.

Scaling VPLS

A service provider VPLS design must address three major scaling factors:

• Scaling of the full mesh of PWs between PE devices: As the number of PE devices scales, each edge device must form an adjacency with all other PE devices. This requires that the edge devices must have the IP address of all remote PEs in its routing table, and also requires the PE to exchange label information with all remote PE devices. That introduces an N – 1 control plane scaling issue.

  H-VPLS helps address this issue by using UPE devices to spread the edge workload across multiple, less-costly devices. The lower number of PWs between the NPE devices helps scale the network by reducing the burden on the core for frame replication and forwarding.

• Frame replication and forwarding: VPLS forwards Ethernet frames using Layer 2 MAC addresses. The operation of VPLS is exactly the same as that found within IEEE 802.1 bridges in that the virtual switch self-learns the source MAC address-to-port associations and forwards frames based on the destination MAC address. If the destination address is unknown, or is a broadcast or multicast address, the frame is flooded to all ports associated with the virtual bridge. H-VPLS needs a lower number of PWs, because only the NPE devices are connected in a full mesh. This helps reduce the burden on the core for frame replication and forwarding.

• MAC address table size: One of the biggest considerations in VPLS provider design is MAC address learning. PE devices need to be capable of handling MAC address tables for many customer devices and many customers. That number is much greater than what a typical enterprise campus switch needs to handle today.

  H-VPLS allows MAC tables to be spread across multiple inexpensive devices to scale the edge. UPE devices need only learn of their local NPE devices and therefore do not need large routing table support. Core NPE devices still need to handle very large MAC address tables. Using MPLS in the core removes the MAC learning requirement from the P devices.

Note

Interconnecting only customer routers and not switches by VPLS would greatly simplify scaling MAC learning, because then only the router MAC addresses would need to be learned in VPLS devices. However, few providers are willing to limit their potential VPLS market by imposing such a requirement.

Some VPLS device vendors use proprietary MAC-in-MAC encapsulation schemes, so that the NPE devices only need learn the MAC addresses of other UPE devices. This approach may also use supporting protocols, reminiscent of ATM LAN Emulation (LANE). At the time of this writing, Cisco devices do not implement these mechanisms.

How well the provider’s design handles all these factors is of interest to the customer. Poor VPLS design can lead to scaling and stability problems as the provider’s network grows. While a customer should not have to be intimately familiar with VPLS designs to evaluate a provider’s service, listening to a service provider’s answers on these topics can provide insight into that provider’s qualifications.

QoS Issues with EMS or VPLS

QoS is relatively easy to provide with point-to-point links. Oversubscription is managed by controlling what is allowed into the point-to-point link.

However, QoS for multipoint networks is harder because it requires coordination between multiple devices with unpredictable and rapidly changing traffic patterns. Careful consideration needs to be taken for interactive services such as video and IP telephony. The need for coordination implies that the service provider provide QoS, because it is not something the customer can layer on top of a connection, nor is it something the customers can just implement for their own use.

The technique of bandwidth slicing can be used by customers in an attempt to compensate for lack of provider QoS. In this approach, each CE device has a slice of the bandwidth to each remote location. Access lists then police traffic to each location, to try to ensure the remote link is not congested. The issue with this approach is that inside the VPLS cloud, Customer A traffic competes not only with Customer A traffic, but with the traffic from other customers, too. In addition, broadcast and multicast traffic fed into the VPLS cloud are received at every service provider edge site, consuming some of the bandwidth, too.

These issues illustrate that if the customer wants QoS, the service provider has to partner to provide it. Customers should expect that QoS is a premium service, and determine bandwidth levels for each QoS class to help manage QoS.

EMS or VPLS and Routing Implications

One concern when using OSPF is that the multiaccess network may not have consistent broadcast or multicast performance. If some sites experience greater packet loss levels, OSPF processing may consume more router CPU. In the extreme case, packet loss or delay might cause significant levels of OSPF instability. This is currently being seen in some large VPN networks that use generic routing encapsulation (GRE) over IP Security (IPsec) for OSPF routing.

Another concern is that retransmissions or flapping status with regard to one site might consume significant CPU resources on all its peer routers. As discussed in Chapter 3, “Developing an Optimum Design for Layer 3,” it is advisable to manage the number of OSPF adjacencies in a full-mesh network, and to use designs to limit the number of adjacencies.

Note

A similar caution applies to Enhanced Interior Gateway Routing Protocol (EIGRP) routing over an EMS or VPLS service. You should avoid having high numbers of peers with EIGRP, too.

VPLS and IP Multicast

In a campus switched network, IP multicast in a VLAN floods to all ports in the VLAN, unless Internet Group Management Protocol (IGMP) snooping is in use. However, IGMP snooping is not an option with VPLS or EMS. A broadcast or multicast frame sent into the VPLS cloud ends up being sent out of the VPLS provider network to every CE device that has a port associated with the virtual switch.

Because the VPLS network has been designed for transparency by emulating a switch, the VPLS PE devices cannot provide any intelligence by delivering multicast only where it is needed. If the CE devices are routers, they will discard undesired multicast packets they receive. IP multicast traffic in MPLS can result in wasted bandwidth and router CPU utilization spent discarding unnecessary traffic.

One conclusion is that VPLS designers with customer networks that have significant amounts of multicast need to use administrative scoping to limit the propagation of multicast packets, or else allow sufficient bandwidth for unnecessary multicast traffic at the edge links.

VPLS Availability

VPLS provides some advantages in the case of P-network outages.

An advantage to using VPLS is that PWs are the underlying technology for the data plane. In the case of a failure in the P-network, traffic will automatically be routed along available backup paths in the P-network. Failover in this case will be much faster than could be achieved with STP.

There is a cost to this fast failover. If redundant PWs are used from redundant PE devices, a failure might require aging of MAC addresses followed by unicast flooding. The resulting lost packets followed by a surge of traffic would have a negative impact on customer traffic.

PW rerouting around outages prevents this potential problem.

Note

Although this section has focused primarily on VPLS, many of the same considerations apply to any Ethernet service offering. Appropriate SLAs for the advanced WAN services are discussed in the “Implementing Advanced WAN Services” lesson in this chapter.

MPLS VPN Overview

As an alternative to Metro Ethernet services, some customers are implementing MPLS VPNs. MPLS VPN services are based on MPLS label paths that are automatically formed based on IP routing. MPLS VPNs experience the same level of stability as exhibited by Layer 3 networks in general. MPLS VPNs can support either Layer 2 transport (typically a long-haul or metro-area Ethernet point-to-point service) or a Layer 3 routed service. The characteristics of the MPLS VPNs vary depending on whether they are implemented at Layer 3 or Layer 2:

• Layer 3 service: Layer 3 MPLS VPNs forward only IP packets. The CE routers become peers of MPLS VPN provider routers. In this case, routing may well be a cooperative venture. Stability of the provider routing, their experience with routing, and speed of provider routing convergence are all valid customer considerations. Layer 3 VPNs can support any access or backbone technology. Service providers can use Layer 3 VPNs as a foundation to provide advanced WAN services.

• Layer 2 service: Layer 2 MPLS VPNs can forward any network protocol based on Layer 2 frames. There is no peering with the provider. Layer 2 service allows customer routers to peer directly with each other without a handoff to SP router. MPLS Layer 2 VPNs provide point-to-point service where the access technology is determined by the VPN type. MPLS Layer 2 VPNs may also be useful for service interworking, or converting Frame Relay or ATM into Ethernet for delivery on a high-bandwidth link at a central site or data center.

The choice of Layer 2 VPN over Layer 3 VPN will depend on how much control the enterprise wants to retain. If an enterprise has a small staff or lacks routing skills, a managed router Layer 3 VPN service puts all CE routers in the hands of one or more providers, and delegates all routing to the provider. Large organizations with considerable in-house routing skills may prefer Layer 2 VPNs because they can maintain control of their Layer 3 policies.

Customer Considerations with MPLS VPNs

There are several considerations for designing customer MPLS VPNs:

• Who does the routing? A major decision when implementing MPLS VPN is who will do the routing. For a simple scenario, the customer may opt to use static routing, where there is no dynamic interaction with the service provider routing. Another option is for the customer to use External Border Gateway Protocol (EBGP), EIGRP, OSPF, or Intermediate System-to-Intermediate System (IS-IS) with the provider, depending on which PE-CE routing protocol the provider supports.

  If the customer redistributes routes learned via MPLS VPN routing into their IGP, these routes may become external routes. In the case of OSPF or EIGRP as the PE-CE routing protocol, the provider may be able to redistribute them as internal routes, which is generally preferable from the customer’s perspective.

• Who manages the CE devices? Depending on the size and routing experience of the customer, they may chose to manage their own CE devices, or buy managed services from the provider. The customer must also determine the level of redundancy needed, depending on whether one or two CE devices will be implemented.

• Should one or two MPLS VPN providers be used? Another key decision is whether to use one or two MPLS VPN providers. Having two providers provides better redundancy than dual-homing to one provider, because two providers are less likely to experience a common failure event. However, two providers can add complexity to the design. If two MPLS VPN providers are used with two CE devices per location, the design needs to support using the appropriate default gateway with the appropriate first-hop redundancy protocol (FHRP) or other mechanism.

• Is QoS needed? If QoS is available from the service provider, the customer needs to decide whether to buy an MPLS service with QoS. Using Layer 3 VPNs allows the customer to implement QoS internally.

• Is IP multicast supported? With Layer 3 MPLS VPN, IP multicast may be supported. Doing IP multicast over a Layer 2 MPLS VPN requires service provider support and might cost extra. It may also require special configuration or working with the provider, particularly to support large IP multicast flows.

Routing Considerations: Backdoor Routes

Backdoor routes need to be considered when designing routing for a Layer 3 MPLS VPN WAN.

If there are internal backdoor routes—for example, a second path between locations through a GRE over IPsec tunnel or other WAN link—the internal route (even over slowed links) will be preferred over the external route. This needs to be taken into account in designing the routing to properly use the Layer 3 MPLS VPN.

In general, sites with one WAN router do not have this problem. When the PE-CE routing protocol is BGP, EBGP has a better administrative distance (AD) than the IGP in use. The potential problem with backdoor routes arises when there are several Layer 3 devices at one site, especially if the WAN routing is split across them. If the provider follows the common practice of handing off routes through EBGP, the customer may end up running EBGP at every site. In addition, the customer may have EBGP with the route reflector over the IPsec VPN.

The service provider may be able to redistribute routes as internal routes for OSPF and EIGRP. Using a separate EIGRP autonomous system for the IPsec VPN is another approach to allow simple redistribution controls and metrics.

As a recommended practice, you should minimize the locations where you implement route redistribution. Route redistribution at many locations can adversely impact network stability.

Routing Considerations: Managed Router Combined with Internal Routing

If a managed router service is purchased, it can become awkward to do your own routing.

You might find with managed routers that you still want to control your own routing, perhaps when your company purchases managed router services from one MPLS provider, and then later purchases a similar service from another organization. If not otherwise specified, both providers may choose to use static routing.

In such a setting, the lack of dynamic routing makes it hard to automate dynamic failover in response to provider failure or problems. One approach is to add another layer of routers at each site, and set up GRE tunnels between these routers, as shown in Figure 4-12. The GRE tunnels will allow an organization to control its own routing and run services such as IP multicast. The providers need to provide enough static routing to deliver packets between the added customer routers.

Figure 4-12: Managed Router Combined with Internal Routing

Several potential issues arise with this approach. The GRE tunnel adds to the cost and complexity of the network. You need another router at each site. It will be more difficult to configure the network, and there are potential performance issues related to the impact of encapsulation and possibly maximum transmission unit (MTU) sizes leading to fragmentation.

Routing Considerations: Managed Router From Two Service Providers

In Figure 4-13, there is a managed CE router service from two different service providers; as a result, the issue of FHRP may arise.

Figure 4-13: Managed Router from Two Service Providers

With two service providers providing parallel paths between your locations, you need to consider what the primary path for specific traffic types is and how to support failover. Hot Standby Router Protocol (HSRP) or another FHRP can be used, but it can be challenging to get your two providers to cooperate with each other to provide the FHRP service.

Note

If your site is large enough to use a router or Layer 3 switch in front of the service provider–managed routers, the FHRP is not an issue. However, if you are managing the Layer 3 device, you probably do not need managed router service.

You should negotiate this design with the service providers at contract time. Getting service providers to cooperate can be easier before signing contracts than after both contracts have been signed.

Implementing Advanced WAN Services

WAN service design involves a partnership between service providers and customers, where the service provider manages services for the customer on an ongoing basis. This section looks at issues that enterprise customers should consider when implementing advanced WAN services. Service providers should also understand these considerations to address customer concerns.

Advanced WAN Service Selection

It is important to know the actual characteristics of advanced WAN services. Sales presentations and marketing literature highlight the best features of a service, but probably do not cover the weak or problem areas. It is up to the customer to ask good questions and do the research. If the service is something you are already using, measure its characteristics. This information can help you, your manager, and the business make an informed decision about supporting your business applications such as IP telephony over that service. When you are considering a new service, you cannot measure its ability to support your business application. You need to make decisions based on the underlying technology, the reputation of the vendor, customer references, trade literature, and other such factors.

One way to mitigate risk with WAN services is to use two providers. When you want to deploy a new technology such as an MPLS VPN, you can implement it on one provider, and continue to use the older service with your other provider in case there are problems. Using two providers also lets you experience their technical and customer service levels. If one is providing inadequate service and not meeting your requirements, it is easier to migrate to the other provider.

In general, an advanced WAN service is a partnership between the service provider and customer. An SLA provides contractual obligations for the service provider. A good WAN contract should have an escape clause that addresses what will occur if the WAN provider consistently fails to meet the SLA or provide reasonable levels of customer service. However, most problems are better solved by working the partner relationship. You should be able to explain issues to your service provider, ask what they will do to make things better, and observe the result. If nothing changes, you have the option of considering another provider.

Business Risk Assessment

Part of a good design is to assess the business risk and base design decisions on how critical the service will be.

Any technology has associated business risks. If a WAN service deployment fails, or if the underlying technology fails, it can have a severe business impact, costing lost business or productivity and amounting to a large cost per hour.

When selecting between providers of advanced WAN services, one factor to consider is risk and the likelihood of problems. It may not be worth saving 10 percent at the risk of repeated outages, or a long outage. You need to consider the chance of such an outage happening. These are hard questions to answer, but a service provider can cite examples such as their experience, how long they have been in the business, and how long they have gone without a major outage on a particular service.

Another approach is due diligence questioning, where you conduct a survey, observe the network operations center (NOC), ask a lot of questions, and in general, try to assess the provider personnel’s level of technical knowledge. There are several questions a customer can ask:

• Does the provider have several experts available for the specific technology?

• How skilled are the lower-level staff?

• How neat is the cabling?

• Are the devices and cables labeled?

• How old is the equipment?

• How big is the provider?

Note

Because the due diligence effort can be so time-consuming, it may only occur with large customers.

Obtaining more than a shallow level of information can be a challenge. Generally, providers do not like to provide these details. Patience and persistence can lead to eventually getting the information you desire.

In general, you should match risk to the purpose of the design. Noncritical internal Internet traffic might be serviced by a low-cost but riskier provider. More critical e-commerce traffic might instead be handled by a more established collocation or Internet provider.

Response to risk always involves trade-offs:

• The time and cost to get more information can be prohibitive.

• The cost of service may make due diligence questioning less practical.

• The value of the data compared to the cost of outage should be considered.

• The likelihood of an outage may not be easy to estimate.

Consultants and resellers can provide their views on risks of various technologies, but the decision of the correct risk versus cost trade-off is really up to the customer.

The scope of a possible outage is another consideration. If a network designer has seen a few Spanning Tree Protocol (STP) loops, he or she may believe that a switch and STP-based provider network is risky, because a loop can bring down the entire STP domain. Because routing outages are usually much more localized, that line of reasoning could lead to preferring an MPLS VPN–based service.

WAN Features and Requirements

Customers need to understand what WAN features are available and what they need to support their organization.

Hasty WAN purchases can result in not getting adequate information. Although it might sound obvious, it can be very useful to make sure all parties agree as to what the requirements are for the new WAN service being purchased. There are several important questions to ask:

• Will the routing work as it does currently, or will changes be required?

• Will existing VLANs be transparently supported across the service provider core?

• What QoS levels can be supported at what cost?

• Is IP multicast available? What is the cost for multicast?

• What level of security is provided to isolate customer traffic? Is there a cost for this service?

• What management services and tools are available at what cost?

Defining these requirements is important because advanced WAN services are not transparent transport connections the way that leased lines, Frame Relay, and ATM are. It is important to verify that traffic fed into the provider network (P-network) will be transported appropriately across the service provider core.

It is important to define what WAN features are standard, and what features are optional upgrades:

• In Layer 3 MPLS VPNs, the service provider’s routing interacts with the customer routing in a secure way. IP multicast has to be handled differently by the provider than unicast routing, so it might not be available, or may cost extra.

• With some Ethernet services, Cisco Discovery Protocol may not work, or may be limited to a specific VLAN.

It is necessary to get the details of any managed service. To one service provider, network management may mean occasional ping tests and some console logging, and to another service provider it means in-depth Simple Network Management Protocol (SNMP) monitoring with access to the SNMP reports. Security services that merely inform you of a threat reported by an intrusion detection system (IDS) are not as useful as an indication of what triggered that alarm and other network activity going on at the time.

In general, buying WAN services requires asking questions to get details:

• What does “managed router” or “managed link” mean? What base services does the provider implement, such as ping testing, SNMP polling for CPU and link utilization, and other functions? What management services are optional upgrades?

• What is the process for changes? What are the costs for changes? How fast do requested configuration changes actually happen?

• Is the managed router cooperatively managed, or are all configuration decisions made by the provider based on some initial design?

• Can I review the configuration of the managed router? Can I poll it via SNMP? Can I receive syslog messages and SNMP traps from it?

• What does “QoS” mean? How many levels are supported, what sort of SLA or bandwidth guarantees are available, and how are the service levels protected and enforced?