General Network Challenges

General Network Challenges - CCIE Section 1.1.c

Section 1.1.c of the CCIE syllabus contains a few over looked subjects. I've already touched on Unicast flooding and Layer 2 asymmetric routing. This sections will talk about out of order packets and the impact of micro bursts.

Out of Order Packets

This is a scenario where packet arrive out of order from when they were sent. A TCP flow can generally handle out of order packets, assuming there are not an abundance and the packets arrive if quick secession (3 ms according to wireshark). UDP flows do not have a mechanism to compensate. If UDP flows are used, the application is responsible for handling the out of order data.

Causes of out of order packets are generally a flow taking multiple paths, asymmetric routing or poor queuing. In the case of multiple paths, this can be most apparent when one of paths experience a delay. Path delay can be caused by a device along a path not CEF routing, process vs interrupt switching, or path saturation.

As discussed in later sections, Cisco CEF aims to reduce the occurrence of out of order packets by load balancing ECMP paths on a per flow basis by default.

References:
http://wiresharkbook.com/tr_samplepages/978-1-893939-97-4ooo174.pdf
https://en.wikipedia.org/wiki/Out-of-order_delivery

Microbursts

A microburst is a very brief period of time, microseconds, where very large amounts of data are sent, generally leading to period of full line-rate transmission, overflowing buffers (on the RX side) and causing short-term packet loss. What makes microbursts special are the difficulty in detection via traditional methods. Traditional monitoring will show that an interface's utilization is normal. This is due to SNMP polling intervals for example. Hints to microbursts can be seen on network interfaces as ignores and/or overruns.

If microbursts are suspected, packet captures can verify and help determine the cause of microbursts. Other than increasing network interface buffers, modifying the offending application can help curb the issue of microbursts.

With the advent of modern cut-through / line rate switching hardware, the impact of microbursts are somewhat mitigated.  Legacy store-and-forward switches, for example, are more susceptible due to interface buffer exhaustion.

References:
http://www.arista.com/assets/data/pdf/7148sx-ixnetwork-microburst.pdf
https://en.wikipedia.org/wiki/Head-of-line_blocking

Unicast Flooding

CCIE - General Network Challenges

Section 1.1.c (i)

The topic of unicast flooding can be one of those overlooked network topics due to its simple and common operation. At best, understanding of the unicast flood process and causes can lead to a more efficient network, yet a lack of understanding can result in a noticeably degraded network.

What are Unicast Floods? To begin, here are a couple definitions:

•  Unicast - transmission intended for a single destination, can be a L2 (destination host MAC address) or L3 (destination host IP address) concept.
•  Unicast Flood - Layer 2. Undesirable behavior of a switch treating a unicast frame as a broadcast frame, flooding out all switchports except the received port. 

When a switch receives a unicast frame destined for an unknown host; more specifically, the destination MAC address of the frame contains a MAC address not stored in the CAM table, the switch will flood the frame out all ports that make up the broadcast domain, save the originating port. The hope is that the destination host will eventually receive the frame and respond, creating a frame of its own, containing its source MAC address that can be used to populate the CAM table of the switch.

Unicast flooding is normal, but undesirable behavior. As mentioned above, it is the result of a lack of information. Often times it only takes a flood of one or two frames for a switch to discover the information required to forward the remaining frames more efficiently. Unicast flooding can really be thought of as a host or endpoint discovery technique.  

The following events, if not compensated for, can cause a unicast flood:

CAM table is full

Probably the simplest and easiest to detect, but the most impactful. If the CAM table of a switch is at capacity, it is unable to learn any new destination MAC addresses a switch will start to flood all unicast frames. This will result in a quick and dramatic impact on network resources, seeing interface utilization spike heavily.

Asymmetric forwarding paths at Layer 2

This is commonly the result of a next hop redundancy protocol between two layer 3 switches acting as the default gateway for host networks. If both the switches present equally valid layer 3 routes, the upstream network will try to leverage all paths. STP will only have a single layer 2 path to any one host. The result is that any traffic inbound on an HSRP standby switch my have a valid host ARP entry, but no associated CAM entry. This switch will be forced to flood every inbound frame until the ARP entry times out. This will result in the standby switch needing to ARP for the destination. This ARP process will serve to not only populate the ARP cache, but also the CAM table.

ARP entry, but no CAM entry

By default ARP timers are longer than the CAM cache timers

== Cisco Default Timers ==

	Minutes	Seconds
CAM Table	5	14,400
ARP Cache	300	240 (4 hours)

If any device needs to forward a frame for which it has an ARP entry (that is an IP to MAC correlation), but no corresponding MAC entry in the CAM table, the forwarding device will be forced to flood the unicast frame. The becomes most apparent with asymmetric forwarding paths, as discussed above. The solution would be to make ARP and CAM timers match; either by:

• Changing CAM timeout to 4 hours/14,400 seconds (recommended for stable networks with a large number of ARP entries)
• Changing ARP timeout to 5 minutes/300 seconds or less (More ARP requests will be generated, maybe better method in smaller networks)
• Matching both ARP and CAM to some arbitrary value (Because, sure why not)

More modern solutions involve the implementation of  technologies such as VSS or vPC to create intelligent, logical blocks in the forwarding path.

STP topology changes

STP topology changes will initiate a TCN (Topology Change Notification) BPDU. These TCNs affect CAM entries on respective interfaces.In the case of traditional 802.1D STP (or PVST+), receiving a TCN on an interface will cause all associated CAM entries to change their aging timers to 15 seconds. 802.1w RSTP (or RPVST) and 802.1s (MST) will age out and flush CAM entries immediately, bypassing the 15 second rule. This will cause an unknown unicast flood until the CAM table is rebuilt. In a stable network, this behavior is acceptable as there should be few TCN BPDUs. Portfast can be used on interfaces known to not participate in STP, such as end hosts, so as to prevent TCNs.

Reference Links:

• General Review
• https://en.wikipedia.org/wiki/Unicast_flood
• Cisco VSS
• http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/VSS30dg/campusVSS_DG/VSS-dg_ch3.html#wp1079669
• HSRP
• http://www.cisco.com/c/en/us/support/docs/ip/hot-standby-router-protocol-hsrp/10583-62.html#t8
• STP
• http://www.ciscopress.com/articles/article.asp?p=336872

Link Aggregation Control Protocol - 802.3ad

So, this is my first real post in what I hope to be a great learning and documentation tool!! I thought I would start off with something that most people are familiar with. It also just so happened that I needed to setup LACP in my lab first. The goal will be to not only review the basics of a protocol/technology, but also a deep dive to levels I've never needed to explore in the real world

What is LACP?  Link Aggregation Control Protocol is a protocol/technique to dynamically bundle many similar Ethernet interfaces into one virtual Port-Channel, or "Po", interface. This bundle can then be handled as a single interface. Po interface numbers do not have to match between member switches. Configurations applied to the Po interface get pushed down to the member interfaces. LACP provides similar functionality to PAgP, the Cisco proprietary cousin. Physical interfaces can be bundled as Layer 2 (generally the Po interface will be a trunk, between switches perhaps) or Layer 3 (an IP address added directly to the Po interface). If a single physical link goes down, the bundle continues to function uninterrupted. Often times the failure isn't even noticeable.

Why use LACP when a statically configured port-channel works great and without the negotiation overhead? LACP aims to protect the network by negotiating a bundle to avoid network degradation. In the event that port-channel is mis-configured, maybe the wrong ports were configured, a static port-channel will blindly forward traffic down whichever link it has decided traffic needs to flow. This is an easy way for a Layer 2 loop to form.

In the same scenario, LACP would not put an interface into a bundle unless it can confirm there is a LACP neighbor on the other side. If LACPDUs time out, the interface is simply removed from the bundle, placed in stand-alone and allowed to function as an independent interface. This means that the switch will see this interface as a unique link and things like STP will kick off. Sure this may not be optimal if for some reason STP decided to block the Po interface, but hey, at least you're not dropping the entire network.

== It is important to keep in mind that the rest of this post will explore LACP from a Cisco perspective. LACP is vendor agnostic so the concepts should be the same, but configurations and commands will be different of course. ==

Definitions

• EtherChannel - Cisco term for a group of bundled physical interfaces
• Port-Channel - Cisco's logical representation of an EtherChannel. This is where configurations are made

Good LACP Reference Links

• IEEE 802.1AX-2014
• Find LACP in section 6.4
• The current IEEE standard, replacing 802.3ad
• Some information about LACP can be found in 802.3-2012, but nothing to this degree
• Cisco resource on LACP for IOS 12.2
• Great EtherChannel post by SCLabs

Basic LACP Concepts

Ethernet Types

• LACP can only be used to bundle Ethernet interfaces of the same type. 
• For example, a 100/full interface cannot be bundled with a 1000/full or 100/half interface. Speed an duplex have to match for each member interface. 
• Note, as long as the above holds true, it doesn't matter which physical interfaces numbers are used.
• LACP supports up to 16 total interfaces in a bundle, with 8 of those interfaces being active. 
• Additional bundled interfaces will be in "hot-standby".

LACP Modes

There are two modes to negotiate LACP, active and passive. (I've read about Cisco saying there is an "off" mode on a test, but that doesn't make any sense to me. If you turn it off, it isn't a "mode"... It is just off and not running.) Keep in mind, these aren't the only possible port-channel modes, just the modes specific to LACP negotiation.

• Active mode - will send and listen for LACPDUs.
• Passive mode - will only listen for LACPDUs.

This means that you need at least one side of a particular link in a bundle to be configured as Active. The other side of this link can be either Active or Passive.

== LACP Mode Chart ==

Link Side A	Link Side B	State
Active	Active	Bundled
Active	Passive	Bundled
Passive	Active	Bundled
Passive	Passive	Not Bundled

Advanced LACP Concepts

In truth, one can mostly get away with only knowing basic LACP concepts and configuration. Over 90% of the time, just setting up the bundle, ensuring that the Po interface is configured correctly and verifying all interfaces are bundled will get things up and running. However, there are cases, I've experienced some personally, where further knowledge of how LACP actually works is important. 

Hashing and Traffic Path Determination

It is easy to think that if three Gigabit interface are bundled then there will be three Gigabit worth of throughput available. Theoretically, yes. In reality no. EtherChannels uses an XOR hashing algorithm to determine which link to forward traffic. This algorithm is only used by the forwarding switch for path selection. The receiving switch doesn't care which physical interface the traffic was received on, the traffic has arrive and needs to be forwarded on. Of note, this hashing algorithm applies to all EtherChannels; LACP, PAgP and static.

The XOR hashing algorithm uses information gathered from the traffic itself. Everything from MAC addresses, IP addresses and transport (TCP/UDP) ports can be used. Since the hashing algorithm is only used by the forwarding switch, load-balancing methods do not have to be the same between EtherChannel neighbors. Also, L2, L3 and L4 information cannot be mixed in the hash algorithm. The load-balancing method is a global configuration and, depending on the switch model, can be any of the following.

== EtherChannel Load-Balancing Algorithms ==

Hash Code	Information Used
dst-ip	Only destination L3 IP address
src-ip	Only source L3 IP address
src-dst-ip	Both source and destination L3 IP address
dst-mac	Only destination L2 MAC address
src-mac	Only source L2 MAC address
src-dst-mac	Both source and destination L2 MAC address
dst-port	Only destination L4 port number
src-port	Only source L4 port number
src-dst-port	Both source and destination L4 port number

Example of configuring the load-balancing algorithm on a 4500-X:

LACP-01(config)#port-channel load-balance ?
  dst-ip        Dst IP Addr
  dst-mac       Dst Mac Addr
  dst-port      Dst TCP/UDP Port
  src-dst-ip    Src XOR Dst IP Addr
  src-dst-mac   Src XOR Dst Mac Addr
  src-dst-port  Src XOR Dst TCP/UDP Port
  src-ip        Src IP Addr
  src-mac       Src Mac Addr
  src-port      Src TCP/UDP Port

LACP-01(config)#port-channel load-balance src-dst-ip ?
  <cr>

LACP-01#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
  IPv4: Source MAC address
  IPv6: Source MAC address

Bundle Master and Link Determination

When more than 8 links are used a mechanism is needed to determine which links we be moved to hot standby. LACP uses the concept of a master, in which one of the LACP devices is elected by way of a sys-id value. The master is responsible for controlling which bundled interfaces are active or hot-standby. The sys-id is made up of two values. When compared, the lowest value is elected master.

• system-priority
• Globally configured value specific to LACP
• Default value is 32768
• System MAC address

LACP-01#show lacp sys-id
32768, 0022.bd3e.6b00

LACP-01(config)#lacp system-priority ?
  <1-65535>  Priority value

To determine a bundle master

1. The systems priority values are compared
1. This value is globally configurable
2. Default value of 32768
3. Lowest wins
2.  In the event of a tie, the system MAC address is then compared
1. Lowest wins

The systems priority values are compared first. In the event of a tie, the system MAC address is then compared. There is no command to show bundle master. You just have to know the rules.

With a master elected, active ports are determined in a similar fashion, leveraging LACP port priority and interface number values.

1. LACP port priority is compared among all member links.
1. Value configurable per port
2. Lowest wins
2. System interface number is then compared for remaining interfaces
1.  Lowest wins

In summary, to enable lower numbered interfaces to be hot-standby, modify the LACP port priority value(s) on the LACP master switch.

LACP-02#sh lacp internal
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi0/15    SP      bndl      32768         0x1       0x1     0xF         0x3C
Gi0/16    SA      bndl      32768         0x1       0x1     0x10        0x3D
Gi0/17    SA      bndl      32768         0x1       0x1     0x11        0x3D
Gi0/18    SA      bndl      32768         0x1       0x1     0x12        0x3D
Gi0/19    SA      bndl      32768         0x1       0x1     0x13        0x3D
Gi0/20    SA      bndl      32768         0x1       0x1     0x14        0x3D
Gi0/21    SA      bndl      32768         0x1       0x1     0x15        0x3D
Gi0/22    SA      hot-sby   32768         0x1       0x1     0x16        0x5
Gi0/23    SA      bndl      16            0x1       0x1     0x17        0x3D
    Port priority set to less than 32768, becomes active.
Gi0/24    SA      hot-sby   32768         0x1       0x1     0x18        0x5

LACP Configuration

Lab Environment

LACP-01 and LACP-02 are connected one for one with ports 15-24. These interfaces are all 1 Gbps ports set to auto-negotiate speed and duplex. This ensure that all ports are like, a requirement of LACP. A layer 2 port channel will the built, but the same steps can be used to build for layer 3.

Configuration

LACP-01(config)#interface range gi0/15-24
LACP-01(config-if-range)#channel-group 1 mode active
Creating a port-channel interface Port-channel 1

LACP-01(config)#interface port-channel 1
LACP-01(config-if)#switchport trunk encapsulation dot1q
LACP-01(config-if)#switchport mode trunk

Verify configurations have been pushed down

LACP-01(config-if)#do show run interface gi0/15
Building configuration...

Current configuration : 127 bytes
!
interface GigabitEthernet0/15
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 1 mode active
end

Replicate this configuration on LACP-02 and verify Po1 is up.

LACP-01#show etherchannel summary
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port


Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi0/15(P)   Gi0/16(P)   Gi0/17(P)
                                 Gi0/18(P)   Gi0/19(P)   Gi0/20(P)
                                 Gi0/21(P)   Gi0/22(H)   Gi0/23(P)
                                 Gi0/24(H)

At this point LACP is active and normally that's all it takes. LACP system and port priority can be used to manage which interfaces are bundled and which are hot-standby. Global port-channel load balancing can also be configured to improve performance.