LACP miscellanea

According to the statistics, a few people have stumbled across this blog because they were searching for certain 7750-specific information relating to LACP. Here are a couple of answers that were missed out from my main LACP article:

What is the failover time for a LAG / etherchannel? 

The answer to this question varies considerably depending on the setup. If a device notices a bundled interface going physically down then it should unbundle it immediately, causing very low loss (50ms should be achievable).

In the event of an interface remaining physically up (i.e. where there is transmission equipment or EoMPLS between the two devices), also known as a silent failure, the failover will be up to 3 times the LACP timer. So the impact would be up to 3 seconds using fast timers or up to 90 seconds using slow timers. Most lower end Cisco kit only supports slow timers.

In the event of a single fibre fault or other asymmetric failure, you may see a combination of these effects where traffic in one direction heals faster than the other. There are other corner cases such as when administratively shutting down an interface - some devices send an out of sync LACPDU to inform the other end the link is about to go away which helps speed convergence. It is really best to lab test where possible to check different failure scenarios.

Be aware that when using load balanced LAGs, the impact to some streams may be zero. Typically traffic that is hashed onto one link in the bundle will not suffer loss when a different link in the bundle fails.

How can I transport, rather than terminate, LACP through epipe services on the Alcatel-Lucent 7750? 

The answer to this is pretty straightforward, but I know I looked in the wrong place when I first needed to use the feature.

All you need to do is to configure "lacp-tunnel" under the configure -> port -> ethernet context.

How can I transport, rather than terminate, LACP through a QinQ tunnel on a Cisco switch? 

Again, this is pretty straightforward. There are a load of different protocols that can optionally be tunneled on a dot1q-tunnel port, but we just need lacp enabled:


Simply configure "l2protocol-tunnel point-to-point lacp" under the dot1q-tunnel interface.

Normally it makes sense to tunnel everything (STP, CDP, VTP, LLDP, LACP, PAgP, ...) for consistency. Either be a tunnel or don't!

What is the valid range for LAG IDs on the 7750?

For IOM-based systems (i.e. SR-7, SR-12), the usable LAG ID range is 1 to 200. For integrated IOM systems such as SR-1 and ESS-1, the LAG ID range is 1 to 64.


Can you tell me something unusual about LACP on the 7750?

When  the Rx fibre for a LACP-speaking port loses light (i.e. fails), right before the port gets pulled down the 7750 sends an LACP out-of-sync message to inform the other end that it is going away. This is useful for single fibre faults and can drastically improve convergence times, particularly where transmission equipment between the two LACP peers does not forward link loss.

What does "mux: during state COLLECTING_DISTRIBUTING, got event 5(in_sync) (ignored)" mean in a Cisco debug?

As best I can tell it means that a LACPDU was received with the sync bit set, indicating that the far end is ready to use the link, but the link was already collecting / distributing (i.e. in use) so no change in state was required.

What does "lag number : partner oper state bits changed on member port : [expired false -> true]" mean on a 7750 debug?

This means that a particular port's state machine moved into the expired state due to missing three inbound LACPDUs from the peer. Once the port reaches the expired state it is removed from the bundle but the peer parameters are remembered for a further 3 intervals, after which point the peer information is flushed and the port enters the defaulted state.

Do the LACP keys need to match at both ends of a LAG?

No - the LACP key is locally significant and corresponds one-to-one with a LAG or etherchannel ID. It is used to check consistency (i.e. to catch crossed cables) so must be identical for all members within a LAG, however the devices at each end of the LAG can select any value they like for any particular LAG.

Why do ports get "suspended" from an etherchannel?

Basically a port gets suspended if its configuration is not in line with that of the port channel with which it is associated. The most common way to accidentally arrive in this state is for a member trunk port to have a different allowed VLAN list than its parent port-channel interface. While IOS allows member ports to be reconfigured, it is much more sensible to make the configuration changes to the port-channel interface - the changes are then pushed down to the member ports automatically, avoiding this kind of conflict.

A Better Way to Compare 7750 Configs

One of the tasks I regularly have to perform as part of my job is to audit router configurations against basebuild templates and occasionally against each other. This can be quite labour-intensive, particularly as good, old-fashioned diff doesn't cope very well with hierarchical configs such as the 7750's. There are many things that make life difficult with traditional diff:

1. Diff is more-or-less reliant on order being preserved between the files being compared. Some config elements are split across the config file when it is saved. Others may be stored in different locations depending on the software release. Certain items such as layer 3 interfaces are stored in the order that they were added, so two configs may contain the same interfaces but in a different order and traditional diff can't generally work that out.
2. Traditional diff does not appreciate the significance of policy names, sequence numbers or service IDs in determining what should be compared to what. Ad-hoc insertions and deletions of policy entries or service often causes diff to get completely out of step, making it compare apples to oranges.
3. Traditional diff does not provide context for differences. Quite often you just get a load of "shutdown" "no shutdown" pairs. It is possible to include a fixed number of lines pre- or post- difference, but even that does not always show the configuration context where the difference actually occurred and brings a load of junk with it. The only sure-fire way is to turn on side by side diff, which shows both configurations in full side by side with changes marked in a centre column.

Point 2 is the real killer, making traditional diff basically useless for audits where a handful of known policies must be validated within a config containing many other policies. Below is a (fairly common) worst case when working with diff - one policy has been removed and another two added:

Traditional diff makes a horrible job of this. Even seeing the two configs side by side it is confusing to look at and it is not immediately apparent what has changed.

I poked and played with a number of "diff" type tools to try and find something that would handle this kind of thing more gracefully but I eventually came to the conclusion that nothing currently existed. I had a rough idea of what I wanted:

•  It must only compare the contents of like policies, i.e. policy "A" should only be compared to policy "A" and never to policy "B".
• It must compare configuration elements that appear in a different order in one config to the other.
• It should, ideally, report the full context of each difference within the hierarchy.

To address these points I decided to write a tool from scratch and called it, unimaginatively, 7750diff. Here's how 7750diff reports the same changes:

D:\7750diff>7750diff a.cfg b.cfg
Unique to a.cfg:
configure
    qos
        scheduler-policy "20000kbps" create
            description "20000kbps Scheduler"
            tier 1
                scheduler "tier1" create
                    rate 20000 cir 20000
                exit
            exit
        exit
    exit
exit
Unique to b.cfg:
configure
    qos
        scheduler-policy "25000kbps" create
            description "25000kbps Scheduler"
            tier 1
                scheduler "tier1" create
                    rate 25000 cir 25000
                exit
            exit
        exit
        scheduler-policy "40000kbps" create
            description "40000kbps Scheduler"
            tier 1
                scheduler "tier1" create
                    rate 40000 cir 40000
                exit
            exit
        exit
    exit
exit

D:\7750diff>

That's not only much clearer (IMHO), but I can take the output and copy / paste it directly into the node to build the missing configuration. Happy days!

How it Works

The basic methodology used by 7750diff is to:

• Read each config into a hierarchical tree structure based on indent levels
• Recursively compare the trees starting at the root:
• For each branch of config A, search for an identical branch on config B within the same context.
• If a match is found, check for subordinate "child" configuration elements.
• If any children exist, recursively process them.
• If no children are then present, remove the matching elements.

Once all elements have been compared only the elements unique to each config will remain, along with the parent elements required to reach the configuration context of the change. The trees can then be output as a list of differences between the two files. In many cases, thanks to the inclusion of context, big chunks of 7750diff output can be directly entered into one node to bring its config into line with the other.

Since the whole thing runs on indents it is not 7750 specific. It "may" work on ISAM configs, it may work on any other sort of config that uses indent / whitespace to denote hierarchy - I just haven't tested it. Try your luck :)

Obtaining the Tool

As usual, the tool is available for download at my github: https://github.com/theclam/7750diff - there is the C source code plus a Windows binary available to download.

The code is pretty horrible as this was my first bit of C coding in over a decade. I may decide to clean the code up at some point but it is quite stable now, meaning I haven't found a config that upsets it for the last couple of versions. I've been running a nightly cron job for quite some time now which pulls down an entire lab's configs and 7750diffs each one against the previous day's and it works great.

If you download 7750diff, please let me know how you get along with it.