Cisco Nexus Spanning Tree History

I've been doing a fair bit of work on Nexus 5k / 6k platforms lately and while I've been less than impressed with certain aspects of the products, one thing that the Nexus is really excellent at is keeping logs, whether you ask it to or not. That comes in super-handy if you've left the buffer logging at its default don't-wake-me-unless-the-world-ends setting...

Pretty much anything that has a state is logged somewhere in the Nexus and you can get lost in a labyrinth of cryptic troubleshooting messages related to virtually any process in the switch. In this post I'm focusing on spanning tree logs as they're pretty universal.

Imagine the scenario shown below:

We have three sites connected over the WAN. We blew the budget on dark fibres out of the Cardiff site so we've had to skimp on switches and only have one per site, with the Caerphilly switch being root bridge. The link between Caerphilly and Newport is a metro Ethernet circuit which doesn't forward link loss.

Now imagine there's a failure within the carrier network which results in a total loss of traffic across the circuit between Caerphilly and Newport. No ports go down, however after a short time spanning tree will detect the fault and converge to use the indirect route via Cardiff. If the user's port is in p2p mode rather than edge, he is going to see a 30 second outage while his port transitions back to forwarding, even with RSTP.

How would you even know this had happened (aside from users complaining bitterly)? If you're really wily you may notice your traffic statistics look a bit odd, but if the primary link is restored relatively quickly that kind of thing gets lost in 5 minute roll-ups and natural variation quite easily. Since no interfaces went down, there will be nothing in your logs (by default).

Luckily, the Nexus logs every STP port state transition in its event history and keeps them seemingly forever. If the link flapped 6 months ago there's a good chance you could still prove it, as long as you haven't rebooted the switch. These logs can be retrieved using the command show spanning-tree internal event-history all - note that it's pretty verbose and you probably want to narrow it down if you have a lot of VLANs. The first section for each STP instance is the overall state history, mostly concerned with who the root is and how it is best reached:

Newport# show spanning-tree internal event-history all | begin VLAN0055
VDC01 VLAN0055
<snip>
77) Transition at 643104 usecs after Tue Jul  7 07:44:47 2015
     Root: 8037.000c.a45e.321c Cost: 0 Age:  0 Root Port: none Port: none [STP_TREE_EV_MULTI_FLUSH_LOCAL]

78) Transition at 762615 usecs after Tue Jul  7 07:44:49 2015
     Root: 8037.000c.a45e.321c Cost: 0 Age:  0 Root Port: none Port: Ethernet1/1 [STP_TREE_EV_UPDATE_TOPO_RCVD_SUP_BPDU]

79) Transition at 763013 usecs after Tue Jul  7 07:44:49 2015
     Root: 8037.000c.ac6d.43ba Cost: 4 Age:  0 Root Port: Ethernet1/1 Port: none [STP_TREE_EV_MULTI_FLUSH_LOCAL]

80) Transition at 722769 usecs after Tue Jul  7 07:44:51 2015
     Root: 8037.000c.ac6d.43ba Cost: 4 Age:  1 Root Port: Ethernet1/1 Port: Ethernet1/1 [STP_TREE_EV_MULTI_FLUSH_RCVD]

81) Transition at 832764 usecs after Tue Jul  7 07:44:51 2015
     Root: 8037.000c.ac6d.43ba Cost: 4 Age:  1 Root Port: Ethernet1/1 Port: Ethernet1/2 [STP_TREE_EV_MULTI_FLUSH_RCVD]

82) Transition at 752841 usecs after Tue Jul  7 07:44:52 2015
     Root: 8037.000c.ac6d.43ba Cost: 4 Age:  1 Root Port: Ethernet1/1 Port: Ethernet1/2 [STP_TREE_EV_MULTI_FLUSH_RCVD]

83) Transition at 782964 usecs after Tue Jul  7 07:44:53 2015
     Root: 8037.000c.ac6d.43ba Cost: 4 Age:  1 Root Port: Ethernet1/1 Port: Ethernet1/1 [STP_TREE_EV_MULTI_FLUSH_RCVD]

The logs are quite verbose but it's clear to see from the "Root Port: none" message that the primary path to the root was lost, then re-gained within a few seconds. Just a minor flap within the carrier network and a few seconds' impact?

Below the main state history are the individual port histories, let's look at our user's port and see what happened there:

VDC01 VLAN0055 <Ethernet1/10>
<snip>
7) Transition at 762694 usecs after Tue Jul  7 07:44:49 2015
     State: BLK  Role: Desg  Age:  2 Inc: no  [STP_PORT_MULTI_STATE_CHANGE]

8) Transition at 640356 usecs after Tue Jul  7 07:45:04 2015
     State: LRN  Role: Desg  Age:  2 Inc: no  [STP_PORT_STATE_CHANGE]

9) Transition at 642846 usecs after Tue Jul  7 07:45:19 2015
     State: FWD  Role: Desg  Age:  2 Inc: no  [STP_PORT_STATE_CHANGE]

Oh. Right at the same time as the WAN dropped out, our user's port went into blocking for 15s then learning for another 15 before finally transitioning to forwarding again. Ouch... and we never would have known were it not for the STP event history!

Side Note

You can save yourself the effort of reading the incredibly verbose event history by setting the logging level of spanning tree to something more useful, such as informational:

Newport(config)#logging level spanning-tree 6

Note, the logging level for the local buffer or syslog server will need to be set to a level that will record the newly verbose logging.

Also, user ports should be forced into edge mode to avoid STP convergence causing massive disruption to them:

Newport(config-if)#spanning-tree port type edge

The switch should "guess" correctly but it's probably best not to take the chance that a user port accidentally go into p2p mode.

tshark one-liners

Since most of the hits on this blog seem to come from tshark filter related searches, and since I spend a good part of my day either running or analysing packet captures, I thought it might be useful to create a series of "tshark one-liners" in homage to the brilliant "sed one-liners" collection compiled by Eric Pement.

These are capture filters, not display filters, and are equally applicable to Wireshark, tshark and tcpdump, since they all use the same pcap filter syntax. In wireshark the capture filter options are now hidden away and you have to double click on the interface under capture options to set or adjust the filter string.

The filters are broadly grouped by purpose and I will try to add more as I think of them. Please comment if there is something you think I have missed or would like added.

Note: if you want to strip off VLAN, MPLS, PPPoE or GRE headers from an existing pcap file, please see this post: Removing VLAN/MPLS/PPPoE/GRE Encapsulation

Ethernet

Match 802.1D spanning tree:
"ether dst 01:00:c2:00:00:00" (manpages say "ether proto stp" but I've had trouble with that)

Match Cisco PVST+:
"ether dst 01:00:0c:cc:cc:cd"

Match Cisco CDP / VTP / DTP / PAgP / UDLD:
"ether dst 01:00:0c:cc:cc:cc"

Match LLDP:
"ether proto 0x88cc"


Match LACP (slow protocols):
"ether dst 01:80:c2:00:00:02"

General IP
Match host A (10.0.0.1) communicating with host B (192.168.0.1):
"host 10.0.0.1 && host 192.168.0.1"

Match host A (10.0.0.1) communicating with anything on network B (192.168.0.0/24):
"host 10.0.0.1 && net 192.168.0.0/24"
or, if you don't like CIDR notation:
"host 10.0.0.1 && net 192.168.0.0 mask 255.255.255.0"

Match ARP:
"ether proto 0x0806"

Match DHCP:"udp port 67 || udp port 68"

VLANs

Match any traffic with at least one VLAN tag:
"vlan"

Match traffic with exactly one VLAN tag:
"vlan && not vlan"

Match traffic with an SVLAN of 100 and any CVLAN:
"vlan 100 && vlan"

Match traffic where the first VLAN tag has an 802.1p marking of:
0: "vlan && ether[14] & 224 == 0"
1: "vlan && ether[14] & 224 == 32"
2: "vlan && ether[14] & 224 == 64"
3: "vlan && ether[14] & 224 == 96"
4: "vlan && ether[14] & 224 == 128"
5: "vlan && ether[14] & 224 == 160"
6: "vlan && ether[14] & 224 == 192"
7: "vlan && ether[14] & 224 == 224"

Note: to match the second VLAN tag use "vlan && vlan && ether[18] & 224" on the left hand side of the equality.

MPLS

Match traffic with at least one MPLS label:
"mpls"

Match traffic with exactly one MPLS label (match S bit of first label):
"mpls && ether[16] & 1 == 1"

Match traffic with a first or single label of 12345:
"mpls 12345"

Match traffic with an inner (e.g. service) label of 67890:
"mpls && mpls 67890"

Match traffic with exactly three MPLS labels (e.g. traffic on facility bypass FRR):
"mpls && mpls && mpls && ether[24] & 1 == 1"

Match 6PE traffic:
With transport label: "mpls && mpls 2"
Without transport label (after PHP): "mpls 2"

Match traffic with an EXP marking (on the first label) of:
0: "mpls && ether[16] & 14 == 0"
1: "mpls && ether[16] & 14 == 2"
2: "mpls && ether[16] & 14 == 4"
3: "mpls && ether[16] & 14 == 6"
4: "mpls && ether[16] & 14 == 8"
5: "mpls && ether[16] & 14 == 10"
6: "mpls && ether[16] & 14 == 12"
7: "mpls && ether[16] & 14 == 14"

Note: to match the EXP marking of the second label, use "mpls && mpls && ether[20] & 14" on the left hand side of the equality.

Multicast

Match any Ethernet multicast:
"ether multicast"

Match IP multicast traffic:
"ip multicast"

Match IGMP traffic:
"ip proto 2" (the manpages say "ip proto igmp" but I've had trouble with that)

Match PIM traffic:
"ip proto 0x67" (the manpages say "ip proto pim" but I've had trouble with that)

OSPFv2

Match all OSPF:
"ip proto 89"

Match specific OSPF packet types:
Hello: "ip proto 89 && ip[20:2] == 0x0201"
DBD: "ip proto 89 && ip[20:2] == 0x0202"
LSR: "ip proto 89 && ip[20:2] == 0x0203"
LSU: "ip proto 89 && ip[20:2] == 0x0204"
LSA: "ip proto 89 && ip[20:2] == 0x0205"

IS-IS

Match all IS-IS traffic:
"isis"

Match specific IS-IS PDU types:
"l1", "l2", "iih", "lsp", "snp", "csnp" or "psnp"

BGP

Note: These rules do not handle multi-segment messages very well but they are good enough for most purposes.

Match only BGP OPEN messages:
"tcp port 179 && tcp[50] == 1"

Match only BGP UPDATE messages:
"tcp port 179 && tcp[50] & 5 != 0"

Match only BGP NOTIFICATION messages:
"tcp port 179 && tcp[50] == 3"

Match only BGP KEEPALIVE messages:
"tcp port 179 && tcp[50] == 4"

 L2TP

Match only L2TP control messages:
"udp port 1701 && udp[8:2] & 0x80ff == 0x8002"

Match L2TP control messages for tunnel ID 1234:
"udp port 1701 && udp[8:2] & 0x80ff == 0x8002 && udp[12:2] == 1234"

Match L2TP data messages for tunnel ID 1234:
"udp port 1701 && udp[8:2] & 0x80ff == 0x0002 && udp[10:2] == 1234"

Match L2TP control messages for session ID 5678:
"udp port 1701 && udp[8:2] & 0x80ff == 0x8002 && udp[14:2] == 5678"

Match L2TP data messages for session ID 5678:
"udp port 1701 && udp[8:2] & 0x80ff == 0x0002 && udp[12:2] == 5678"

PPPoE

Note: Offsets will need to be manually increased by 4 bytes  for each VLAN tag or MPLS label present.

Match PPPoE discovery phase (PADI / PADO / PADR / PADS / PADT):
"pppoed"

Match PPPoE session phase (i.e. PPP traffic):
"pppoes"

Match PPPoE LCP messages:
"pppoes && ether[20:2] == 0xc021"

Match PPPoE CHAP authentication messages:
"pppoes && ether[20:2] == 0xc223"