Testing the Impact of Local Packet Capture on the Cisco 6500 Series

For a while now, many of the larger Cisco devices (such as 6500 and 7600s) have supported local packet capture. I've always been hesitant to use these in a production environment, primarily due to concerns about the potential performance hit it could cause. Anyway, I decided to test out the potential impact in the lab to see whether it was sometimes / always / never safe to run local captures.

Test Setup

For my test setup I used a 6504-E switch with a modest SUP32-GE-3B supervisor and a 12.2 Advanced IP Services IOS - if it works on that, it should be safe anywhere. For traffic, I used a spare server running Ubuntu with a combination of Ostinato and Scapy.
The configuration was as follows:

Running at full tilt, Ostinato was happily producing 1 Gbps of traffic which went into my node on VLAN 100, out around the loop cable, back into the node and then out of the same interface on VLAN 101. Basically the port is running at 1 Gbps in each direction, so the worst possible case for mirroring a gig port.

Default Settings

The default settings for the capture are pretty conservative - a tiny 2 MB linear capture buffer with a rate limit of 10,000 frames per second. With this config, a 1 Gbps stream of 1500 byte packets fills the buffer in ~ 2.5s, triggering the capture to end. The impact of this is almost impossible to measure at all, with the capture being over so quickly you may not actually see any change in CPU on the 5 second roll-ups.

Lab-6503E#monitor capture start
*Jul 11 14:30:59.205: %SPAN-5-PKTCAP_START: Packet capture session 1 started
Lab-6503E#
*Jul 11 14:31:01.449: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended as the buffer is full, 21845 packets captured
Lab-6503E#show proc cpu hist
                                                              
                                                              
                         222223333322222               8888811
100                                                           
 90                                                           
 80                                                           
 70                                                           
 60                                                           
 50                                                           
 40                                                           
 30                                                           
 20                                                           
 10                                                    *****  
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

Worst Case

OK, so the world didn't end. The next step was to see how bad it could be so I made the following changes:

• Increased the rate limit to 100,000 fps (max)
• Increased the packet buffer to 64 MB (max)
• Enabled a circular buffer (why?!)

This time I set the capture to run for 60 seconds and checked the impact, which was much more severe:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 14:45:02.953: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 14:46:02.945: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 699040 packets captured
Lab-6503E#show proc cpu hist

    2222244444444444444455555444444444444444444444444444444444
    9999999999888889999966666999999999999999999998888899999999
100
 90
 80
 70
 60                     *****
 50      *****************************************************
 40      *****************************************************
 30 **********************************************************
 20 **********************************************************
 10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)

The impact to the supervisor in this case was much more noticeable - up to 60% CPU utilisation. The scenario is pretty unrealistic (forget circular buffers!) but suffice to say I wouldn't want to do that on a production device.
Now we have a the best and the worst cases, let's look at some realistic use cases and explore some of the other capture parameters that might help us capture what we need without causing havoc on the network.

Narrowing Down the Capture

It's possible to define criteria to decide what gets captured - as the CLI points out, some of these criteria are processed in hardware while others are handled in software:

Lab-6503E(config-mon-capture)#filter ?
  access-group  Filter access-list (hardware based)
  ethertype     Matching ethertype (software based)
  length        Matching L2-packet length (software based)
  mac-address   Matching mac-address (software-based)
  vlan          Filter vlan (hardware based)

Our test traffic consists of nearly a gig of junk run alongside a small 1-per-second ping, which we will decide is "interesting" to us and we want to capture. This traffic profile makes it easy to test ACL, MAC address and length filters and their relative performances.

ACL Filter

As the CLI says, ACL filters are applied in hardware so the junk is discarded at source before it hits the CPU. In this example I set up an ACL as follows:

Lab-6503E(config)#ip access-list extended icmp-only
Lab-6503E(config-ext-nacl)#permit icmp any any
Lab-6503E(config-ext-nacl)#deny ip any any

... and applied it to my capture as follows:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#filter access-group icmp-only

Now, I re-ran the "worst case" test, with massively different results:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 14:49:00.345: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 14:50:00.337: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 60 packets captured
Lab-6503E#show proc cpu hist

         22222222223333355555          11111
100
 90
 80
 70
 60
 50
 40
 30
 20
 10                     *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Two things to note here - because this is a hardware filter, applied in the ASICs:

• Only 1 packet per second was punted to the CPU, resulting in essentially no hit at all
  
• All 60 of the ping packets were received and nothing else
  

In summary, ACL filters are ideal when you know what you want to pull from a big stream as you only pay a CPU penalty for the "good stuff".

MAC Filter

In contrast, the MAC filter runs in software. It's also pretty limited, only matching on source MAC. I repeated the above test but instead of an ACL match, applied a MAC filter as follows:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#filter mac-address 0011.2233.4455

This took us more-or-less back to the worst case:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 15:05:55.197: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 15:06:55.189: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 5 packets captured
Lab-6503E#show proc cpu hist

      44444444443333344444333334444444444444444444444444333333
    4422222111119999900000999990000011111333331111111111888889
100
 90
 80
 70
 60
 50
 40   ********************************************************
 30   ********************************************************
 20   ********************************************************
 10   ********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Yuck. I wouldn't do that in production. This is basically because the software filters are applied after the packets have been punted to the CPU, so you pay a penalty for the garbage as well as the good stuff. You'll notice that it only captured 5 packets as well, more on this later but that's another side effect of software filters.

Length Filter

The frame length filter is another software-based mechanism, which means it's pretty terrible under load, too. Our junk traffic consists of large frames, our interesting traffic is small, so let's configure the capture to only catch the short frames:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#filter length 0 100 

Again, the output is pretty miserable:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 15:15:12.145: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 15:16:12.137: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 17 packets captured
Lab-6503E#show proc cpu hist

    1111444443333344444444443333333333333333333333333444443333
    4444000009999911111222229999999999999999999999999222229999
100
 90
 80
 70
 60
 50
 40     ******************************************************
 30     ******************************************************
 20     ******************************************************
 10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Again, the CPU took a hammering and we only captured a few of the ping packets - 17 out of 60.

Quirks / Order of Operations

Now you may think that software filters might be OK if we just reduce the rate-limit configured on the capture:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#rate-limit 100

This *does* do what we want for the CPU load - here's an example with a MAC filter:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 15:26:43.793: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 15:27:43.785: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 0 packets captured
Lab-6503E#show proc cpu hist

                    11111          11111          6666633333
100
 90
 80
 70
 60
 50
 40
 30
 20
 10                                               *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Great - nothing to see here. But also nothing to see in the capture buffer - 0 packets captured!
Just for fun let's try the same with a hardware ACL filter:

*Jul 11 15:25:26.921: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 60 packets captured

Why is this? Well, it's the order of operations. Basically the flow for hardware ACL filters is:

So the filter throws out the junk before the rate limiter, meaning that the rate limiter only counts the good stuff. If the good stuff exceeds the rate limit then you'll lose some of it but the junk doesn't count.
Compare that to the flow for the software filters:

The software filters are applied after the rate limiter, so clearly when the rate limit is exceeded you throw out a mix of good and bad traffic, then pick out what's left of the good. If your traffic is overwhelmingly garbage, you may not get any of the good stuff at all!

Summary - Play it Safe

So in answer to the question "is it safe to run a local capture on a production 6500" - packet capture on even a relatively modest SUP32-3B supervisor is pretty safe provided you are cautious. If you want to do this in a busy production environment then my message to you is:

• Use ACLs where at all possible
  
• Set the rate limit to a sensible value (the default 1000 fps is fine for most cases)
  
• Use linear buffers of a sensible size (do you really need 64MB of capture?)
  
• Limit the frame count or capture duration at first (it may turn out there is a lot more "interesting" traffic than you thought!)
  

I'm convinced enough to do this in production but obviously I'm only testing one device, on one release of code, so don't blame me if you try it on something different and encounter a bug!

Re-assembling IP Fragments in PCAP Files

Some time ago I created "stripe", a tool for stripping back layers of encapsulation headers from PCAP files leaving plain payload (typically IP) over Ethernet. "stripe" works with a variety of encapsulation types, from simple VLAN tags up to GRE and GTP, however one thing that stripe couldn't handle was if packets were fragmented after being encapsulated.

One user, LisbethS, suggested that I should build in IP fragment reassembly capabilities into stripe, however my first thought was that the two should be separate utilities. As I thought about it more, though, I realised that the two functions (decapsulation and re-assembly) were actually intertwined - if you treat them as separate processes then you can't re-assemble IP that is encapsulated within something else, nor can you decapsulate GRE or GTP that has been fragmented. The key, then, was to do both re-assembly and decapsulation as part of the same process.

Reassembling Packet Fragments

RFC 815 describes a minimal way to re-assemble IP fragments which is not that complex in principle, so I thought I'd add the functionality. I soon realised it wasn't quite as straightforward as I thought and you have to be very careful about the order of operations.

For example, if you have a packet that gets encapsulated then subsequently fragmented, then trying to decapsulate without reassembling first will fail (the first fragment decapsulates to a partial frame, then the subsequent fragment(s) fail to decapsulate). On the other hand, if you re-assemble first then decapsulate then you don't catch the case where a packet is fragmented before encapsulation. Neither approach can catch the case where a packet is fragmented, then encapsulated, then subsequently fragmented again.

To cut a long story short, the answer appears to be that you need to iteratively re-assemble, decapsulate until fragments are found, re-assemble again, decapsulate again... until there are no more fragments and everything is fully decapsulated.

Anyway, stripe now does both decapsulation and IP fragment re-assembly, meaning that it can take a pcap file containing fragmented and / or encapsulated packets, strip off all the encapsulation and re-assemble the fragments and write out the result to a new pcap file.

Download

The latest version is available for download at https://github.com/theclam/stripe - it is available as source code (compiles without dependancies in almost any Linux distro) and there are also Mac and Windows binaries for easy download.

UPDATE:

I've managed to recreate some of the SEGFAULTs that people have been kindly reporting to me. It turns out there was a typo / n00b mistake (I'm not sure which, most of this is coded way too late at night) which I have now corrected. If you tried before and got an error, it may be fixed now. The memory leaks have also been reduced from "raging" to "moderate" :)

Removing VLAN/MPLS/PPPoE/GRE/GTP/VXLAN Encapsulation Headers from pcap Files

Many years ago, when I worked in a school, I used to port mirror our proxy server to an old PC running driftnet and leave the screen where the kids could see it as a warning that staff could "see what you're doing on the Internet". I haven't played with driftnet since but certainly at the time it could only handle native frames (no VLAN tags, certainly no MPLS or PPPoE). I vaguely remember some other tools being similar, unfortunately I can't remember which ones.

Looking at the analytics for this blog, I can see I'm not the only one who's had the problem. It's certainly not the number one issue that people are searching for when they get here but there have been a few and the thought occurred that the packet processing engine I wrote for dechap would be really good for this task - it already stripped back VLANs and MPLS, plus it knows how to detect PPPoE and L2TP.

After a couple of hours it was working to the point of being able to strip VLANs and MPLS off, with a little more effort PPPoE also gave way. GRE came quite easily, too, as it has simple headers and uses the same etypes as Ethernet.

Anyway, here is "stripe" (from STRIP Encapsulation), a command line tool which takes a pcap file as input, re-assembles IP fragments and strips off all the encap it can (currently VLAN tags, MPLS shim headers, PPPoE, L2TP, GRE GTP and VXLAN) then outputs another pcap containing just payload over Ethernet.

**UPDATE** - Version 0.3b now adds support for VXLAN.

Download

Stripe is available from my github: https://github.com/theclam/stripe

Usage

The command line is pretty straightforward, as shown in the online help:

Harrys-MacBook-Air:stripe foeh$ ./stripe
stripe: a utility to remove VLAN tags, MPLS shims, PPPoE, L2TP headers,
etc. from the frames in a PCAP file and return untagged IP over Ethernet.
Version v0.1 alpha, November 2014

Usage:
./stripe -r inputcapfile -w outputcapfile

Where inputcapfile is a tcpdump-style .cap file containing encapsulated IP 
outputcapfile is the file where the decapsulated IP will be saved

Harrys-MacBook-Air:stripe foeh$ 

Simply specify the files you want to read encapsulated packets from (-r) and write the cleaned up packets to (-w). Stripe will remove as many layers of encap as it can until you are left with straight payload over Ethernet.

How it Works

The majority of stripe's work is done by the "decap" function. This function takes in a block of memory, a length parameter, a data type hint and a frame template. The process runs as follows:

1. If the type is Ethernet, populate the source / destination MACs of the frame template
2. If the type has an Ethertype or protocol type field, use this to populate the ethertype of the frame template
3. If the next protocol is possibly or definitely payload, set the payload pointer of the frame template to the address of the next protocol and return
4. If the next protocol is possibly or definitely encapsulation, call decap against the remainder of the packet

So essentially it eats up encap, recording MACs and protocol types as it goes, until there is no more encap left. By the end there is a fully populated frame template with source and destination MAC (the innermost copy if there are multiple as in the case of MPLS pseudowires), the etherype of the payload and the payload itself. Piecing these together gives a minimally encapsulated frame, i.e. one with just an Ethernet header and payload.

Here is a worked example for a frame with VLAN, MPLS, GRE over IP and an IP payload:

Step 1 - The "decap" function is called on the entire frame. Since the first header is Ethernet, the frame template gets populated with the source / destination MACs and the etype from the Ethernet header. The frame template's length field gets populated with the size of the frame minus the Ethernet header and the payload pointer is adjusted to point at the next header. The decap function then calls itself on the remainder of the frame, hinting that the type is VLAN tag based on the current header's etype.

Step 2 - The decap function now considers the partial frame starting at the VLAN tag. Since the VLAN tag has an etype associated, the frame template's etype is overwritten with the one from the VLAN header. The length is overwritten with the length of the payload after the VLAN header and the pointer adjusted to point at the next header. The decap function then calls itself again with a hint of MPLS, based on the etype in the VLAN header.

Step 3 - The decap function now considers the partial frame starting at the MPLS label. Since the MPLS label is bottom of stack, we know there are no more MPLS labels left . Unfortunately there is no protocol type in an MPLS header (these are signaled on the control plane) so we have to take a peek at the byte immediately following the label. If we find a "4" or a "6" in the high order nibble then we have to guess that the next protocol is IPv4 or IPv6, respectively. If the following four bytes are all zeroes then we assume Ethernet over MPLS with control word, otherwise we assume Ethernet over MPLS without control word. In this case we find a 4 in the low nibble, so call decap with an "IP" hint.

Step 4 - The IP header tells us that GRE is the next protocol so for now nothing changes in the frame template (the remainder could be decodable or not). We just call decap again on the GRE part...

Step 5 - The GRE header is decoded and the etype is copied into the frame header. The length of the remaining payload is updated in the frame template and the pointer is adjusted. Decap is called on the next header, which is IP. When the decap function inspects the IP payload it can go no further and just returns the frame template.

In essence, the process has started with a deeply encapsulated frame and ended with IP over Ethernet. The source and destination MACs are taken from the innermost ones found (which in this case is the outermost Ethernet header) but with the etype changed to match the payload, which is the first non-encapsulating payload found in the frame, in this case the second IP.

References

https://tools.ietf.org/html/rfc2784
https://tools.ietf.org/html/rfc1701
http://www.ieee802.org/1/pages/802.1Q.html
http://www.3gpp.org/DynaReport/29060.htm

Crippling CPU Load on Back to Back ASAs

I was recently involved in troubleshooting a problem where an ASA firewall's CPU was hitting 100%. One of its interfaces was seeing much higher traffic levels than the others, so we did some fairly run-of-the-mill troubleshooting including a packet capture. What this showed was the same, seemingly innocuous, packet repeated thousands upon thousands of times.

The payload was identical, in fact everything from the IP layer and up remained identical from one frame to the next. The only thing that varied was that the source and destination MAC addresses were swapped each time - clearly the packet was ping-ponging between two devices.

We checked the MACs and found they were legitimate - one was the local firewall, while the other was its default gateway - another ASA upstream towards the Internet.

This got our attention. First of all there was a routing loop, which is bad enough, but a packet should never be able to loop forever like that. That's why we have Time To Live (TTL) after all - the number which decrements by one each time a packet goes through a routed hop with the packet being thrown away when its value reaches zero. The key thing here is that the packet, including its TTL, was not changing at all so it never got removed from the system.

The cause of the routing loop was relatively easily found by looking at the source and destination IPs on the packet. The setup was as follows:

What had happened here was that a RAS user had connected to the tenant firewall using their IPSec client and started talking to some devices on the server LAN:

At some point the IPSec session had ended while an internal device was still sending traffic towards the user. This creates an interesting corner case:

The routing is "correct" here - the multi-tenant firewall needs to route the RAS subnet via the tenant firewall so that RAS users can connect to shared resources. The tenant firewall needs to route the traffic outwards for it to hit the right crypto maps. The problem comes when a packet is destined for an IP in the RAS pool which is not associated with a live VPN session.

Our bodge to get us out of the immediate hole was to put a deny entry inbound on the mutli-tenant firewall for anything targeted at a RAS pool address. These packets should never make it onto the transit LAN as any legitimate traffic to that range would need to be tunneled via IPSec and therefore the multi-tenant firewall would see a public IP as the destination. After a lot of thinking we couldn't come up with a better answer than this and decided just to stop calling it a bodge.

OK, so first problem solved. Next question, why was the packet looping forever without ever reducing its TTL?

Root Cause

As it turns out this is by design on the ASA (and the good old fashioned PIX & FWSM before it). The idea is that if the firewall behaved like any other routed hop and decremented the TTL then it would be visible in traceroutes. To be fair if it did decrement TTL it would just appear as a black hole in the trace as the ASA doesn't really "do" unreachables unless you force its hand. This normally doesn't cause any problems, even if there is a routing loop. Take a typical deployment where an ASA is attached to a router as shown below:

If we get a loop between the ASA and a traditional router then the packet will eventually be taken out of the loop. Even though the ASA doesn't decrement the TTL, the router does so it eventually gets dropped - half as fast as normal and always by the router (which will punt the packet to the CPU and usually generate an ICMP TTL expired, which can be pretty CPU intensive on small devices), but it does get dropped eventually.

The problem comes when we have a pair of non-decrementing devices (ASAs) back to back at layer 2. Both devices route the packet but neither device decrements the TTL, so if there is a loop between the two it the packet will go around and around forever. Eugh...

The moral of the story is that it's probably best not to put ASAs back to back. I could have sworn I'd seen this setup in Cisco whitepapers before but, now that I look, I can't find it anywhere. The closest I can find is IOS firewall back to back with ASA or two ASAs with a server between. Perhaps there's a good reason for that :)

As with my situation, though, in most cases by the time you get to realise there is a problem the hardware has long since been bought, installed and is carrying live service. So what can you do?

Well, as noted above you can use ACLs to block potential loop traffic but in all honesty that just fixes by exception. You could be fairly liberal with what you block (e.g. drop all RFC 1918 addresses where you would expect to only see public IPs) but it's still imperfect.

Making the ASA decrement TTL

A better idea would be to have at least one of the ASAs decrement TTL. It's a bit uncomfortable to retro-fit but there is a way built in to ASA versions 8.0(3) and above using "set connection decrement-ttl" under a policy map. There are two different ways to do it, one is to adjust the "global_policy" policy map which applies to the entire device by default, or you can create a new policy map to apply to a single interface.

Here's how to apply it to the entire device:

policy-map global_policy
 class class-default
  set connection decrement-ttl
!

The effect is immediate as the global_policy is applied to all traffic by default. Alternatively, if you only want to apply it to specific interfaces, you can create a separate policy map and apply it as follows:

policy-map asa_workaround
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
  inspect icmp
 class class-default
  set connection decrement-ttl
!
service-policy asa_workaround interface interface-name

The above is modeled on the standard default policy & inspections, if you've changed yours from default you probably don't need to be reading this!

Summary

So there you have it - ASAs back to back is a bit dangerous unless you take measures to protect against routing loops. This can be in the form of strict ACLs or by enabling TTL decrement, either globally or on specific interfaces.

References

Cisco guide to enabling traceroute through ASA

Cisco guide to modular policy framework on ASA

BGP support added to dechap

Hot on the heels of adding the ability to attack OSPF MD5 authentication, I've added BGP support to dechap. It is now possible to feed a pcap file with PPPoE, L2TP, RADIUS, OSPF and BGP packets to the same tool and perform offline dictionary attacks on the authentications within.

As usual, if you're not interested in the theory just skip right to the end for the download link.

TCP MD5 Signatures

BGP authentication uses the MD5 Signature TCP option field, which is defined in RFC 2385. Personally, I found this RFC very vague and it took a lot of iterations to get the technique right. It's particularly fuzzy about what is included in the hash, what isn't and how to present values correctly. I'm hoping to document the process a little more clearly for the next poor guy who tries to implement it as I couldn't find a sufficiently detailed reference anywhere.

RFC 2385 states that the hash must be calculated over the following:

1. the TCP pseudo-header (in the order: source IP address,
   destination IP address, zero-padded protocol number, and
   segment length)
2. the TCP header, excluding options, and assuming a checksum of
   zero
3. the TCP segment data (if any)
4. an independently-specified key or password, known to both TCPs
   and presumably connection-specific

Now, maybe it's just me, but this raised a lot of questions in my mind. Zero padding usually means to fill the trailing space with zeros, but padding the second byte would effectively multiply the protocol number by 256 so should it be a leading zero? Which headers and options are included in the "segment length"? Should the pad bytes be copied with the TCP header?

Through a lot of trial and error I found that:

• The zero padding goes before the protocol number
• The "segment length" includes the TCP header, the TCP options (including room for the MD5 signature option being calculated) and the actual payload data
• The copied TCP header should be 20 bytes long, i.e. includes two padding bytes after the (zeroed out) checksum. The header length remains as-is, including the length of the options.
• The TCP segment data starts immediately after the TCP options and runs to the last byte indicated by the IP length field
• The null byte terminating the password is not passed to the hash algorithm

 The resulting hash value is then stored inside the MD5 signature option (kind 19, length 18).

Checking / Attacking BGP Packets

Using the above method it is straightforward to run a dictionary attack as follows:

• Start with a sniffed BGP packet (see the original dechap blog post for info on how this is extracted).
• Extract and store the authentication hash (look for option kind 19) for later comparison
• Put together the "pseudoheader" as described above
• Append the TCP header without options
• Append the TCP payload
• Append the candidate password
• Calculate the MD5 hash over the complete data set and compare to the value seen in the sniffed packet. A matching hash indicates a matching password.

As of v0.4a, dechap can now be used to automate this process.

Obtaining the Tool

The C source code may be downloaded from: https://github.com/theclam/dechap

Provided the OpenSSL dev libraries are installed it should be possible to simply extract the source code, cd into the directory then run "make". I've only tested this under Ubuntu Linux but there are very few dependancies so I would imagine it will work on most distributions.

Using the Tool

As usual - this is for legitimate audit and recovery purposes and must not be used for any kind of malicious activity.

The usage is pretty straightforward - there are only two parameters and both are mandatory. Specify your capture file (original pcap format) with the -c flag and your word list with the -w flag. Here's an example:

lab@lab:~/dechap$ ./dechap -w mywords.txt -c bgp.cap
Found password "password1" for TCP from 10.0.0.2 to 10.0.0.1.
Found password "password1" for TCP from 10.0.0.1 to 10.0.0.2.
Found password "password1" for TCP from 10.0.0.2 to 10.0.0.1.lab@lab:~/dechap$
I'm not sure how quickly it runs but it doesn't seem quite as quick as the OSPF version. I suppose BGP packets tend to be a little bigger than OSPF so there's more to hash. You can improve the speed by only including one packet for each source / destination pair in each capture as, at present, it doesn't check for multiple packets between pairs and attacks each instance individually.

If you try this out, please leave a comment on this post with your experiences - good or bad. Any suggestions would also be welcome, particularly for other protocols to attack.

References

RFC2385 - Protection of BGP Sessions via the TCP MD5 Signature Option
RFC1321 - The MD5 Message-Digest Algorithm

Offline Attack on MD5 keys in captured OSPF packets

A few months ago I released a tool called dechap which finds PPPoE, L2TP and RADIUS authentications in pcap files and performs dictionary attacks against them. Since writing dechap I've always thought it would be more useful if it were able to do a similar thing with OSPF packets.

Well, the good news is that I've finally got around to adding OSPF support to dechap! Woo and yay! If you just want the tool, scroll straight to the bottom. If you're interested in the theory, read on.

OSPF Authentication Basics

OSPF, or more accurately OSPFv2 as defined in RFC2328, has three options for authenticating incoming packets:

Null: no authentication is performed at all.

Password: a plaintext password is added in the clear to each OSPF packet. If the password contained in an incoming packet matches the one configured locally then the packet is considered valid and is processed, otherwise it is silently ignored.

Message Digest: an MD5 hash is calculated over a combination of the OSPF packet contents and the password. The hash output is then added to the OSPF packet before transmission. When a packet arrives, the receiving router computes an MD5 hash of the packet contents plus its locally stored password. If the calculated hash matches the one attached to the incoming packet then the check passes and the packet is processed; otherwise it is silently dropped.

Note that this is authentication only - in other words the password only serves to verify that the packet contents are authentic. It does not offer privacy, so all the information within the packet is visible  in the clear.

OSPF MD5 Authentication Detail

One thing I found unclear in RFC 2328 was exactly what data the MD5 hash was calculated over. The RFC states:

Input to the authentication algorithm consists of the OSPF packet and the secret key.

... and clarifies that:

(a) The 16 byte MD5 key is appended to the OSPF packet.

(b) Trailing pad and length fields are added, as
    specified in [Ref17].

(c) The MD5 authentication algorithm is run over the
    concatenation of the OSPF packet, secret key, pad
    and length fields, producing a 16 byte message
    digest (see [Ref17]).

(d) The MD5 digest is written over the OSPF key (i.e.,
    appended to the original OSPF packet). The digest is
    not counted in the OSPF packet's length field, but
    is included in the packet's IP length field. Any
    trailing pad or length fields beyond the digest are
    not counted or transmitted.

Confusingly, Ref17 refers to RFC1321, which defines the MD5 algorithm. MD5 defines a method to pad the input before the hash is calculated, so it's easy to assume that point (b) refers to that - it doesn't. I spent a couple of hours trying to work out why my hashes were coming out to the wrong value before finally figuring it out. To aid others, I've taken the liberty of rewriting the instructions so that they can be understood by thickos such as myself:

Calculating the MD5 Hash

In order to calculate the correct MD5 hash, the following method should be used:

(a) Build the OSPF packet as normal, ensuring that the key number and authentication sequence number are populated. The OSPF length field must contain the total number of bytes in the packet at this point. The checksum must be set to zero.

(b) The authentication key / password in plaintext must be adjusted to exactly 16 bytes, i.e. if the key is longer than 16 bytes then it must be truncated, shorter keys must be padded with null (0x00) bytes until 16 bytes long. The resulting 16 byte "modified authentication key" is then appended to the packet.

(c) The MD5 hash must be calculated over the entire result, i.e. the original OSPF packet plus the 16 byte modified authentication key.

(d) The resulting hash is then written over the modified authentication key in the last 16 bytes of the packet.

Testing / Attacking OSPF Packets

Using the above method it is straightforward to run a dictionary attack as follows:

• Start with a sniffed OSPF packet (see the original dechap blog post for info on how this is extracted).
• Extract the original OSPF packet (start immediately after the IP header and continue up to the length specified in the OSPF header)
• Extract and store the authentication hash (the 16 bytes following the packet) for later comparison
• Zero out the checksum
• For each candidate password, pad or truncate to 16 bytes and append to the original OSPF packet. 
• Calculate the MD5 hash as described above and compare to the value seen in the sniffed packet. A matching hash indicates a matching password.

As of v0.3a, dechap can now be used to automate this process.

Obtaining the Tool

The C source code may be downloaded from: https://github.com/theclam/dechap

Provided the OpenSSL dev libraries are installed it should be possible to simply extract the source code, cd into the directory then run "make".

Using the Tool

As usual - this is for legitimate audit and recovery purposes and must not be used for any kind of malicious activity.

The usage is pretty straightforward - there are only two parameters and both are mandatory. Specify your capture file (original pcap format) with the -c flag and your word list with the -w flag. Here's an example:

lab@lab:~/dechap$ ./dechap -w mywords.txt -c ospf-bcast.cap
Found password "password1" for user OSPF host 10.1.1.1 key 1.
Found password "password1" for user OSPF host 10.1.1.2 key 1.
Found password "password1" for user OSPF host 10.1.1.1 key 1.
lab@lab:~/dechap$

I haven't tried any serious benchmarks for this but it seems reasonably fast. In a worst case scenario (correct key not present) on my creaky old Athlon XP 2100 it can try 100k passwords in under 100ms.

If you try this out, please leave a comment on this post with your experiences - good or bad. Any suggestions would also be welcome (yes, I know BGP exists).

References

RFC2328 - OSPF Version 2
RFC1321 - The MD5 Message-Digest Algorithm

A Tool for Measuring Forwarding Delay in Packet Captures

I have access to some pretty expensive test kit in work. One of its main purposes in life is to measure the latency of traffic streams passing through a network, which is a pretty useful feature. Occasionally, though, the figures produced can be hard to believe and it would be nice to be able to validate them independently. It would also sometimes be nice to be able to see down to the packet level how the delay varies over time.

Since the tester inserts a "unique" signature into each frame, it is possible to do the calculations by hand - simply take a packet capture of traffic entering and leaving the device (at minimum capture using 2 ports on the same box, preferably use one port with a two-source port mirror), then manually compare the timestamps of the packets pre- and post-routing.

Finding matching pairs of packets is pretty tedious, especially for large captures and particularly where high throughput rates mean that there may be thousands of other frames between a "before" and "after". The technique is sound, though, if you're patient.

For some work I was doing recently, I needed to do this on a grand scale. A multi-megabit stream did not appear to be queuing as expected and it was unclear why. Eventually that particular problem was traced, using Wireshark IO graphs, back to overly bursty traffic being offered into the device under test but it made me think it would be very nice to have a tool for doing this kind of verification and, actually, it would not be difficult to write one. So I wrote a tool, in case I needed it in a hurry later on. The impatient may just want to scroll down to "Obtaining the Tool".

How it Works

Packet headers are inevitably be changed by routing (and, in fact, encapsulation could be added or removed in the process) so packet payloads must be compared in order to find "before and after" pairs. The Spirent TestCenter tester includes a 20 byte "signature" in each generated packet, which is always at the very end of the payload. In practice it is not necessary to compare the entire, variable length, payload to pair up packets. Rather it is sufficient, and much faster, to compare the last 20 bytes of each packet for a matching signature.

The process implemented in the tool is to read each packet from the pcap file, storing the following details in a list entry:

• Frame number
• Arrival time
• Signature

Each entry is then stored in a linked list, as in the following diagram:

Each packet read in adds another node of around 36 - 44 bytes in size. This is smaller than the original capture but can still be a considerable amount of memory when working with very large captures.

Once the complete list has been built, the next job is to identify the "before and after" pairs. This is done by considering each list entry in turn, then looking forward in the list for an entry with a matching signature. If such an entry is found, the frame numbers and timestamps of each frame are output along with the time delta between the two frames. Pretty simple, really.

Obtaining the Tool

The tool is available to download as C source code and as a Windows binary at https://github.com/theclam/fwding.

To build from source code simply extract the source,  change into the directory and type "make".

Using the Tool

Once the binary has been compiled or downloaded, simply run it with the name of the pcap file as its only parameter: For example:

lab@lab:~/Projects/fwding$ ./fwding input.cap

Arrival Frame Number, Arrival Time, Departure Frame Number, Departure Time, Forwarding Delay
1, 1359461693.826304, 2, 1359461693.826354, 0.000050
5, 1359461693.826418, 6, 1359461693.826468, 0.000050
7, 1359461693.826585, 8, 1359461693.826701, 0.000116
9, 1359461693.826818, 11, 1359461693.826946, 0.000128
10, 1359461693.826830, 12, 1359461693.826958, 0.000128
17, 1359461693.826999, 18, 1359461693.827014, 0.000015
21, 1359461693.827078, 22, 1359461693.827128, 0.000050
25, 1359461693.827192, 26, 1359461693.827242, 0.000050
27, 1359461693.827359, 29, 1359461693.827488, 0.000129

[...snip...]

The output produced is standard CSV-formatted text.which can be piped or redirected to a file as necessary for manipulation by your favourite spreadsheet or command line tool. Timestamps are in seconds since Unix epoch. Delay is reported in seconds.

Note: The pairing-up mechanism is highly dependent on the test traffic containing unique data in the last 20 bytes of each frame. For tester traffic that's taken care of automatically but your mileage with "real" traffic will vary. I would expect that FTPing a compressed file or playing music / white noise over VOIP  should give relatively good entropy to your data if a tester is not available. For best results filter out non-test traffic beforehand - OSPF hellos and LACPDUs are very repetitive so will generate lots of false hits.

For example, if you want a quick-and-dirty graph of latency over arrival time using gnuplot, just pipe the output to file then use a command such as:

gnuplot> plot "all-ways.txt" using 2:5 with points pt 2

Alternatively, graph the latency by frame number:

gnuplot> plot "all-ways.txt" using 1:5 with points pt 2

Hopefully your output won't look like this - it is an intentionally odd example caused by sending very bursty traffic.

Finally

I always ask but it's never happened yet... if you try the tool out, please leave a comment. I'm interested in feedback, good or bad, and if it doesn't quite do what you want I may change it!

Using Capture Filters with Encapsulated Packets

One of the most annoying things I found when I started working on carrier networks was that while Wireshark's display filters worked perfectly, the capture filters frequently did not. I would regularly set up a capture filter only to find that no packets at all were saved - that's a real pain if you want to pull a few easily described packets out of a 50 Mbps stream across a period of 20 minutes.

After a while I realised that my problem was related to encapsulation. Unlike the hierarchical and detailed display filters, capture filters have to be really fast - that basically means using bit masks and comparing values at fixed offsets. With plain old untagged Ethernet frames the filters work fine, however as soon as you add 802.1Q tags, PPP or MPLS suddenly all the offsets are no longer valid and anything you match will be purely coincidental.

Luckily there are filter keywords to handle that situation. All of the following adjust the offsets for you each time they are used:

vlan [x] - matches a single VLAN tag, the ID of which may optionally be specified by the user
pppoes - matches a PPPoE session header
mpls [x] - matches a single MPLS label, the number of which may optionally be specified by the user

These are very flexible - for example if you are capturing QinQ traffic, you could match all the SMTP packets using:

vlan && vlan && tcp port 25

If you know the VLAN IDs (or MPLS labels) in use, you can narrow the selection based on those. To show all the IGMP passing over a particular MPLS pseudowire with VLAN ID 200, you could use:

mpls 131066 && mpls 131068 && vlan 200 && pppoes && ip proto 2

For a long time I was using makeshift capture filters along the lines of "ether[39] = 2" to match pertinent bytes in the packet (see my next blog post for info on that) however you will probably agree this is much simpler. These filters are equally applicable to Wireshark, Tshark and tcpdump so they may be useful even when forced to capture using some really obscure UNIX box. For Tshark and tcpdump don't forget to put quotes around any expressions that use the ampersand (&).

IGMP Testing, part 1

Maybe it's just my famous inability to find things that are right in front of me but I've needed some tools over the last week that would let me 'play' with IGMP and I've drawn (almost) a total blank.

Firstly, I wanted to generate a good old-fashioned flood of reports to test processing performance and rate limiting.

Plan A was to use the tester for this - despite not having a specific tool for flood testing it does let you create streams of, more or less arbitrary, hand crafted packets. That's the theory, anyway. After carefully putting together a stream profile that should have given me join after join for cycling group numbers I put it to the test - only to find that it had other ideas and was generating complete garbage. By garbage I mean not even the IP headers were correct - the protocol was coming out set to 0xfd (unknown) rather than 0x02 for IGMP and, strangely, the source and destination IPs were populated with the group ID and source that should have been in the report payload. Based on bitter past experiences I didn't waste my time trying to fix that.

OK, time for plan B - back to the packet crafting on a PC. I thought I'd be spoiled for choice but, for Linux anyway, the only option for generating arbitrary IGMP seemed to be nemesis. Nemesis seems to be exactly what I want but it is no longer maintained and won't compile on a modem distro - at least *I* couldn't get it to compile.

My favourite scapy knows what IGMP is from its protocol ID but doesn't have a stack for it, so there was no straightforward way to use that.

Dead end. I couldn't find anything to build me one packet let alone throw 1000 out per second.

Then it occurred that, actually, in normal use the tester can generate valid joins at a civil pace... So I mirrored the tester port and sniffed a genuine join off the wire, whittled the capture file down to the single frame I wanted and fed it to tcpreplay. Yay.

One small problem - it could only manage 100pps and I needed 1000. I noticed it was generating a message every time it sent a packet saying it had re-opened the file, which gave me a hunch that the file operations and CLI might be a bottleneck. I solved that problem the same way as the first - by sniffing the 100pps output for a while and then replaying *that* at full tilt. 960pps... Not quite 1000pps but close enough!

At the last moment it occurred to me that it would be a more convincing test if I cycled the group IDs rather than always reporting on one group. I went back to scapy and, with Wireshark in the other hand, started to play. I thought if I just loaded in the original join packet I could use a loop to tweak a byte or two for the group ID and dump it out to a file which I could then replay.

When I did that I noticed that my router still only showed one group as joined. Rubbish. I had obviously missed something. Looking at the generated file in Wireshark I could see that its checksum was incorrect.

Scapy could re-calculate the IP checksum for me but it didn't understand IGMP so that was going to be a programming exercise. The checksum is only 2 bytes in the payload so it wasn't too hard to adjust. I won't bore you with the maths, check out RFC 3376 if you're curious.

Finally, with that done, I had a pcap file full of valid joins over 100 groups and the ability to fire them out at (nearly) 1000pps. I can't help thinking it should have been easier, though!

Source code to follow - it's very scruffy and fairly fragile but might be useful to someone else... You never know!