Testing the Impact of Local Packet Capture on the Cisco 6500 Series

For a while now, many of the larger Cisco devices (such as 6500 and 7600s) have supported local packet capture. I've always been hesitant to use these in a production environment, primarily due to concerns about the potential performance hit it could cause. Anyway, I decided to test out the potential impact in the lab to see whether it was sometimes / always / never safe to run local captures.

Test Setup

For my test setup I used a 6504-E switch with a modest SUP32-GE-3B supervisor and a 12.2 Advanced IP Services IOS - if it works on that, it should be safe anywhere. For traffic, I used a spare server running Ubuntu with a combination of Ostinato and Scapy.
The configuration was as follows:

Running at full tilt, Ostinato was happily producing 1 Gbps of traffic which went into my node on VLAN 100, out around the loop cable, back into the node and then out of the same interface on VLAN 101. Basically the port is running at 1 Gbps in each direction, so the worst possible case for mirroring a gig port.

Default Settings

The default settings for the capture are pretty conservative - a tiny 2 MB linear capture buffer with a rate limit of 10,000 frames per second. With this config, a 1 Gbps stream of 1500 byte packets fills the buffer in ~ 2.5s, triggering the capture to end. The impact of this is almost impossible to measure at all, with the capture being over so quickly you may not actually see any change in CPU on the 5 second roll-ups.

Lab-6503E#monitor capture start
*Jul 11 14:30:59.205: %SPAN-5-PKTCAP_START: Packet capture session 1 started
Lab-6503E#
*Jul 11 14:31:01.449: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended as the buffer is full, 21845 packets captured
Lab-6503E#show proc cpu hist
                                                              
                                                              
                         222223333322222               8888811
100                                                           
 90                                                           
 80                                                           
 70                                                           
 60                                                           
 50                                                           
 40                                                           
 30                                                           
 20                                                           
 10                                                    *****  
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)

Worst Case

OK, so the world didn't end. The next step was to see how bad it could be so I made the following changes:

• Increased the rate limit to 100,000 fps (max)
• Increased the packet buffer to 64 MB (max)
• Enabled a circular buffer (why?!)

This time I set the capture to run for 60 seconds and checked the impact, which was much more severe:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 14:45:02.953: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 14:46:02.945: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 699040 packets captured
Lab-6503E#show proc cpu hist

    2222244444444444444455555444444444444444444444444444444444
    9999999999888889999966666999999999999999999998888899999999
100
 90
 80
 70
 60                     *****
 50      *****************************************************
 40      *****************************************************
 30 **********************************************************
 20 **********************************************************
 10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)

The impact to the supervisor in this case was much more noticeable - up to 60% CPU utilisation. The scenario is pretty unrealistic (forget circular buffers!) but suffice to say I wouldn't want to do that on a production device.
Now we have a the best and the worst cases, let's look at some realistic use cases and explore some of the other capture parameters that might help us capture what we need without causing havoc on the network.

Narrowing Down the Capture

It's possible to define criteria to decide what gets captured - as the CLI points out, some of these criteria are processed in hardware while others are handled in software:

Lab-6503E(config-mon-capture)#filter ?
  access-group  Filter access-list (hardware based)
  ethertype     Matching ethertype (software based)
  length        Matching L2-packet length (software based)
  mac-address   Matching mac-address (software-based)
  vlan          Filter vlan (hardware based)

Our test traffic consists of nearly a gig of junk run alongside a small 1-per-second ping, which we will decide is "interesting" to us and we want to capture. This traffic profile makes it easy to test ACL, MAC address and length filters and their relative performances.

ACL Filter

As the CLI says, ACL filters are applied in hardware so the junk is discarded at source before it hits the CPU. In this example I set up an ACL as follows:

Lab-6503E(config)#ip access-list extended icmp-only
Lab-6503E(config-ext-nacl)#permit icmp any any
Lab-6503E(config-ext-nacl)#deny ip any any

... and applied it to my capture as follows:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#filter access-group icmp-only

Now, I re-ran the "worst case" test, with massively different results:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 14:49:00.345: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 14:50:00.337: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 60 packets captured
Lab-6503E#show proc cpu hist

         22222222223333355555          11111
100
 90
 80
 70
 60
 50
 40
 30
 20
 10                     *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Two things to note here - because this is a hardware filter, applied in the ASICs:

• Only 1 packet per second was punted to the CPU, resulting in essentially no hit at all
  
• All 60 of the ping packets were received and nothing else
  

In summary, ACL filters are ideal when you know what you want to pull from a big stream as you only pay a CPU penalty for the "good stuff".

MAC Filter

In contrast, the MAC filter runs in software. It's also pretty limited, only matching on source MAC. I repeated the above test but instead of an ACL match, applied a MAC filter as follows:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#filter mac-address 0011.2233.4455

This took us more-or-less back to the worst case:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 15:05:55.197: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 15:06:55.189: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 5 packets captured
Lab-6503E#show proc cpu hist

      44444444443333344444333334444444444444444444444444333333
    4422222111119999900000999990000011111333331111111111888889
100
 90
 80
 70
 60
 50
 40   ********************************************************
 30   ********************************************************
 20   ********************************************************
 10   ********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Yuck. I wouldn't do that in production. This is basically because the software filters are applied after the packets have been punted to the CPU, so you pay a penalty for the garbage as well as the good stuff. You'll notice that it only captured 5 packets as well, more on this later but that's another side effect of software filters.

Length Filter

The frame length filter is another software-based mechanism, which means it's pretty terrible under load, too. Our junk traffic consists of large frames, our interesting traffic is small, so let's configure the capture to only catch the short frames:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#filter length 0 100 

Again, the output is pretty miserable:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 15:15:12.145: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 15:16:12.137: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 17 packets captured
Lab-6503E#show proc cpu hist

    1111444443333344444444443333333333333333333333333444443333
    4444000009999911111222229999999999999999999999999222229999
100
 90
 80
 70
 60
 50
 40     ******************************************************
 30     ******************************************************
 20     ******************************************************
 10 **********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Again, the CPU took a hammering and we only captured a few of the ping packets - 17 out of 60.

Quirks / Order of Operations

Now you may think that software filters might be OK if we just reduce the rate-limit configured on the capture:

Lab-6503E(config)#monitor session 1
Lab-6503E(config-mon-capture)#rate-limit 100

This *does* do what we want for the CPU load - here's an example with a MAC filter:

Lab-6503E#monitor capture circular buffer size 65535 start for 60 sec
*Jul 11 15:26:43.793: %SPAN-5-PKTCAP_START: Packet capture session 1 started
*Jul 11 15:27:43.785: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 0 packets captured
Lab-6503E#show proc cpu hist

                    11111          11111          6666633333
100
 90
 80
 70
 60
 50
 40
 30
 20
 10                                               *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

Great - nothing to see here. But also nothing to see in the capture buffer - 0 packets captured!
Just for fun let's try the same with a hardware ACL filter:

*Jul 11 15:25:26.921: %SPAN-5-PKTCAP_STOP: Packet capture session 1 ended after the specified time, 60 packets captured

Why is this? Well, it's the order of operations. Basically the flow for hardware ACL filters is:

So the filter throws out the junk before the rate limiter, meaning that the rate limiter only counts the good stuff. If the good stuff exceeds the rate limit then you'll lose some of it but the junk doesn't count.
Compare that to the flow for the software filters:

The software filters are applied after the rate limiter, so clearly when the rate limit is exceeded you throw out a mix of good and bad traffic, then pick out what's left of the good. If your traffic is overwhelmingly garbage, you may not get any of the good stuff at all!

Summary - Play it Safe

So in answer to the question "is it safe to run a local capture on a production 6500" - packet capture on even a relatively modest SUP32-3B supervisor is pretty safe provided you are cautious. If you want to do this in a busy production environment then my message to you is:

• Use ACLs where at all possible
  
• Set the rate limit to a sensible value (the default 1000 fps is fine for most cases)
  
• Use linear buffers of a sensible size (do you really need 64MB of capture?)
  
• Limit the frame count or capture duration at first (it may turn out there is a lot more "interesting" traffic than you thought!)
  

I'm convinced enough to do this in production but obviously I'm only testing one device, on one release of code, so don't blame me if you try it on something different and encounter a bug!

Adjusting timestamps in PCAP files

Many times in the past I've had to look at a pair of pcap files side by side in order to troubleshoot an issue. More often than not, one of the PCAP files was produced on a ropey old laptop whose clock is "almost right" - the timestamps between the two files then don't tie up and it is a pain to keep working out "if it's time X in that file, I need to look at time Y in this file..."

This week I overheard a colleague in the office having exactly that problem and thought it wouldn't be too hard to build a utility to time shift pcap files by a specified amount. So here it is:

https://github.com/theclam/capshift

Installation

As explained in the readme, it should be possible to compile on any system with gcc using only the standard libraries. Just download the capshift.c and capshift.h files and compile (gcc -o capshift capshift.c), or download a binary if one exists for your system.

Usage

Capshift takes three arguments, all mandatory:

• The input pcap file, specified using -r
• The output pcap file, specified using -w
• The time offset value (positive or negative), specified using -o

Here's an example:

Harrys-MacBook-Air:capshift foeh$ tshark -ta -r before.cap
  1 15:30:45.978539 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4748/35858, ttl=128
  2 15:30:45.979407 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4748/35858, ttl=255
  3 15:30:46.979315 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4749/36114, ttl=128
  4 15:30:46.980274 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4749/36114, ttl=255
  5 15:30:47.980323 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4750/36370, ttl=128
  6 15:30:47.981215 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4750/36370, ttl=255
  7 15:30:48.981387 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4751/36626, ttl=128
  8 15:30:48.982277 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4751/36626, ttl=255
Harrys-MacBook-Air:capshift foeh$ capshift -r before.cap -w after.cap -o -0.5

Parsing capfile, attempting to shift backward by 0.500000 seconds...

8 frames processed.
Harrys-MacBook-Air:capshift foeh$ tshark -ta -r after.cap
  1 15:30:45.478539 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4748/35858, ttl=128
  2 15:30:45.479407 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4748/35858, ttl=255
  3 15:30:46.479315 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4749/36114, ttl=128
  4 15:30:46.480274 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4749/36114, ttl=255
  5 15:30:47.480323 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4750/36370, ttl=128
  6 15:30:47.481215 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4750/36370, ttl=255
  7 15:30:48.481387 192.168.1.25 -> 192.168.1.1 ICMP 74 Echo (ping) request  id=0x0001, seq=4751/36626, ttl=128
  8 15:30:48.482277 192.168.1.1 -> 192.168.1.25 ICMP 74 Echo (ping) reply    id=0x0001, seq=4751/36626, ttl=255

As usual, if you find this useful or have any feedback (good or bad) please leave a comment!

Removing VLAN/MPLS/PPPoE/GRE/GTP/VXLAN Encapsulation Headers from pcap Files

Many years ago, when I worked in a school, I used to port mirror our proxy server to an old PC running driftnet and leave the screen where the kids could see it as a warning that staff could "see what you're doing on the Internet". I haven't played with driftnet since but certainly at the time it could only handle native frames (no VLAN tags, certainly no MPLS or PPPoE). I vaguely remember some other tools being similar, unfortunately I can't remember which ones.

Looking at the analytics for this blog, I can see I'm not the only one who's had the problem. It's certainly not the number one issue that people are searching for when they get here but there have been a few and the thought occurred that the packet processing engine I wrote for dechap would be really good for this task - it already stripped back VLANs and MPLS, plus it knows how to detect PPPoE and L2TP.

After a couple of hours it was working to the point of being able to strip VLANs and MPLS off, with a little more effort PPPoE also gave way. GRE came quite easily, too, as it has simple headers and uses the same etypes as Ethernet.

Anyway, here is "stripe" (from STRIP Encapsulation), a command line tool which takes a pcap file as input, re-assembles IP fragments and strips off all the encap it can (currently VLAN tags, MPLS shim headers, PPPoE, L2TP, GRE GTP and VXLAN) then outputs another pcap containing just payload over Ethernet.

**UPDATE** - Version 0.3b now adds support for VXLAN.

Download

Stripe is available from my github: https://github.com/theclam/stripe

Usage

The command line is pretty straightforward, as shown in the online help:

Harrys-MacBook-Air:stripe foeh$ ./stripe
stripe: a utility to remove VLAN tags, MPLS shims, PPPoE, L2TP headers,
etc. from the frames in a PCAP file and return untagged IP over Ethernet.
Version v0.1 alpha, November 2014

Usage:
./stripe -r inputcapfile -w outputcapfile

Where inputcapfile is a tcpdump-style .cap file containing encapsulated IP 
outputcapfile is the file where the decapsulated IP will be saved

Harrys-MacBook-Air:stripe foeh$ 

Simply specify the files you want to read encapsulated packets from (-r) and write the cleaned up packets to (-w). Stripe will remove as many layers of encap as it can until you are left with straight payload over Ethernet.

How it Works

The majority of stripe's work is done by the "decap" function. This function takes in a block of memory, a length parameter, a data type hint and a frame template. The process runs as follows:

1. If the type is Ethernet, populate the source / destination MACs of the frame template
2. If the type has an Ethertype or protocol type field, use this to populate the ethertype of the frame template
3. If the next protocol is possibly or definitely payload, set the payload pointer of the frame template to the address of the next protocol and return
4. If the next protocol is possibly or definitely encapsulation, call decap against the remainder of the packet

So essentially it eats up encap, recording MACs and protocol types as it goes, until there is no more encap left. By the end there is a fully populated frame template with source and destination MAC (the innermost copy if there are multiple as in the case of MPLS pseudowires), the etherype of the payload and the payload itself. Piecing these together gives a minimally encapsulated frame, i.e. one with just an Ethernet header and payload.

Here is a worked example for a frame with VLAN, MPLS, GRE over IP and an IP payload:

Step 1 - The "decap" function is called on the entire frame. Since the first header is Ethernet, the frame template gets populated with the source / destination MACs and the etype from the Ethernet header. The frame template's length field gets populated with the size of the frame minus the Ethernet header and the payload pointer is adjusted to point at the next header. The decap function then calls itself on the remainder of the frame, hinting that the type is VLAN tag based on the current header's etype.

Step 2 - The decap function now considers the partial frame starting at the VLAN tag. Since the VLAN tag has an etype associated, the frame template's etype is overwritten with the one from the VLAN header. The length is overwritten with the length of the payload after the VLAN header and the pointer adjusted to point at the next header. The decap function then calls itself again with a hint of MPLS, based on the etype in the VLAN header.

Step 3 - The decap function now considers the partial frame starting at the MPLS label. Since the MPLS label is bottom of stack, we know there are no more MPLS labels left . Unfortunately there is no protocol type in an MPLS header (these are signaled on the control plane) so we have to take a peek at the byte immediately following the label. If we find a "4" or a "6" in the high order nibble then we have to guess that the next protocol is IPv4 or IPv6, respectively. If the following four bytes are all zeroes then we assume Ethernet over MPLS with control word, otherwise we assume Ethernet over MPLS without control word. In this case we find a 4 in the low nibble, so call decap with an "IP" hint.

Step 4 - The IP header tells us that GRE is the next protocol so for now nothing changes in the frame template (the remainder could be decodable or not). We just call decap again on the GRE part...

Step 5 - The GRE header is decoded and the etype is copied into the frame header. The length of the remaining payload is updated in the frame template and the pointer is adjusted. Decap is called on the next header, which is IP. When the decap function inspects the IP payload it can go no further and just returns the frame template.

In essence, the process has started with a deeply encapsulated frame and ended with IP over Ethernet. The source and destination MACs are taken from the innermost ones found (which in this case is the outermost Ethernet header) but with the etype changed to match the payload, which is the first non-encapsulating payload found in the frame, in this case the second IP.

References

https://tools.ietf.org/html/rfc2784
https://tools.ietf.org/html/rfc1701
http://www.ieee802.org/1/pages/802.1Q.html
http://www.3gpp.org/DynaReport/29060.htm

Crippling CPU Load on Back to Back ASAs

I was recently involved in troubleshooting a problem where an ASA firewall's CPU was hitting 100%. One of its interfaces was seeing much higher traffic levels than the others, so we did some fairly run-of-the-mill troubleshooting including a packet capture. What this showed was the same, seemingly innocuous, packet repeated thousands upon thousands of times.

The payload was identical, in fact everything from the IP layer and up remained identical from one frame to the next. The only thing that varied was that the source and destination MAC addresses were swapped each time - clearly the packet was ping-ponging between two devices.

We checked the MACs and found they were legitimate - one was the local firewall, while the other was its default gateway - another ASA upstream towards the Internet.

This got our attention. First of all there was a routing loop, which is bad enough, but a packet should never be able to loop forever like that. That's why we have Time To Live (TTL) after all - the number which decrements by one each time a packet goes through a routed hop with the packet being thrown away when its value reaches zero. The key thing here is that the packet, including its TTL, was not changing at all so it never got removed from the system.

The cause of the routing loop was relatively easily found by looking at the source and destination IPs on the packet. The setup was as follows:

What had happened here was that a RAS user had connected to the tenant firewall using their IPSec client and started talking to some devices on the server LAN:

At some point the IPSec session had ended while an internal device was still sending traffic towards the user. This creates an interesting corner case:

The routing is "correct" here - the multi-tenant firewall needs to route the RAS subnet via the tenant firewall so that RAS users can connect to shared resources. The tenant firewall needs to route the traffic outwards for it to hit the right crypto maps. The problem comes when a packet is destined for an IP in the RAS pool which is not associated with a live VPN session.

Our bodge to get us out of the immediate hole was to put a deny entry inbound on the mutli-tenant firewall for anything targeted at a RAS pool address. These packets should never make it onto the transit LAN as any legitimate traffic to that range would need to be tunneled via IPSec and therefore the multi-tenant firewall would see a public IP as the destination. After a lot of thinking we couldn't come up with a better answer than this and decided just to stop calling it a bodge.

OK, so first problem solved. Next question, why was the packet looping forever without ever reducing its TTL?

Root Cause

As it turns out this is by design on the ASA (and the good old fashioned PIX & FWSM before it). The idea is that if the firewall behaved like any other routed hop and decremented the TTL then it would be visible in traceroutes. To be fair if it did decrement TTL it would just appear as a black hole in the trace as the ASA doesn't really "do" unreachables unless you force its hand. This normally doesn't cause any problems, even if there is a routing loop. Take a typical deployment where an ASA is attached to a router as shown below:

If we get a loop between the ASA and a traditional router then the packet will eventually be taken out of the loop. Even though the ASA doesn't decrement the TTL, the router does so it eventually gets dropped - half as fast as normal and always by the router (which will punt the packet to the CPU and usually generate an ICMP TTL expired, which can be pretty CPU intensive on small devices), but it does get dropped eventually.

The problem comes when we have a pair of non-decrementing devices (ASAs) back to back at layer 2. Both devices route the packet but neither device decrements the TTL, so if there is a loop between the two it the packet will go around and around forever. Eugh...

The moral of the story is that it's probably best not to put ASAs back to back. I could have sworn I'd seen this setup in Cisco whitepapers before but, now that I look, I can't find it anywhere. The closest I can find is IOS firewall back to back with ASA or two ASAs with a server between. Perhaps there's a good reason for that :)

As with my situation, though, in most cases by the time you get to realise there is a problem the hardware has long since been bought, installed and is carrying live service. So what can you do?

Well, as noted above you can use ACLs to block potential loop traffic but in all honesty that just fixes by exception. You could be fairly liberal with what you block (e.g. drop all RFC 1918 addresses where you would expect to only see public IPs) but it's still imperfect.

Making the ASA decrement TTL

A better idea would be to have at least one of the ASAs decrement TTL. It's a bit uncomfortable to retro-fit but there is a way built in to ASA versions 8.0(3) and above using "set connection decrement-ttl" under a policy map. There are two different ways to do it, one is to adjust the "global_policy" policy map which applies to the entire device by default, or you can create a new policy map to apply to a single interface.

Here's how to apply it to the entire device:

policy-map global_policy
 class class-default
  set connection decrement-ttl
!

The effect is immediate as the global_policy is applied to all traffic by default. Alternatively, if you only want to apply it to specific interfaces, you can create a separate policy map and apply it as follows:

policy-map asa_workaround
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
  inspect icmp
 class class-default
  set connection decrement-ttl
!
service-policy asa_workaround interface interface-name

The above is modeled on the standard default policy & inspections, if you've changed yours from default you probably don't need to be reading this!

Summary

So there you have it - ASAs back to back is a bit dangerous unless you take measures to protect against routing loops. This can be in the form of strict ACLs or by enabling TTL decrement, either globally or on specific interfaces.

References

Cisco guide to enabling traceroute through ASA

Cisco guide to modular policy framework on ASA

Be Careful where you use TACACS!

As part of my on-going work to add more and more protocols into my hobby project dechap, I started looking into the workings of TACACS+ today. I was looking to see whether TACACS+ would be a likely candidate as the next attackable protocol. I had in my mind a couple of events from my past that made me suspect that a TACACS+ server couldn't really tell when an incorrect key was in use, other than that the packet decoded to garbage that it was then not able to interpret. If that is the case, it would be very hard to attack the protocol as there is not a straightforward way to tell when you've hit the correct key. I'll let you know when I've figured that out because as I read the protocol spec something else derailed my train of thought.

Like most network engineers I was raised on Cisco's literature and read through plenty of their whitepapers such as this one comparing RADIUS and TACACS+. This document is full of useful facts such as "RADIUS does not allow users to control which commands can be executed on a router and which cannot" (no, Cisco, because for some reason you notched out the ability to do so in IOS) and that RFC compliance doesn't guarantee interoperability. One of the parts I always remembered and believed, in the olden days at least, was the part that says that far beyond the argument of TCP being better than UDP, TACACS+ is more secure than RADIUS due to the way it encrypts the entire message body. I suspect most people just swallow that without chewing, I know I did. Encryption is good, so more encryption must be better.

RADIUS, by comparison, uses CHAP for secure password authentication but makes no attempt to encrypt the parameters within requests and responses. To be fair, these can just be read in plain text straight off the wire. The shared key used when configuring RADIUS is purely an authentication measure - it guards against arbitrary spoofed requests and tampering but does not offer any kind of privacy.

In reality the biggest part of the decision when choosing one or the other will be a "horses for courses" argument. Want to authenticate PPP subscribers with a nice wholesale / retail proxy model? Use RADIUS. Want to authenticate logins to infrastructure devices and authorise specific commands down to the parameter level? Use TACACS+.

A Slightly More Balanced Comparison

Let's compare the relative merits, security wise of each protocol.

Message Encryption

TACACS+, as previously mentioned, encrypts the entire message body using a pre-shared key. It only leaves the header in the clear, so without the key it is only really possible to determine who is client and who is server, plus what kind of messages are being passed (authentication or authorisation, query or response).

RADIUS uses a pre-shared key to authenticate messages going back and forth, but the messages themselves are unencrypted and can easily be read straight off the wire.

Credentials

RADIUS relies on CHAP for user credential validation. The NAS sends a "random" challenge to the user, who produces a one-way hash of the challenge data and password (plus some other stuff) and returns that to the NAS. The NAS then sends the challenge and response off to the RADIUS, meaning that the credentials are never sent over the wire in any reversible way. In order to get the password an attacker must capture the challenge and response data then run a dictionary or brute force attack. On the down side, the RADIUS server itself must have a plaintext copy of the password available in order to verify that a response is correct given the challenge. Clearly if the RADIUS server's password database is compromised then things get quite sticky. For proxy RADIUS, the proxy does not need access to plaintext passwords. In summary, passwords are safe in flight but exposed at rest.

TACACS+ relies on the pre-shared key to encrypt everything, including password information. No form of CHAP or similar system is used, so credentials are passed in a reversible form over the wire. It's encrypted, though, so don't worry - unless an attacker knows the key it's all just gibberish. On the positive side of this, the TACACS+ server does not need to store plaintext passwords for the end users and can instead keep one-way hashes on disk meaning that a compromised database is arguably less of an issue. Safe at rest and safe in flight.

Or is it? Think about the typical use case again. TACACS+ is more-or-less always used to authenticate CLI users logging into routers and switches. The key used to encrypt the TACACS+ communications is stored in the device config, either completely in plain text or using (trivially) reversible type 7 encryption. Virtually all devices are left with the password recovery mechanism enabled. Most of the time the key is re-used across every device in the estate since it makes administration easy and, what's the risk anyway?

A Really Easy Attack

I'd like to point out I'm not suggesting or endorsing any kind of illegal or immoral behaviour. Even as a joke :)

The thought occurs that many TACACS+ managed devices are in remote locations - far flung or sparsely populated offices, in accessible wiring closets, even (*shudder*) customer sites. Given physical access to a device, it's very possible to make a terminal emulator script to perform a password recovery, dump out the config then reset the config register to its original value within only a couple of seconds more than it takes to double-reboot the device. I know this because I did it many moons ago (I don't have it any more - it was pretty easy to write, though).

If I were an evil adversary who wanted to get some credentials, perhaps a good way would be:

1. Feign a power cut, on-site work or some other convincing reason for a device to go down
2. Take the device off the network (to avoid it phoning home by syslog / SNMP) and perform a quick password recovery / config dump before putting it back to its original condition:

*break*
Readonly ROMMON initialized
program load complete, entry point: 0x8000f000, size: 0xcb80

monitor: command "boot" aborted due to user interrupt
rommon 1 > confreg 0x2142

You must reset or power cycle for new config to take effect
rommon 2 > reset
*snip snip*
Would you like to enter the initial configuration dialog? [yes/no]: no

Press RETURN to get started!
Router>enable
Router#show startup-config | include tacacs
aaa authentication login default group tacacs+ local
aaa authorization exec default group tacacs+ none
aaa authorization configuration default group tacacs+
tacacs-server host 10.4.4.10
tacacs-server key supersecret
Router#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#config-register 0x2102
Router(config)#^Z
Router#reload
System configuration has been modified. Save? [yes/no]: no
Proceed with reload? [confirm]


3. Stick a sniffer inline between the device and wherever its administrators are
4. Call in a fault saying that since the power cut / whatever nothing attached to that router / switch is able to see the network - perhaps leave the LAN side disconnected for authenticity
5. Capture TACACS+ packets as the administrators log in to investigate
6. Come up with some compelling reason for comms to go down again while the sniffer is taken out

Now with the config dump it is trivial to get the TACACS server key - it's either just there in the clear or can be decoded from the type 7 encrypted version using any number of free tools. If you put this key into the TACACS+ protocol settings of Wireshark (in the preferences screen expand the protocols area then scroll down to TACACS+), it will happily decrypt the captured packets from step 5:

Configuring Wireshark

Viewing the Decrypted Payload

Now you have the administrator username(s) and password(s) in plain text!

Yikes!

Are you sure you still want to run TACACS on that remote box?

BGP support added to dechap

Hot on the heels of adding the ability to attack OSPF MD5 authentication, I've added BGP support to dechap. It is now possible to feed a pcap file with PPPoE, L2TP, RADIUS, OSPF and BGP packets to the same tool and perform offline dictionary attacks on the authentications within.

As usual, if you're not interested in the theory just skip right to the end for the download link.

TCP MD5 Signatures

BGP authentication uses the MD5 Signature TCP option field, which is defined in RFC 2385. Personally, I found this RFC very vague and it took a lot of iterations to get the technique right. It's particularly fuzzy about what is included in the hash, what isn't and how to present values correctly. I'm hoping to document the process a little more clearly for the next poor guy who tries to implement it as I couldn't find a sufficiently detailed reference anywhere.

RFC 2385 states that the hash must be calculated over the following:

1. the TCP pseudo-header (in the order: source IP address,
   destination IP address, zero-padded protocol number, and
   segment length)
2. the TCP header, excluding options, and assuming a checksum of
   zero
3. the TCP segment data (if any)
4. an independently-specified key or password, known to both TCPs
   and presumably connection-specific

Now, maybe it's just me, but this raised a lot of questions in my mind. Zero padding usually means to fill the trailing space with zeros, but padding the second byte would effectively multiply the protocol number by 256 so should it be a leading zero? Which headers and options are included in the "segment length"? Should the pad bytes be copied with the TCP header?

Through a lot of trial and error I found that:

• The zero padding goes before the protocol number
• The "segment length" includes the TCP header, the TCP options (including room for the MD5 signature option being calculated) and the actual payload data
• The copied TCP header should be 20 bytes long, i.e. includes two padding bytes after the (zeroed out) checksum. The header length remains as-is, including the length of the options.
• The TCP segment data starts immediately after the TCP options and runs to the last byte indicated by the IP length field
• The null byte terminating the password is not passed to the hash algorithm

 The resulting hash value is then stored inside the MD5 signature option (kind 19, length 18).

Checking / Attacking BGP Packets

Using the above method it is straightforward to run a dictionary attack as follows:

• Start with a sniffed BGP packet (see the original dechap blog post for info on how this is extracted).
• Extract and store the authentication hash (look for option kind 19) for later comparison
• Put together the "pseudoheader" as described above
• Append the TCP header without options
• Append the TCP payload
• Append the candidate password
• Calculate the MD5 hash over the complete data set and compare to the value seen in the sniffed packet. A matching hash indicates a matching password.

As of v0.4a, dechap can now be used to automate this process.

Obtaining the Tool

The C source code may be downloaded from: https://github.com/theclam/dechap

Provided the OpenSSL dev libraries are installed it should be possible to simply extract the source code, cd into the directory then run "make". I've only tested this under Ubuntu Linux but there are very few dependancies so I would imagine it will work on most distributions.

Using the Tool

As usual - this is for legitimate audit and recovery purposes and must not be used for any kind of malicious activity.

The usage is pretty straightforward - there are only two parameters and both are mandatory. Specify your capture file (original pcap format) with the -c flag and your word list with the -w flag. Here's an example:

lab@lab:~/dechap$ ./dechap -w mywords.txt -c bgp.cap
Found password "password1" for TCP from 10.0.0.2 to 10.0.0.1.
Found password "password1" for TCP from 10.0.0.1 to 10.0.0.2.
Found password "password1" for TCP from 10.0.0.2 to 10.0.0.1.lab@lab:~/dechap$
I'm not sure how quickly it runs but it doesn't seem quite as quick as the OSPF version. I suppose BGP packets tend to be a little bigger than OSPF so there's more to hash. You can improve the speed by only including one packet for each source / destination pair in each capture as, at present, it doesn't check for multiple packets between pairs and attacks each instance individually.

If you try this out, please leave a comment on this post with your experiences - good or bad. Any suggestions would also be welcome, particularly for other protocols to attack.

References

RFC2385 - Protection of BGP Sessions via the TCP MD5 Signature Option
RFC1321 - The MD5 Message-Digest Algorithm

Offline Attack on MD5 keys in captured OSPF packets

A few months ago I released a tool called dechap which finds PPPoE, L2TP and RADIUS authentications in pcap files and performs dictionary attacks against them. Since writing dechap I've always thought it would be more useful if it were able to do a similar thing with OSPF packets.

Well, the good news is that I've finally got around to adding OSPF support to dechap! Woo and yay! If you just want the tool, scroll straight to the bottom. If you're interested in the theory, read on.

OSPF Authentication Basics

OSPF, or more accurately OSPFv2 as defined in RFC2328, has three options for authenticating incoming packets:

Null: no authentication is performed at all.

Password: a plaintext password is added in the clear to each OSPF packet. If the password contained in an incoming packet matches the one configured locally then the packet is considered valid and is processed, otherwise it is silently ignored.

Message Digest: an MD5 hash is calculated over a combination of the OSPF packet contents and the password. The hash output is then added to the OSPF packet before transmission. When a packet arrives, the receiving router computes an MD5 hash of the packet contents plus its locally stored password. If the calculated hash matches the one attached to the incoming packet then the check passes and the packet is processed; otherwise it is silently dropped.

Note that this is authentication only - in other words the password only serves to verify that the packet contents are authentic. It does not offer privacy, so all the information within the packet is visible  in the clear.

OSPF MD5 Authentication Detail

One thing I found unclear in RFC 2328 was exactly what data the MD5 hash was calculated over. The RFC states:

Input to the authentication algorithm consists of the OSPF packet and the secret key.

... and clarifies that:

(a) The 16 byte MD5 key is appended to the OSPF packet.

(b) Trailing pad and length fields are added, as
    specified in [Ref17].

(c) The MD5 authentication algorithm is run over the
    concatenation of the OSPF packet, secret key, pad
    and length fields, producing a 16 byte message
    digest (see [Ref17]).

(d) The MD5 digest is written over the OSPF key (i.e.,
    appended to the original OSPF packet). The digest is
    not counted in the OSPF packet's length field, but
    is included in the packet's IP length field. Any
    trailing pad or length fields beyond the digest are
    not counted or transmitted.

Confusingly, Ref17 refers to RFC1321, which defines the MD5 algorithm. MD5 defines a method to pad the input before the hash is calculated, so it's easy to assume that point (b) refers to that - it doesn't. I spent a couple of hours trying to work out why my hashes were coming out to the wrong value before finally figuring it out. To aid others, I've taken the liberty of rewriting the instructions so that they can be understood by thickos such as myself:

Calculating the MD5 Hash

In order to calculate the correct MD5 hash, the following method should be used:

(a) Build the OSPF packet as normal, ensuring that the key number and authentication sequence number are populated. The OSPF length field must contain the total number of bytes in the packet at this point. The checksum must be set to zero.

(b) The authentication key / password in plaintext must be adjusted to exactly 16 bytes, i.e. if the key is longer than 16 bytes then it must be truncated, shorter keys must be padded with null (0x00) bytes until 16 bytes long. The resulting 16 byte "modified authentication key" is then appended to the packet.

(c) The MD5 hash must be calculated over the entire result, i.e. the original OSPF packet plus the 16 byte modified authentication key.

(d) The resulting hash is then written over the modified authentication key in the last 16 bytes of the packet.

Testing / Attacking OSPF Packets

Using the above method it is straightforward to run a dictionary attack as follows:

• Start with a sniffed OSPF packet (see the original dechap blog post for info on how this is extracted).
• Extract the original OSPF packet (start immediately after the IP header and continue up to the length specified in the OSPF header)
• Extract and store the authentication hash (the 16 bytes following the packet) for later comparison
• Zero out the checksum
• For each candidate password, pad or truncate to 16 bytes and append to the original OSPF packet. 
• Calculate the MD5 hash as described above and compare to the value seen in the sniffed packet. A matching hash indicates a matching password.

As of v0.3a, dechap can now be used to automate this process.

Obtaining the Tool

The C source code may be downloaded from: https://github.com/theclam/dechap

Provided the OpenSSL dev libraries are installed it should be possible to simply extract the source code, cd into the directory then run "make".

Using the Tool

As usual - this is for legitimate audit and recovery purposes and must not be used for any kind of malicious activity.

The usage is pretty straightforward - there are only two parameters and both are mandatory. Specify your capture file (original pcap format) with the -c flag and your word list with the -w flag. Here's an example:

lab@lab:~/dechap$ ./dechap -w mywords.txt -c ospf-bcast.cap
Found password "password1" for user OSPF host 10.1.1.1 key 1.
Found password "password1" for user OSPF host 10.1.1.2 key 1.
Found password "password1" for user OSPF host 10.1.1.1 key 1.
lab@lab:~/dechap$

I haven't tried any serious benchmarks for this but it seems reasonably fast. In a worst case scenario (correct key not present) on my creaky old Athlon XP 2100 it can try 100k passwords in under 100ms.

If you try this out, please leave a comment on this post with your experiences - good or bad. Any suggestions would also be welcome (yes, I know BGP exists).

References

RFC2328 - OSPF Version 2
RFC1321 - The MD5 Message-Digest Algorithm

A Tool for Measuring Forwarding Delay in Packet Captures

I have access to some pretty expensive test kit in work. One of its main purposes in life is to measure the latency of traffic streams passing through a network, which is a pretty useful feature. Occasionally, though, the figures produced can be hard to believe and it would be nice to be able to validate them independently. It would also sometimes be nice to be able to see down to the packet level how the delay varies over time.

Since the tester inserts a "unique" signature into each frame, it is possible to do the calculations by hand - simply take a packet capture of traffic entering and leaving the device (at minimum capture using 2 ports on the same box, preferably use one port with a two-source port mirror), then manually compare the timestamps of the packets pre- and post-routing.

Finding matching pairs of packets is pretty tedious, especially for large captures and particularly where high throughput rates mean that there may be thousands of other frames between a "before" and "after". The technique is sound, though, if you're patient.

For some work I was doing recently, I needed to do this on a grand scale. A multi-megabit stream did not appear to be queuing as expected and it was unclear why. Eventually that particular problem was traced, using Wireshark IO graphs, back to overly bursty traffic being offered into the device under test but it made me think it would be very nice to have a tool for doing this kind of verification and, actually, it would not be difficult to write one. So I wrote a tool, in case I needed it in a hurry later on. The impatient may just want to scroll down to "Obtaining the Tool".

How it Works

Packet headers are inevitably be changed by routing (and, in fact, encapsulation could be added or removed in the process) so packet payloads must be compared in order to find "before and after" pairs. The Spirent TestCenter tester includes a 20 byte "signature" in each generated packet, which is always at the very end of the payload. In practice it is not necessary to compare the entire, variable length, payload to pair up packets. Rather it is sufficient, and much faster, to compare the last 20 bytes of each packet for a matching signature.

The process implemented in the tool is to read each packet from the pcap file, storing the following details in a list entry:

• Frame number
• Arrival time
• Signature

Each entry is then stored in a linked list, as in the following diagram:

Each packet read in adds another node of around 36 - 44 bytes in size. This is smaller than the original capture but can still be a considerable amount of memory when working with very large captures.

Once the complete list has been built, the next job is to identify the "before and after" pairs. This is done by considering each list entry in turn, then looking forward in the list for an entry with a matching signature. If such an entry is found, the frame numbers and timestamps of each frame are output along with the time delta between the two frames. Pretty simple, really.

Obtaining the Tool

The tool is available to download as C source code and as a Windows binary at https://github.com/theclam/fwding.

To build from source code simply extract the source,  change into the directory and type "make".

Using the Tool

Once the binary has been compiled or downloaded, simply run it with the name of the pcap file as its only parameter: For example:

lab@lab:~/Projects/fwding$ ./fwding input.cap

Arrival Frame Number, Arrival Time, Departure Frame Number, Departure Time, Forwarding Delay
1, 1359461693.826304, 2, 1359461693.826354, 0.000050
5, 1359461693.826418, 6, 1359461693.826468, 0.000050
7, 1359461693.826585, 8, 1359461693.826701, 0.000116
9, 1359461693.826818, 11, 1359461693.826946, 0.000128
10, 1359461693.826830, 12, 1359461693.826958, 0.000128
17, 1359461693.826999, 18, 1359461693.827014, 0.000015
21, 1359461693.827078, 22, 1359461693.827128, 0.000050
25, 1359461693.827192, 26, 1359461693.827242, 0.000050
27, 1359461693.827359, 29, 1359461693.827488, 0.000129

[...snip...]

The output produced is standard CSV-formatted text.which can be piped or redirected to a file as necessary for manipulation by your favourite spreadsheet or command line tool. Timestamps are in seconds since Unix epoch. Delay is reported in seconds.

Note: The pairing-up mechanism is highly dependent on the test traffic containing unique data in the last 20 bytes of each frame. For tester traffic that's taken care of automatically but your mileage with "real" traffic will vary. I would expect that FTPing a compressed file or playing music / white noise over VOIP  should give relatively good entropy to your data if a tester is not available. For best results filter out non-test traffic beforehand - OSPF hellos and LACPDUs are very repetitive so will generate lots of false hits.

For example, if you want a quick-and-dirty graph of latency over arrival time using gnuplot, just pipe the output to file then use a command such as:

gnuplot> plot "all-ways.txt" using 2:5 with points pt 2

Alternatively, graph the latency by frame number:

gnuplot> plot "all-ways.txt" using 1:5 with points pt 2

Hopefully your output won't look like this - it is an intentionally odd example caused by sending very bursty traffic.

Finally

I always ask but it's never happened yet... if you try the tool out, please leave a comment. I'm interested in feedback, good or bad, and if it doesn't quite do what you want I may change it!

Recovering CHAP Passwords from Sniffed PPPoE Sessions

In a previous blog post I outlined the theory behind setting up a PPPoE session including PPPoE discovery, LCP, NCPs and, more relevant to this post, the basics of CHAP authentication. At the time I was writing the post I wondered how easy it would be to work back from the CHAP messages on the wire to the original credentials, so I decided to find out.

Recap of CHAP Theory

As a reminder (or a very quick introduction), the CHAP process works something like this:

CHAP Authentication

1. The party requiring the opposite peer to authenticate (i.e. "server") sends a CHAP challenge message containing a challenge ID and some unpredictable "random" data.
2. The party being authenticated (i.e. "client") concatenates the authentication ID, the password and the challenge data into a single unit, then generates an MD5 hash of that. The resulting hash, plus the client name (user ID or hostname) is passed to the server as a CHAP response.
3. The server compares the incoming hash to the value it obtains by performing the same calculation locally and returns a CHAP success or CHAP failure message.

Now, clearly, if the CHAP challenge and response messages can be captured then an offline brute force attack can be mounted against the password. This can be achieved by simply extracting the authentication ID, challenge data and response from the relevant messages and then trying candidate passwords until one (hopefully) generates a hash identical to that seen in the response message.

The Attack in Practice

While the process is intuitively simple, as usual there are a few corner cases to cover. Recovering CHAP authentications from a capture file full of other junk requires a certain amount of processing logic, then responses must be re-united with their corresponding challenges before they can be attacked.

Gathering CHAP Packets

I wanted the tool to be flexible with regards to encap. Since I work primarily on carrier networks, I get really frustrated by tools that do a job perfectly but only accept untagged, unencapsulated frames. Once you have a packet capture in your hand, realising that it can't be used because it has two VLAN tags and a pair of MPLS labels is a nuisance.

The approach that seemed most sensible was to build a recursive decap function which would take in a (partial) frame plus a "hint" as to what type of header to expect. The function would then check for and record any matching criteria present (i.e. MACs for Ethernet, VLAN ID for 802.1Q - more on this in the next section) before either returning or calling itself on the remainder of the packet with a "hint" derived from the current header.

Worked Example

Let's process the following frame as an example. Data in black are used by the algorithm while data in grey are not.

The initial call to the function passes the entire frame with an "Ethernet" hint. In the Ethernet header, the source and destination MAC addresses are read and stored. The EtherType field contains 0x8100, indicating an 802.1Q VLAN header is next. The function calls itself against the contents of the frame from byte 15 and on with a hint of "VLAN".

Now the function reads and stores the VLAN ID. Since this is the first VLAN we have seen it is stored as the C-VLAN for now. The EtherType is, again, 0x8100 so the function calls itself against bytes 5 and onward using a hint of "VLAN".

Again,  the function reads and stores the VLAN ID. Since this is not the first VLAN tag found, the previously known VLAN ID is moved into the S-VLAN field and the value from the frame is stored in the C-VLAN field. This time the EtherType is 0x8864, indicating a PPPoE session header follows. The function calls itself against bytes 5 and onwards using a hint of "PPPoE".

The function now reads and stores the PPPoE session ID (SID). The only valid thing to follow a PPPoE session header is a PPP header, so the function calls itself on bytes 7 and onward, using a hint of "PPP".

The function now simply checks that the protocol ID in the PPP header is 0xC223 for CHAP. If so, it calls itself one last time against bytes 3 and onward using a hint of CHAP.

Finally we are down to the payload. The CHAP message type is checked and:

• For challenges, the authentication ID, challenge length and challenge data are stored.
• For responses, the authentication ID, response and client name are stored.

Each instance of the function can then return to its parent, eventually resulting in a fully populated record of all the data relevant to authentication. The completed records can then be stored in a doubly linked list for later consumption.

Pairing Up

A CHAP response must be paired up with its respective CHAP challenge, otherwise the maths don't work. In real life there may be several authentications in progress at one time across multiple PPPoE sessions, possibly over multiple different VLANs. Often the CHAP authentication ID is only unique within a PPPoE session. Similarly, the PPPoE session ID only needs to be unique within a broadcast domain so these are often re-used across VLANs. Care must be taken to ensure that the challenge and response really do belong together.

In order to be considered a challenge / response pair, I decided the following criteria must match:

• Server and Client MACs
• S & C VLAN IDs (if present)
• PPPoE SID
• CHAP authentication ID

I considered including MPLS labels in this but I struggled to think of a realistic scenario in which two authentications would match the above criteria but use a different label.

Additionally, the thought occurred that even with the above details matching, there may be more than one challenge / response pair for the same PPPoE session so a response would have to be paired with the most recent challenge for which the criteria matched. In the program this is achieved by working backwards through the linked list, starting at the response, until a match is found. Data from matching challenge / response pairs are stored in another list for later consumption. If the search reaches the beginning without a matching challenge being found then the response cannot be used and is ignored.

Brute Force Password Guessing

For each challenge / response pair in the list, the next step is to cycle through a list of password guesses. Each candidate password is combined with the authentication ID and challenge data from the captured authentication and hashed. The resulting hash is compared to the one from the captured response and, for those that match, a correct guess is reported. If no password generated a matching hash then the word list does not contain the correct password and this is also reported back.

Downloading the Tool

The C source code may be downloaded from: https://github.com/theclam/dechap

Provided the OpenSSL dev libraries are installed it should be possible to simply extract the source code, cd into the directory then run "make".

In the future I may add the capability to pull the auths from L2TP or RADIUS interactions but for now only PPPoE is supported. It also assumes that Ethernet control words are not present in MPLS encapsulated traffic.

Using the Tool

The usage is pretty straightforward - there are only two parameters and both are mandatory. Specify your capture file (original pcap format) with the -c flag and your word list with the -w flag. Here's an example:

lab@lab:~/dechap$ ./dechap -w mywords.txt -c someauths.cap
Found password "tangerine" for user user1@testisp.com.
Unable to find a password for user user2@testisp.com.
Found password "password1" for user user3@testisp.com.
Found password "Africa" for user user4@testisp.com.
Found password "Frankenstein" for user user5@testisp.com.
lab@lab:~/dechap$

Considering that I've made no effort at all to make the code efficient, I've found the speed pretty good. On my '90s PC, a worst-case run (i.e. where no passwords are found) against 800 auths with 100k candidate passwords, a run still completes inside a minute. I don't think that's bad for parsing 15,000 packets and running 80 million concatenate - hash - compare sequences.

If you try this out, please leave a comment on this post with your experiences - good or bad.

Using Capture Filters to Match Higher Layer Protocols

In my previous post I went through some of the tricks that can be used to match MPLS and / or 802.1Q tagged traffic in packet filters. That's a great benefit when analysing traffic on carrier networks or large corporate networks but it only goes up to the transport layer (i.e. TCP and UDP port numbers).

Sometimes it's very desirable to filter on upper layer protocol information which has no corresponding parameters in the pcap-filter syntax. Take for example a situation where you are monitoring a busy BGP route reflector where you only want to see NOTIFICATION messages without all the KEEPALIVEs  and UPDATEs cluttering things up. It's possible to match these cases quite easily using a display filter, however your capture files could get quite large in relation to the amount of useful data. Once again it would be nice to be able to restrict at source, using a capture filter.

The following method can be used reliably for some protocols, somewhat reliably for a few and is completely inapplicable to others. In general if your protocol uses a fixed packet format or you want to match part of a fixed-format header then you're in luck.

Many protocols such as RADIUS, encode their parameters using attribute / value pairs (AVPs) or type / length / value (TLV) format, which can present parameters in an arbitrary order. If the parameter you want to match is in an AVP or TLV, your results are likely to be variable at best. Remember that capture filters work on fixed offsets and cannot cycle through parameters until the right one is found. If you're lucky the particular implementation you're looking at may put the AVPs / TLVs into the same order every time and your value may be early enough in the list not to get 'bumped' by other parameters inserted before it. In general, though, this technique is unlikely to work well.

Method

If at all possible, the best approach is to get a few sample packets of the data you want to capture. The captures should be taken from the same point in the network where you intend to run the real mirror to avoid any differences in encapsulation that would throw out the offsets. Generally it's possible to 'seed' such packets by, for example, manually clearing sessions.

In our example, we want to just see the BGP packets which contain a NOTIFICATION message. We start by obtaining a sample capture, obtained by shutting down a BGP session at one end while sniffing at the point where we intend to monitor. Below is the capture we get, with the interesting packet selected:

Here we can see there are two VLAN headers, beyond which we can see the IP details and the expanded decode of the BGP message. Logically, if we want to catch all the NOTIFICATION messages, we need to do the following:

• Parse and discard the VLAN tags so that IP can be decoded correctly
• Match only TCP traffic using either source or destination port 179
• Of this, match only those packets of type NOTIFICATION

Starting at the first line, we can begin to write our capture filter. Assuming that we don't care which VLAN IDs are being used, just that they are present, the following will match traffic with any two VLAN tags:

"vlan && vlan"

Note that this not only matches traffic which has two VLAN headers - it also adjusts the decoding offset. This is critical to the success of the filter as in a normal, untagged frame the IP header would start directly after the Ethernet header at offset 14 (decimal). With two VLAN tags, the IP header will actually be at offset 20 (decimal). By matching the VLAN tags in this way, the capture filter knows that the IP will start further into the frame. The same happens with the "mpls" and "pppoes" keywords, so if you have these headers make sure you match them.

So next we want to make sure that only BGP packets are matched - this is as simple as you would expect using "tcp port 179" - this will match either a source or a destination port of 179 so you don't have to worry about which end initiated the BGP session. Let's add it to the expression:

"vlan && vlan && tcp port 179"

Now this rule will match any double-tagged BGP traffic. The tricky part is that there are no capture filter keywords for matching BGP packet types and we want to do precisely that. The only option remaining for us is to match bytes at a given offset. Eek!

It takes a little getting used to but for fixed format headers it can be very reliable. I find the easiest way to do this is to:

• Select the field you want to match in Wireshark
• Find the offset in the packet where that value is stored
• Set a filter to match the required value at the required offset

So in our example, I have selected the BGP message type. This is at offset 005C hex / 92 decimal and a type of NOTIFICATION is encoded as a byte of value 3. A simple filter to match this would be "ether[92] == 3". Matching this on its own would get all the BGP NOTIFICATIONs, but also a load of other junk so let's combine it with the rest of our filter:

"vlan && vlan && tcp port 179 && ether[92] == 3"

OK, we can be pretty sure now that this will only match genuine NOTIFICATIONs. The BGP header ip to and including the type field is fixed length, so it is not going to move, and the value of 3 always means NOTIFICATION. Now let's test it out with a real capture on the same conversation as above:

root@sniff:~# tshark -i eth1 "vlan && vlan && tcp port 179 && ether[92] == 3"

Running as user "root" and group "root". This could be dangerous.
Capturing on eth1
  0.000000      1.1.1.2 -> 1.1.1.1      BGP NOTIFICATION Message
^C1 packet captured
root@sniff:~#

Perfect. Exactly what we wanted to capture!

Note: It's easy to see the offset from the start of the frame by just looking at the packet capture, but where possible you should consider using offsets from IP or TCP. That way, if you want to re-use your filter with more or less encap, you can just add or remove VLANs, MPLS, etc, without having to re-calculate the offsets.

You will have to use your imagination and ingenuity to work out whether this technique can be used to match your interesting traffic reliably. There are many aspects I have not covered which may prove essential, depending on what you are trying to do, for example:

• It is possible to match multiple byte fields using [offset:size] notation in place of the simple [offset] used in this example
• It is possible to bitmask values using the normal bitwise operators, so for example to check if the least significant bit of byte 80 is set, the expression "ether[80] & 1 == 1" can be used
• Offsets within a protocol can be used, i.e. ip[12]. Offsets like this start from the beginning of the layer being referenced.
• It is possible to put together some very complex filter statements using AND (&&), OR (||) and NOT (!) operators in conjunction with parentheses.

See the pcap-filter manpage for further details. With experimentation you can almost certainly filter out most of the junk even if it is not possible to cut it out altogether.

Final Tip

While you are practising with these filters you will probably find that you make mistakes with offsets and generally defining the filter correctly. One good way to learn and also to prove your filters work before deploying them is to take a live capture at the point where you plan to sniff, then, on a non-production box, use tcpreplay to pass the traffic while capturing with your filter applied.

References

RFC 4271 - A Border Gateway Protocol 4 (BGP-4) -  http://tools.ietf.org/html/rfc4271
pcap-filter manpage - http://manpages.ubuntu.com/manpages/lucid/man7/pcap-filter.7.html