Removing VLAN/MPLS/PPPoE/GRE/GTP/VXLAN Encapsulation Headers from pcap Files

Many years ago, when I worked in a school, I used to port mirror our proxy server to an old PC running driftnet and leave the screen where the kids could see it as a warning that staff could "see what you're doing on the Internet". I haven't played with driftnet since but certainly at the time it could only handle native frames (no VLAN tags, certainly no MPLS or PPPoE). I vaguely remember some other tools being similar, unfortunately I can't remember which ones.

Looking at the analytics for this blog, I can see I'm not the only one who's had the problem. It's certainly not the number one issue that people are searching for when they get here but there have been a few and the thought occurred that the packet processing engine I wrote for dechap would be really good for this task - it already stripped back VLANs and MPLS, plus it knows how to detect PPPoE and L2TP.

After a couple of hours it was working to the point of being able to strip VLANs and MPLS off, with a little more effort PPPoE also gave way. GRE came quite easily, too, as it has simple headers and uses the same etypes as Ethernet.

Anyway, here is "stripe" (from STRIP Encapsulation), a command line tool which takes a pcap file as input, re-assembles IP fragments and strips off all the encap it can (currently VLAN tags, MPLS shim headers, PPPoE, L2TP, GRE GTP and VXLAN) then outputs another pcap containing just payload over Ethernet.

**UPDATE** - Version 0.3b now adds support for VXLAN.

Download

Stripe is available from my github: https://github.com/theclam/stripe

Usage

The command line is pretty straightforward, as shown in the online help:

Harrys-MacBook-Air:stripe foeh$ ./stripe
stripe: a utility to remove VLAN tags, MPLS shims, PPPoE, L2TP headers,
etc. from the frames in a PCAP file and return untagged IP over Ethernet.
Version v0.1 alpha, November 2014

Usage:
./stripe -r inputcapfile -w outputcapfile

Where inputcapfile is a tcpdump-style .cap file containing encapsulated IP 
outputcapfile is the file where the decapsulated IP will be saved

Harrys-MacBook-Air:stripe foeh$ 

Simply specify the files you want to read encapsulated packets from (-r) and write the cleaned up packets to (-w). Stripe will remove as many layers of encap as it can until you are left with straight payload over Ethernet.

How it Works

The majority of stripe's work is done by the "decap" function. This function takes in a block of memory, a length parameter, a data type hint and a frame template. The process runs as follows:

1. If the type is Ethernet, populate the source / destination MACs of the frame template
2. If the type has an Ethertype or protocol type field, use this to populate the ethertype of the frame template
3. If the next protocol is possibly or definitely payload, set the payload pointer of the frame template to the address of the next protocol and return
4. If the next protocol is possibly or definitely encapsulation, call decap against the remainder of the packet

So essentially it eats up encap, recording MACs and protocol types as it goes, until there is no more encap left. By the end there is a fully populated frame template with source and destination MAC (the innermost copy if there are multiple as in the case of MPLS pseudowires), the etherype of the payload and the payload itself. Piecing these together gives a minimally encapsulated frame, i.e. one with just an Ethernet header and payload.

Here is a worked example for a frame with VLAN, MPLS, GRE over IP and an IP payload:

Step 1 - The "decap" function is called on the entire frame. Since the first header is Ethernet, the frame template gets populated with the source / destination MACs and the etype from the Ethernet header. The frame template's length field gets populated with the size of the frame minus the Ethernet header and the payload pointer is adjusted to point at the next header. The decap function then calls itself on the remainder of the frame, hinting that the type is VLAN tag based on the current header's etype.

Step 2 - The decap function now considers the partial frame starting at the VLAN tag. Since the VLAN tag has an etype associated, the frame template's etype is overwritten with the one from the VLAN header. The length is overwritten with the length of the payload after the VLAN header and the pointer adjusted to point at the next header. The decap function then calls itself again with a hint of MPLS, based on the etype in the VLAN header.

Step 3 - The decap function now considers the partial frame starting at the MPLS label. Since the MPLS label is bottom of stack, we know there are no more MPLS labels left . Unfortunately there is no protocol type in an MPLS header (these are signaled on the control plane) so we have to take a peek at the byte immediately following the label. If we find a "4" or a "6" in the high order nibble then we have to guess that the next protocol is IPv4 or IPv6, respectively. If the following four bytes are all zeroes then we assume Ethernet over MPLS with control word, otherwise we assume Ethernet over MPLS without control word. In this case we find a 4 in the low nibble, so call decap with an "IP" hint.

Step 4 - The IP header tells us that GRE is the next protocol so for now nothing changes in the frame template (the remainder could be decodable or not). We just call decap again on the GRE part...

Step 5 - The GRE header is decoded and the etype is copied into the frame header. The length of the remaining payload is updated in the frame template and the pointer is adjusted. Decap is called on the next header, which is IP. When the decap function inspects the IP payload it can go no further and just returns the frame template.

In essence, the process has started with a deeply encapsulated frame and ended with IP over Ethernet. The source and destination MACs are taken from the innermost ones found (which in this case is the outermost Ethernet header) but with the etype changed to match the payload, which is the first non-encapsulating payload found in the frame, in this case the second IP.

References

https://tools.ietf.org/html/rfc2784
https://tools.ietf.org/html/rfc1701
http://www.ieee802.org/1/pages/802.1Q.html
http://www.3gpp.org/DynaReport/29060.htm

RADIUS and L2TP Support Added to "dechap"

This is just a very short message to say that I have enhanced the "dechap" tool mentioned in my previous post. In addition to the original PPPoE support it can now extract and attack CHAP authentications sniffed from RADIUS and L2TPv2 protocols.

The syntax remains exactly the same and it should "just work". The code is available to download at https://www.github.com/theclam/dechap. Please post a comment if you have any feedback or suggestions.

tshark one-liners

Since most of the hits on this blog seem to come from tshark filter related searches, and since I spend a good part of my day either running or analysing packet captures, I thought it might be useful to create a series of "tshark one-liners" in homage to the brilliant "sed one-liners" collection compiled by Eric Pement.

These are capture filters, not display filters, and are equally applicable to Wireshark, tshark and tcpdump, since they all use the same pcap filter syntax. In wireshark the capture filter options are now hidden away and you have to double click on the interface under capture options to set or adjust the filter string.

The filters are broadly grouped by purpose and I will try to add more as I think of them. Please comment if there is something you think I have missed or would like added.

Note: if you want to strip off VLAN, MPLS, PPPoE or GRE headers from an existing pcap file, please see this post: Removing VLAN/MPLS/PPPoE/GRE Encapsulation

Ethernet

Match 802.1D spanning tree:
"ether dst 01:00:c2:00:00:00" (manpages say "ether proto stp" but I've had trouble with that)

Match Cisco PVST+:
"ether dst 01:00:0c:cc:cc:cd"

Match Cisco CDP / VTP / DTP / PAgP / UDLD:
"ether dst 01:00:0c:cc:cc:cc"

Match LLDP:
"ether proto 0x88cc"


Match LACP (slow protocols):
"ether dst 01:80:c2:00:00:02"

General IP
Match host A (10.0.0.1) communicating with host B (192.168.0.1):
"host 10.0.0.1 && host 192.168.0.1"

Match host A (10.0.0.1) communicating with anything on network B (192.168.0.0/24):
"host 10.0.0.1 && net 192.168.0.0/24"
or, if you don't like CIDR notation:
"host 10.0.0.1 && net 192.168.0.0 mask 255.255.255.0"

Match ARP:
"ether proto 0x0806"

Match DHCP:"udp port 67 || udp port 68"

VLANs

Match any traffic with at least one VLAN tag:
"vlan"

Match traffic with exactly one VLAN tag:
"vlan && not vlan"

Match traffic with an SVLAN of 100 and any CVLAN:
"vlan 100 && vlan"

Match traffic where the first VLAN tag has an 802.1p marking of:
0: "vlan && ether[14] & 224 == 0"
1: "vlan && ether[14] & 224 == 32"
2: "vlan && ether[14] & 224 == 64"
3: "vlan && ether[14] & 224 == 96"
4: "vlan && ether[14] & 224 == 128"
5: "vlan && ether[14] & 224 == 160"
6: "vlan && ether[14] & 224 == 192"
7: "vlan && ether[14] & 224 == 224"

Note: to match the second VLAN tag use "vlan && vlan && ether[18] & 224" on the left hand side of the equality.

MPLS

Match traffic with at least one MPLS label:
"mpls"

Match traffic with exactly one MPLS label (match S bit of first label):
"mpls && ether[16] & 1 == 1"

Match traffic with a first or single label of 12345:
"mpls 12345"

Match traffic with an inner (e.g. service) label of 67890:
"mpls && mpls 67890"

Match traffic with exactly three MPLS labels (e.g. traffic on facility bypass FRR):
"mpls && mpls && mpls && ether[24] & 1 == 1"

Match 6PE traffic:
With transport label: "mpls && mpls 2"
Without transport label (after PHP): "mpls 2"

Match traffic with an EXP marking (on the first label) of:
0: "mpls && ether[16] & 14 == 0"
1: "mpls && ether[16] & 14 == 2"
2: "mpls && ether[16] & 14 == 4"
3: "mpls && ether[16] & 14 == 6"
4: "mpls && ether[16] & 14 == 8"
5: "mpls && ether[16] & 14 == 10"
6: "mpls && ether[16] & 14 == 12"
7: "mpls && ether[16] & 14 == 14"

Note: to match the EXP marking of the second label, use "mpls && mpls && ether[20] & 14" on the left hand side of the equality.

Multicast

Match any Ethernet multicast:
"ether multicast"

Match IP multicast traffic:
"ip multicast"

Match IGMP traffic:
"ip proto 2" (the manpages say "ip proto igmp" but I've had trouble with that)

Match PIM traffic:
"ip proto 0x67" (the manpages say "ip proto pim" but I've had trouble with that)

OSPFv2

Match all OSPF:
"ip proto 89"

Match specific OSPF packet types:
Hello: "ip proto 89 && ip[20:2] == 0x0201"
DBD: "ip proto 89 && ip[20:2] == 0x0202"
LSR: "ip proto 89 && ip[20:2] == 0x0203"
LSU: "ip proto 89 && ip[20:2] == 0x0204"
LSA: "ip proto 89 && ip[20:2] == 0x0205"

IS-IS

Match all IS-IS traffic:
"isis"

Match specific IS-IS PDU types:
"l1", "l2", "iih", "lsp", "snp", "csnp" or "psnp"

BGP

Note: These rules do not handle multi-segment messages very well but they are good enough for most purposes.

Match only BGP OPEN messages:
"tcp port 179 && tcp[50] == 1"

Match only BGP UPDATE messages:
"tcp port 179 && tcp[50] & 5 != 0"

Match only BGP NOTIFICATION messages:
"tcp port 179 && tcp[50] == 3"

Match only BGP KEEPALIVE messages:
"tcp port 179 && tcp[50] == 4"

 L2TP

Match only L2TP control messages:
"udp port 1701 && udp[8:2] & 0x80ff == 0x8002"

Match L2TP control messages for tunnel ID 1234:
"udp port 1701 && udp[8:2] & 0x80ff == 0x8002 && udp[12:2] == 1234"

Match L2TP data messages for tunnel ID 1234:
"udp port 1701 && udp[8:2] & 0x80ff == 0x0002 && udp[10:2] == 1234"

Match L2TP control messages for session ID 5678:
"udp port 1701 && udp[8:2] & 0x80ff == 0x8002 && udp[14:2] == 5678"

Match L2TP data messages for session ID 5678:
"udp port 1701 && udp[8:2] & 0x80ff == 0x0002 && udp[12:2] == 5678"

PPPoE

Note: Offsets will need to be manually increased by 4 bytes  for each VLAN tag or MPLS label present.

Match PPPoE discovery phase (PADI / PADO / PADR / PADS / PADT):
"pppoed"

Match PPPoE session phase (i.e. PPP traffic):
"pppoes"

Match PPPoE LCP messages:
"pppoes && ether[20:2] == 0xc021"

Match PPPoE CHAP authentication messages:
"pppoes && ether[20:2] == 0xc223"

Simulating a broken LNS

A common requirement when testing a LAC is to confirm its reaction when various failure codes are returned by the LNS. In theory you would expect the LAC to react to an LNS failure in the same way (i.e. try another) irrespective of the error type or code returned, but as we all know theory and practice don't always align and that is why we test.

I recently had to prove exactly this area of functionality and found that, while it is relatively easy to put an LNS together which will terminate sessions, it's actually quite hard to get a real LNS to return error messages. Would you believe that they appear to be designed not to fail?

So the aim was:

• To have an 'LNS' which could be configured to reject incoming start control connection requests (SCCRQs)
• To be able to configure the result code, error code and, to make the packet captures easier to read and more authentic, the error message contained within the StopCCN message
• Ideally, to be able to service requests arriving on multiple IP addresses

As usual, the answer to this problem turned out to be scapy.

Important:

The script shown below does exactly what I needed but doesn't exactly work how you might expect. In order to reduce reconfiguration between test cases I have made it respond to queries arriving on any IP address - it does this by inspecting the incoming SCCRQ's source and destination MAC and IP addresses, then flipping them around on the response. That means that it does not attempt to bind to port 1701 on the host, therefore if the LAC sends an SCCRQ to the host's real IP it will get an ICMP unreachable and a StopCCN back. This is almost certainly not what you want.

The intended use case for this script is to have the LAC attempt to connect to an LNS which is "behind" the host running scapy, i.e. the last hop router should have a static route directing traffic for the LNS via the scapy host, in effect creating the following topology:

Alternatively, you could use a static ARP entry on the gateway router to direct traffic for an address on the attached LAN to the scapy host.

Usage

Usage is simple - firstly run scapy, then call 'execfile("BrokenLNS.py")' to load the script. You must create an instance of "LNS" and then, if the defaults to not suit, set the following member values:
 interface (default "eth1")

• resultcode (default 0)
• errorcode (default 4)
• errormessage (default "Internal error")

The script will sit there and close as many sessions as you care to offer it. Press control-C to stop.

Example

root@scapyhost:~/Projects/BrokenLNS# scapy
WARNING: No route found for IPv6 destination :: (no default route?)
Welcome to Scapy (2.0.1)
>>> execfile('BrokenLNS.py')
>>> lns = LNS()
>>> lns.resultcode = 1
>>> lns.errorcode = 6
>>> lns.errormessage = "Oh, no!"
>>> lns.run()
Received L2TP packet from 1.2.3.4
Got an SCCRQ
Sending spoofed StopCCN from 172.16.0.20  to 1.2.3.4.
.
Sent 1 packets.
Received L2TP packet from 1.2.3.4
^C>>>

Code

import os
# Flags
MANDATORY = 32768
HIDDEN = 16384
CONTROL = 32768
L = 16384
S = 2048
# Types
CONTROLMESSAGE = 0
ERRORMESSAGE = 1
PROTOCOLVERSION = 2
HOSTNAME = 7
RECVWIN = 10
FRAMING = 3
BEARER = 4
FIRMWARE = 6
TUNNELID = 9
CHALLENGE = 11
# Control Message Types
SCCRQ = '\x00\x01'
SCCRP = '\x00\x02'
StopCCN = '\x00\x04'
def word(value):
# Generates a two byte representation of the provided number
  return(chr((value/256)%256)+chr(value%256))
def AVP(bitmask, vendor, attribute_type, data):
# Generates an L2TP AVP using the given attribute number and payload
  length = len(data) + 6
  return(word(bitmask + (length % 1024)) + word(0) + word(attribute_type) + data)
def genL2TP(flags, tunid, sessid, ns, nr, payload):
# Generates an L2TP payload with the given parameters and AVP payload
  length = len(payload) + 12
  return(word(flags | 2) + word(length) + tunid + sessid + word(ns) + word(nr) + payload)
def getAVP(avp, payload):
  loc = 0
  while(loc < len(payload)):
    avp_type = payload[loc+2:loc+6]
    avp_len = ((ord(payload[loc:loc+1]) & 3) * 256) + ord(payload[loc+1:loc+2])
    # Uncomment the following line if you want to see info on every AVP checked
#    print "Got AVP " + str(ord(avp_type[0:1])).zfill(2) + str(ord(avp_type[1:2])).zfill(2)  + str(ord(avp_type[2:3])).zfill(2) + str(ord(avp_type[3:4])).zfill(2) + " of length " + str(avp_len) + " value " + payload[loc+6:loc+avp_len]
    if avp_type == avp:
      return(payload[loc+6:loc+avp_len])
    loc = loc + avp_len
class LNS(Automaton):
  interface = "eth1"
  resultcode = 0
  errorcode = 4
  errormessage = "Internal error"
# Define possible states
# Since this is so simple we only need one state :)
  @ATMT.state(initial=1)
  def WAIT(self):
    pass
# Define transitions
# Transitions from WAIT
  @ATMT.receive_condition(WAIT)
  def receive_sccrq(self,pkt):
    if (UDP in pkt) and pkt.dport==1701:
      print "Received L2TP packet from " + pkt[IP].src
      # scapy's built in L2TP handling doesn't deal well with control messages so
      # we just grab the raw data from beyond the UDP header
      payload = pkt[UDP].build_payload()
      # Check what type of L2TP message arrived by chopping off the header and passing
      # the rest to getAVP
      packet_type = getAVP(word(0) + word(CONTROLMESSAGE), payload[12:])
      if(packet_type == SCCRQ):
        # If we get an SCCRQ, generate a StopCCN in response.
        print "Got an SCCRQ"
        client_ip = pkt[IP].src
        server_ip = pkt[IP].dst
        client_mac = pkt[Ether].src
        server_mac = pkt[Ether].dst
        tun_id = getAVP(word(0) + word(TUNNELID), payload[12:])
        print "Sending spoofed StopCCN from " + server_ip + "  to " + client_ip + "."
        sendp(Ether(src=server_mac, dst=client_mac)/IP(src=server_ip, dst=client_ip)/UDP(sport=1701, dport=1701)/Raw(load=genL2TP(CONTROL | L | S, tun_id, word(0), 0, 1, AVP(MANDATORY, 0, CONTROLMESSAGE, StopCCN) + AVP(MANDATORY, 0, ERRORMESSAGE, word(self.resultcode) + word(self.errorcode) + self.errormessage) + AVP(MANDATORY, 0, TUNNELID, word(12345)))), iface=self.interface)
        raise self.WAIT()
      elif(packet_type == SCCRP):
        print "is an SCCRP"
      elif(packet_type == StopCCN):
        print "is a StopCCN"
      else:
        print "is a ZLB or non-control message"

L2TP Quirk

I've been looking forward to arriving at one of the items on my test agenda for a couple of weeks now. The customer would like to see some evidence that enabling LAC functionality in one VRF doesn't inadvertently open up L2TP connectivity in other VRFs. Pretty unlikely, everyone agrees. Two immediate thoughts on this:

1 - I would say it's impossible to *prove* that there are *no* side effects, but we should rule out any obvious clangers such as L2TP connectivity appearing in VRFs where you don't want it.
2 - How could I even prove there's no LAC running? L2TP is UDP based and UDP protocols are a pain like this. If you run a standard, dumb, UDP port scan it will often miss that SNMP running on a device because most security policy templates disable ICMP unreachables, so seeing "no response" to a sent UDP datagram can either mean the port is open or it is closed but no ICMP port unreachable is generated.

You will only prove a UDP port is open if you can solicit a response from the protocol listening behind, which generally means you have to send it something legal in that protocol.

But what could I send, unsolicited, to a LAC and expect it to respond? We all know that LACs connect out to LNS nodes when they have an incoming call to terminate, right?

Well, reading through RFC 2661 (what can I say, I'm a real party boy!) I noticed that section 5.1 about connection establishment shows that either side, LAC or LNS, can initiate the connection by sending an SCCRQ to the other. I suppose in the days of ISDN and modem racks it made sense for outgoing calls but for someone who has only ever used it in the context of DSL that's easily missed.

An SCCRQ is easily built or replayed, and I found that, sure enough, firing one at my LAC solicited a response. OK, the response was a StopCCN (L2TP for "go away") but enough to confirm there is a LAC or an LNS listening.

As expected, trying it against a different VRF where L2TP was not configured didn't solicit any response at all.

I was hoping it would have taken a bit more hacking than that. Maybe I'll try a few other things out, too, but I think this pretty much covers it.

I'd put the code for generating the SCCRQ on here, but it's part of a python script I'm working on to complete the whole handshake and allow sessions to come up. It'll be a while before I get that functionality working but I'll upload the code when it does.