As a rule, I log every CLI session I ever run. It has saved my neck on countless occasions and just this morning saved me a couple of hours of effort tallying serial numbers. It's surprising how often I use these logs to find some forgotten bit of info or to prove that "it really was all working when I left it"!
Every saved session I create in PuTTY is configured to log into a common directory using a file name in the format year-month-day-time-hostname.log, which gives me a nice, searchable, date-sorted record of what I typed and saw for every session I've ever opened.
The best way to do this is by just configuring Session -> Logging to "All session output" and the file name to something appropriate (the string I use is "&Y-&M-&D-&T-&H.log"). If you like, while you're there, go into Window and set the scrollback buffer to 2 million lines. Finally, return to the Session screen and save as "Default Settings".
From then on all your sessions you create, including ones where you just enter the IP and connect, will be logged. Note that any pre-existing saved sessions will need to be edited if they were not originally set to log.
Do it now, you know it makes sense!
Friday, 31 August 2012
Monday, 23 January 2012
Using Capture Filters with Encapsulated Packets
One of the most annoying things I found when I started working on carrier networks was that while Wireshark's display filters worked perfectly, the capture filters frequently did not. I would regularly set up a capture filter only to find that no packets at all were saved - that's a real pain if you want to pull a few easily described packets out of a 50 Mbps stream across a period of 20 minutes.
After a while I realised that my problem was related to encapsulation. Unlike the hierarchical and detailed display filters, capture filters have to be really fast - that basically means using bit masks and comparing values at fixed offsets. With plain old untagged Ethernet frames the filters work fine, however as soon as you add 802.1Q tags, PPP or MPLS suddenly all the offsets are no longer valid and anything you match will be purely coincidental.
Luckily there are filter keywords to handle that situation. All of the following adjust the offsets for you each time they are used:
vlan [x] - matches a single VLAN tag, the ID of which may optionally be specified by the user
pppoes - matches a PPPoE session header
mpls [x] - matches a single MPLS label, the number of which may optionally be specified by the user
These are very flexible - for example if you are capturing QinQ traffic, you could match all the SMTP packets using:
vlan && vlan && tcp port 25
If you know the VLAN IDs (or MPLS labels) in use, you can narrow the selection based on those. To show all the IGMP passing over a particular MPLS pseudowire with VLAN ID 200, you could use:
mpls 131066 && mpls 131068 && vlan 200 && pppoes && ip proto 2
For a long time I was using makeshift capture filters along the lines of "ether[39] = 2" to match pertinent bytes in the packet (see my next blog post for info on that) however you will probably agree this is much simpler. These filters are equally applicable to Wireshark, Tshark and tcpdump so they may be useful even when forced to capture using some really obscure UNIX box. For Tshark and tcpdump don't forget to put quotes around any expressions that use the ampersand (&).
After a while I realised that my problem was related to encapsulation. Unlike the hierarchical and detailed display filters, capture filters have to be really fast - that basically means using bit masks and comparing values at fixed offsets. With plain old untagged Ethernet frames the filters work fine, however as soon as you add 802.1Q tags, PPP or MPLS suddenly all the offsets are no longer valid and anything you match will be purely coincidental.
Luckily there are filter keywords to handle that situation. All of the following adjust the offsets for you each time they are used:
vlan [x] - matches a single VLAN tag, the ID of which may optionally be specified by the user
pppoes - matches a PPPoE session header
mpls [x] - matches a single MPLS label, the number of which may optionally be specified by the user
These are very flexible - for example if you are capturing QinQ traffic, you could match all the SMTP packets using:
vlan && vlan && tcp port 25
If you know the VLAN IDs (or MPLS labels) in use, you can narrow the selection based on those. To show all the IGMP passing over a particular MPLS pseudowire with VLAN ID 200, you could use:
mpls 131066 && mpls 131068 && vlan 200 && pppoes && ip proto 2
For a long time I was using makeshift capture filters along the lines of "ether[39] = 2" to match pertinent bytes in the packet (see my next blog post for info on that) however you will probably agree this is much simpler. These filters are equally applicable to Wireshark, Tshark and tcpdump so they may be useful even when forced to capture using some really obscure UNIX box. For Tshark and tcpdump don't forget to put quotes around any expressions that use the ampersand (&).
Friday, 13 January 2012
IGMP Testing, part 1
Maybe it's just my famous inability to find things that are right in front of me but I've needed some tools over the last week that would let me 'play' with IGMP and I've drawn (almost) a total blank.
Firstly, I wanted to generate a good old-fashioned flood of reports to test processing performance and rate limiting.
Plan A was to use the tester for this - despite not having a specific tool for flood testing it does let you create streams of, more or less arbitrary, hand crafted packets. That's the theory, anyway. After carefully putting together a stream profile that should have given me join after join for cycling group numbers I put it to the test - only to find that it had other ideas and was generating complete garbage. By garbage I mean not even the IP headers were correct - the protocol was coming out set to 0xfd (unknown) rather than 0x02 for IGMP and, strangely, the source and destination IPs were populated with the group ID and source that should have been in the report payload. Based on bitter past experiences I didn't waste my time trying to fix that.
OK, time for plan B - back to the packet crafting on a PC. I thought I'd be spoiled for choice but, for Linux anyway, the only option for generating arbitrary IGMP seemed to be nemesis. Nemesis seems to be exactly what I want but it is no longer maintained and won't compile on a modem distro - at least *I* couldn't get it to compile.
My favourite scapy knows what IGMP is from its protocol ID but doesn't have a stack for it, so there was no straightforward way to use that.
Dead end. I couldn't find anything to build me one packet let alone throw 1000 out per second.
Then it occurred that, actually, in normal use the tester can generate valid joins at a civil pace... So I mirrored the tester port and sniffed a genuine join off the wire, whittled the capture file down to the single frame I wanted and fed it to tcpreplay. Yay.
One small problem - it could only manage 100pps and I needed 1000. I noticed it was generating a message every time it sent a packet saying it had re-opened the file, which gave me a hunch that the file operations and CLI might be a bottleneck. I solved that problem the same way as the first - by sniffing the 100pps output for a while and then replaying *that* at full tilt. 960pps... Not quite 1000pps but close enough!
At the last moment it occurred to me that it would be a more convincing test if I cycled the group IDs rather than always reporting on one group. I went back to scapy and, with Wireshark in the other hand, started to play. I thought if I just loaded in the original join packet I could use a loop to tweak a byte or two for the group ID and dump it out to a file which I could then replay.
When I did that I noticed that my router still only showed one group as joined. Rubbish. I had obviously missed something. Looking at the generated file in Wireshark I could see that its checksum was incorrect.
Scapy could re-calculate the IP checksum for me but it didn't understand IGMP so that was going to be a programming exercise. The checksum is only 2 bytes in the payload so it wasn't too hard to adjust. I won't bore you with the maths, check out RFC 3376 if you're curious.
Finally, with that done, I had a pcap file full of valid joins over 100 groups and the ability to fire them out at (nearly) 1000pps. I can't help thinking it should have been easier, though!
Source code to follow - it's very scruffy and fairly fragile but might be useful to someone else... You never know!
Firstly, I wanted to generate a good old-fashioned flood of reports to test processing performance and rate limiting.
Plan A was to use the tester for this - despite not having a specific tool for flood testing it does let you create streams of, more or less arbitrary, hand crafted packets. That's the theory, anyway. After carefully putting together a stream profile that should have given me join after join for cycling group numbers I put it to the test - only to find that it had other ideas and was generating complete garbage. By garbage I mean not even the IP headers were correct - the protocol was coming out set to 0xfd (unknown) rather than 0x02 for IGMP and, strangely, the source and destination IPs were populated with the group ID and source that should have been in the report payload. Based on bitter past experiences I didn't waste my time trying to fix that.
OK, time for plan B - back to the packet crafting on a PC. I thought I'd be spoiled for choice but, for Linux anyway, the only option for generating arbitrary IGMP seemed to be nemesis. Nemesis seems to be exactly what I want but it is no longer maintained and won't compile on a modem distro - at least *I* couldn't get it to compile.
My favourite scapy knows what IGMP is from its protocol ID but doesn't have a stack for it, so there was no straightforward way to use that.
Dead end. I couldn't find anything to build me one packet let alone throw 1000 out per second.
Then it occurred that, actually, in normal use the tester can generate valid joins at a civil pace... So I mirrored the tester port and sniffed a genuine join off the wire, whittled the capture file down to the single frame I wanted and fed it to tcpreplay. Yay.
One small problem - it could only manage 100pps and I needed 1000. I noticed it was generating a message every time it sent a packet saying it had re-opened the file, which gave me a hunch that the file operations and CLI might be a bottleneck. I solved that problem the same way as the first - by sniffing the 100pps output for a while and then replaying *that* at full tilt. 960pps... Not quite 1000pps but close enough!
At the last moment it occurred to me that it would be a more convincing test if I cycled the group IDs rather than always reporting on one group. I went back to scapy and, with Wireshark in the other hand, started to play. I thought if I just loaded in the original join packet I could use a loop to tweak a byte or two for the group ID and dump it out to a file which I could then replay.
When I did that I noticed that my router still only showed one group as joined. Rubbish. I had obviously missed something. Looking at the generated file in Wireshark I could see that its checksum was incorrect.
Scapy could re-calculate the IP checksum for me but it didn't understand IGMP so that was going to be a programming exercise. The checksum is only 2 bytes in the payload so it wasn't too hard to adjust. I won't bore you with the maths, check out RFC 3376 if you're curious.
Finally, with that done, I had a pcap file full of valid joins over 100 groups and the ability to fire them out at (nearly) 1000pps. I can't help thinking it should have been easier, though!
Source code to follow - it's very scruffy and fairly fragile but might be useful to someone else... You never know!
Friday, 23 September 2011
L2TP Quirk
I've been looking forward to arriving at one of the items on my test agenda for a couple of weeks now. The customer would like to see some evidence that enabling LAC functionality in one VRF doesn't inadvertently open up L2TP connectivity in other VRFs. Pretty unlikely, everyone agrees. Two immediate thoughts on this:
1 - I would say it's impossible to *prove* that there are *no* side effects, but we should rule out any obvious clangers such as L2TP connectivity appearing in VRFs where you don't want it.
2 - How could I even prove there's no LAC running? L2TP is UDP based and UDP protocols are a pain like this. If you run a standard, dumb, UDP port scan it will often miss that SNMP running on a device because most security policy templates disable ICMP unreachables, so seeing "no response" to a sent UDP datagram can either mean the port is open or it is closed but no ICMP port unreachable is generated.
You will only prove a UDP port is open if you can solicit a response from the protocol listening behind, which generally means you have to send it something legal in that protocol.
But what could I send, unsolicited, to a LAC and expect it to respond? We all know that LACs connect out to LNS nodes when they have an incoming call to terminate, right?
Well, reading through RFC 2661 (what can I say, I'm a real party boy!) I noticed that section 5.1 about connection establishment shows that either side, LAC or LNS, can initiate the connection by sending an SCCRQ to the other. I suppose in the days of ISDN and modem racks it made sense for outgoing calls but for someone who has only ever used it in the context of DSL that's easily missed.
An SCCRQ is easily built or replayed, and I found that, sure enough, firing one at my LAC solicited a response. OK, the response was a StopCCN (L2TP for "go away") but enough to confirm there is a LAC or an LNS listening.
As expected, trying it against a different VRF where L2TP was not configured didn't solicit any response at all.
I was hoping it would have taken a bit more hacking than that. Maybe I'll try a few other things out, too, but I think this pretty much covers it.
I'd put the code for generating the SCCRQ on here, but it's part of a python script I'm working on to complete the whole handshake and allow sessions to come up. It'll be a while before I get that functionality working but I'll upload the code when it does.
1 - I would say it's impossible to *prove* that there are *no* side effects, but we should rule out any obvious clangers such as L2TP connectivity appearing in VRFs where you don't want it.
2 - How could I even prove there's no LAC running? L2TP is UDP based and UDP protocols are a pain like this. If you run a standard, dumb, UDP port scan it will often miss that SNMP running on a device because most security policy templates disable ICMP unreachables, so seeing "no response" to a sent UDP datagram can either mean the port is open or it is closed but no ICMP port unreachable is generated.
You will only prove a UDP port is open if you can solicit a response from the protocol listening behind, which generally means you have to send it something legal in that protocol.
But what could I send, unsolicited, to a LAC and expect it to respond? We all know that LACs connect out to LNS nodes when they have an incoming call to terminate, right?
Well, reading through RFC 2661 (what can I say, I'm a real party boy!) I noticed that section 5.1 about connection establishment shows that either side, LAC or LNS, can initiate the connection by sending an SCCRQ to the other. I suppose in the days of ISDN and modem racks it made sense for outgoing calls but for someone who has only ever used it in the context of DSL that's easily missed.
An SCCRQ is easily built or replayed, and I found that, sure enough, firing one at my LAC solicited a response. OK, the response was a StopCCN (L2TP for "go away") but enough to confirm there is a LAC or an LNS listening.
As expected, trying it against a different VRF where L2TP was not configured didn't solicit any response at all.
I was hoping it would have taken a bit more hacking than that. Maybe I'll try a few other things out, too, but I think this pretty much covers it.
I'd put the code for generating the SCCRQ on here, but it's part of a python script I'm working on to complete the whole handshake and allow sessions to come up. It'll be a while before I get that functionality working but I'll upload the code when it does.
Tuesday, 20 September 2011
Topics for Future Posts
I have a few things I'd like to post about but not much free time so here's a list of topics I'd like to post about. I can bump things up the list if anyone has an interest in something specific, but I can't promise to do anything quickly.
Here are the "interesting" things I've been doing lately:
I will say, though - I am not a programmer so my code will be scruffy and certainly not written with security in mind. I'm definitely not saying my methods are the best way to achieve anything, but they might get you out of a hole when time is short. In short, anything I suggest on here is for lab use only, and should not be considered suitable for live environments!
Here are the "interesting" things I've been doing lately:
- Using scapy to emulate a broken LNS
- Using scapy to emulate a PPPoE subscriber (to prove that IP packets sent before IPCP completes are discarded)
- Using one FreeRADIUS server to authenticate for both the LAC and LNS
- Using FreeRADIUS realms and DEFAULTs to minimise config for a wholesale ISP model
- Creating a bunch of scripts for automating RADIUS CoA randomly throughout a given user base
- Using expect scripts to take router health snapshots & statistics throughout a multi-day stability test
- Configuring a very quick-and-dirty LNS using Cisco hardware
- Configuring Ubuntu as a PPPoE client
- Using scripts to keep a pair of redundant RADIUS servers in sync using SCP
- Using byte references to build Wireshark capture filters for MPLS / PPP encapsulated or 802.1Q tagged frames
I will say, though - I am not a programmer so my code will be scruffy and certainly not written with security in mind. I'm definitely not saying my methods are the best way to achieve anything, but they might get you out of a hole when time is short. In short, anything I suggest on here is for lab use only, and should not be considered suitable for live environments!
What's this all about?
I work for a large networking equipment vendor and at the moment I'm working on a project where I'm responsible for getting a lot of testing completed on broadband technologies. We must be testing some pretty good stuff because for a lot of the cases, the tools required either:
a) don't exist
b) don't work
c) pretend to work but are really doing some other, rubbish, thing
d) don't scale
I'm finally getting to the end of some of the toughest challenges in my working life to date and I thought it would be worth starting a blog to document and share some of the tricks and tools I've used. Most of these are just quick tricks I've used in well known products like FreeRADIUS or Wireshark but many have been absolutely instrumental.
I'm not an expert on most of this stuff, so a lot of what I think is great will be old hat to many. I had to search a while to find most of it, though, so hopefully posting on here will save someone else some time and effort.
a) don't exist
b) don't work
c) pretend to work but are really doing some other, rubbish, thing
d) don't scale
I'm finally getting to the end of some of the toughest challenges in my working life to date and I thought it would be worth starting a blog to document and share some of the tricks and tools I've used. Most of these are just quick tricks I've used in well known products like FreeRADIUS or Wireshark but many have been absolutely instrumental.
I'm not an expert on most of this stuff, so a lot of what I think is great will be old hat to many. I had to search a while to find most of it, though, so hopefully posting on here will save someone else some time and effort.
Subscribe to:
Posts (Atom)