A "real" full Internet table / route server in the lab

A common requirement when lab / stress testing carrier kit is to have at least one copy of a full Internet routing table. While many people will use a network tester to generate, say, 700,000 routes programmatically, this is not really representative. Real Internet routes are not all the same size. They are not contiguous or predictable. They have different AS_PATHs of varying lengths. There are aggregates and specific prefixes, often both covering the same IP space. These factors may make no difference to your system under test, or they might make a huge difference - it's not unusual for TCAM to be partitioned or optimised by prefix length so surely it's best to test with the real thing, right?

This post shows how to build a fully populated and very fast BGP route server based on Ubuntu Bionic Beaver 18.04 LTS.

There are three main elements to this:

• A daily dump of the RIPE RIB, which we will manipulate and shrink down to a single "view" ready for processing
• A copy of RIPE's bgpdump and a (slightly tweaked) instance of the bgp_simple perl script which can be used to replay the processed dump file back to a listening BGP instance
• An instance of BIRD, which is loaded up with routes by bgp_simple and can be used to re-advertise them (in a very fast and resilient fashion) to your systems under test.

This method overcomes a number of horrible issues - Trimming the RIB snapshot massively reduces the time to advertise the table, without compromising the number of prefixes. The bgp_simple instance is modified to reduce the amount of text output, which also increases speed. The BIRD instance sitting between bgp_simple and the system(s) under test increases robustness (BIRD will re-establish dropped BGP sessions whereas bgp_simple does not) and provides much faster update capability.

With these tweaks, a full table can be loaded into BIRD in around 2 minutes and, once fully loaded, BIRD can advertise the full table onward in a few seconds.

Topology

In this setup we will assume our route server will be 10.0.0.1 in AS 65001 and the system under test will be 10.0.0.2 in AS 65002.

Host Setup

This guide starts with a fresh Ubuntu 18.04 LTS instance. This can be a bare metal install, a VM or even a LXC container. You will need to set up this host's networking so that it can access the Internet, at least to begin with. Since this is a lab box, I'm being super sloppy and doing everything as root. You can mentally put "sudo" in front of everything if you'd rather.

First, update the packages and install a few necessary extras:

apt update && apt install bird git perl build-essential libz-dev libbz2-dev screen

Now we need to get some Perl libraries via CPAN. First open CPAN:

cpan

Accepting the auto config should normally do what you want. Within there, run the following:

install CPAN
reload CPAN
install Net::BGP

Exit out by pressing Ctrl-D

Now, edit /etc/bird/bird.conf as follows (replacing the existing "router id") config:

router id 10.0.0.1;
listen bgp address 10.0.0.1;
protocol bgp {
        local 10.0.0.1 as 65001;
        neighbor 10.0.0.2 as 65002;
        multihop;
        source address 10.0.0.1;
        next hop self;
        import all;
        export all;
}
protocol bgp {
        local 10.0.0.1 as 65001;
        neighbor 192.2.0.1 as 64999;
        multihop;
        source address 10.0.0.1;
        next hop self;
        passive;
        import all;
        export none;
}

Now, restart BIRD as follows:

service bird restart

At this point you should have BIRD running (you can check its status by running "birdc" and running some "show" commands - if your system under test is already configured you should see the peer come up). Next, we will set up bgp_simple to load the table into BIRD.

Before we can do that we need to download a few things:

git clone https://github.com/xdel/bgpsimple.git
wget ris.ripe.net/source/bgpdump/libbgpdump-1.6.0.tgz
wget data.ris.ripe.net/rrc00/latest-bview.gz

The daily dump file is pretty big and could take several minutes. Either wait for it or open another tab to build libbgpdump:

tar -xzf libbgpdump-1.6.0.tgz
cd libbgpdump-1.6.0/
./configure && make && make install
cd ..
rm -r libbgpdump-*

Now, the dump file contains many, many views of the full table. We just need one, so let's get a list of neighbours:

zcat latest-bview.gz | bgpdump -m - | head -50000 | cut -d '|' -f 4 | grep -v ':' | sort | uniq

Pick one of the IPs listed at random and filter down for just that neighbour. You can repeat this process using different addresses on different route servers if you'd like to simulate getting a table from two providers. In this example, we'll use 185.210.224.254, but any should be just as good.

zcat latest-bview.gz | bgpdump -m - | grep '185.210.224.254' > bgpsimple/routes.txt

This takes a looooong time, perhaps 25 minutes on a rubbish PC. Grab a cuppa.

Everything up until here will still be there after a reboot, the parts that follow will need to be re-run each time the route server gets rebooted.

Almost there, we need to add the unroutable test address onto the loopback interface:
ip address add 192.2.0.1/32 dev lo

Finally, we will make some minor tweaks to bgp_simple:

cd bgpsimple

Edit bgp_simple.pl and comment out lines 640-649 - this prevents the script echoing all 700,000 routes to the console as they are advertised (complete with their AS_PATH, MED, etc).

Now run an instance of "screen" to keep bgp_simple running after your CLI session ends:

screen -an

And execute the script:

./bgp_simple.pl -myas 64999 -myip 192.2.0.1 -peerip 10.0.0.1 -peeras 65001 -p routes.txt -holdtime 300 -keepalive 30

The script should connect and, after a short pause, tell you that it is advertising its routes to the peer. After a couple of minutes you should see a message to say the advertisement is complete.

At this point you can break out of screen using Ctrl-a, followed by "d" (reconnect later using "screen -r").

Finally, run "birdc" and execute "show route count" to confirm how many routes you are seeing. Once BIRD is loaded up it can blow a table down to a peer in seconds and can be configured for as many peers as you like.

Happy labbing!

A Down in the Weeds look at Route Distinguishers

I was recently involved in a discussion on reddit about VRF route targets and route distinguishers and I noticed that there was a lot of misinformation flying around. That doesn't really surprise me as a lot of the folks on there are learning and I've heard some jarring misconceptions on the topic come from  senior guys who have worked with MPLS for years. Most of the route target stuff was straightened out quite quickly and I will not get into any of that here, however the route distinguisher debate went on longer and covered some areas that seemed to be new or controversial to a lot of people.

The crux of the issue is that a lot of people believe the route distinguisher to be only locally significant - apparently there are many resources on the Internet which say this. I'll grant you that many are ambiguous, for example the first hit on Google for "route distinguisher and route target" says that "The route distinguisher has only one purpose, to make IPv4 prefixes globally unique. It is used by the PE routers to identify which VPN a packet belongs to". The well-respected packet life blog says "As its name implies, a route distinguisher (RD) distinguishes one set of routes (one VRF) from another. It is a unique number prepended to each route within a VRF to identify it as belonging to that particular VRF or customer." To be fair it goes on to clarify that "An RD is carried along with a route via MP-BGP when exchanging VPN routes with other PE routers", which suggests at the global significance.

In this post I hope to prove to anyone who is interested that route distinguishers are, in fact, both locally and globally significant and to demonstrate why this is important to understand.

Local Significance

If you've got this far then I assume you will already be familiar with what route targets and route distinguishers do, if not then I suggest you read up and play in the lab a while before venturing on.

The reason for needing a route distinguisher locally within a device is to extend the normal IPv4 prefixes that are known within each VRF in order to make them unique. Any locally learned IPv4 prefixes (connected, static or learned via an IPv4 routing protocol) are extended with the route distinguisher assigned to the VRF, as shown here:

It is also true that different PEs may use different route distinguishers for the same VRF without breaking anything:

PE1#show run vrf A
Building configuration...

Current configuration : 316 bytes
ip vrf A
 rd 100:2439
 route-target export 100:100
 route-target import 100:100
!
!
interface FastEthernet1/0
 ip vrf forwarding A
 ip address 192.168.1.1 255.255.255.0
 speed auto
 duplex auto
!
router bgp 100
 !
 address-family ipv4 vrf A
  redistribute connected
  redistribute static
 exit-address-family
!
end

PE1#show ip route vrf A

Routing Table: A


Gateway of last resort is not set

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
B     192.168.24.0/24 [200/0] via 10.255.255.2, 00:04:03
PE1#



PE2#show run vrf A
Building configuration...

Current configuration : 317 bytes
ip vrf A
 rd 100:2458
 route-target export 100:100
 route-target import 100:100
!
!
interface FastEthernet1/0
 ip vrf forwarding A
 ip address 192.168.24.1 255.255.255.0
 speed auto
 duplex auto
!
router bgp 100
 !
 address-family ipv4 vrf A
  redistribute connected
  redistribute static
 exit-address-family
!
end

PE2#show ip route vrf A

Routing Table: A

Gateway of last resort is not set

B     192.168.1.0/24 [200/0] via 10.255.255.1, 00:03:44
      192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.24.0/24 is directly connected, FastEthernet1/0
L        192.168.24.1/32 is directly connected, FastEthernet1/0
PE2#


So it's easy to see how the idea got started that RDs are only locally significant:

Route distinguishers don't need to match between devices in the same VRF in order for routes to be shared between them.

Global Significance

The first clue at the global significance of the route distinguisher is that it is carried in the MP-BGP updates:

PE1#show bgp vpnv4 unicast all 192.168.24.0/24 
BGP routing table entry for 100:2439:192.168.24.0/24, version 8
Paths: (1 available, best #1, table A)
  Not advertised to any peer
  Refresh Epoch 2
  Local, imported path from 100:2458:192.168.24.0/24 (global)
    10.255.255.2 (metric 3) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/28
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2458:192.168.24.0/24, version 7
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 2
  Local
    10.255.255.2 (metric 3) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/28
      rx pathid: 0, tx pathid: 0x0
PE1#

Interestingly, we have 2 different prefixes here. One is the original (100:2458:192.168.24.0/24) which we learned over the network, while the other is the same IPv4 prefix but prepended with the RD of the VRF which imports it (100:2439:192.168.24.0/24). If we imported it into multiple VRFs then we would have an additional copy for each RD used by the respective VRFs.

If the RD were only locally significant then why would the protocol designers send it? You may be thinking "otherwise you couldn't overlap prefixes!", but surely route targets would be enough to achieve this? If you heard a prefix of 10.0.0.0/8 announced with a route target imported by VRF A then you would import it into VRF A and not VRF B, if you heard a different announcement for 10.0.0.0/8 with a route target imported by VRF B then you would import it into VRF B and not VRF A.

That could kind of work, in theory, but it would essentially break the whole BGP paradigm as you would have multiple copies of the same prefix in use concurrently for different purposes. BGP likes to determine the best path and only offers that into the FIB. With a unique RD against each of the two 10.0.0.0/8 routes, BGP is able to do its best path determination and pass the two, now different, routes into their respective VRFs.

So the route distinguishers overcome that problem, but is that the only reason why they are carried in MP-BGP? That would be a fairly weak argument for global significance, but the best path point here touches on a much stronger case.

The Route Reflector Problem

One key thing to bear in mind which is often forgotten in the grand scheme of things is that the PE is not the only place where BGP best path calculations happen. Any MPLS network of even moderate scale will be using BGP route reflectors to keep the number of BGP sessions under control, and the route reflectors themselves perform a best path determination on the routes they receive before sending them out to their route reflector clients.

This extends the previous case to all the route reflector's clients, so essentially the entire AS. Let's take an example where the admin has been sloppy and has failed to keep RDs globally unique:

Notice that VRF A uses import / export RT 100:100, VRF B uses import / export RT of 100:200. The network administrator has tried to assign unique route distinguishers per VRF per device, but has made an error and overlapped the route targets used on PE1's VRF A and PE3's VRF B.

The two VRFs are completely distinct from one another and they are not even present on the same PEs. We can see that VRF A is only learning VRF A's routes and VRF B is only learning VRF B's routes:

PE1#show ip vrf
  Name                             Default RD          Interfaces
  A                                100:2439            Fa1/0
PE1#show ip route vrf A
Routing Table: A

Gateway of last resort is not set

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, FastEthernet1/0
L        192.168.1.1/32 is directly connected, FastEthernet1/0
B     192.168.24.0/24 [200/0] via 10.255.255.2, 00:05:01
PE1#

PE2#show ip vrf
  Name                             Default RD          Interfaces
  A                                100:2458            Fa1/0
PE2#show ip route vrf A

Routing Table: A

Gateway of last resort is not set

B     192.168.1.0/24 [200/0] via 10.255.255.1, 00:05:15
      192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.24.0/24 is directly connected, FastEthernet1/0
L        192.168.24.1/32 is directly connected, FastEthernet1/0
PE2#

PE3#show ip vrf
  Name                             Default RD          Interfaces
  B                                100:2439            Fa1/0
PE3#show ip route vrf B

Routing Table: B

Gateway of last resort is not set

      192.168.3.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.3.0/24 is directly connected, FastEthernet1/0
L        192.168.3.1/32 is directly connected, FastEthernet1/0
B     192.168.19.0/24 [200/0] via 10.255.255.4, 00:01:54
PE3#

PE4#show ip vrf
  Name                             Default RD          Interfaces
  B                                100:2895            Fa1/0
PE4#show ip route vrf B

Routing Table: B

Gateway of last resort is not set

B     192.168.3.0/24 [200/0] via 10.255.255.3, 00:02:21
      192.168.19.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.19.0/24 is directly connected, FastEthernet1/0
L        192.168.19.1/32 is directly connected, FastEthernet1/0
PE4#

Now, let's introduce an additional subnet on VRF A. It uses the same address space as VRF B but they are completely separate so that should be fine (right?!).

PE1(config)#ip route vrf A 192.168.3.0 255.255.255.0 192.168.1.10

Customer A is now happy as their new network is reachable over the VRF but all of a sudden we have customer B on the phone, complaining that their site (which used to work) is off the air. Looking into PE 4 we can see why:

PE4#show ip route vrf B

Routing Table: B

Gateway of last resort is not set

      192.168.19.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.19.0/24 is directly connected, FastEthernet1/0
L        192.168.19.1/32 is directly connected, FastEthernet1/0
PE4#

The route to 192.168.3.0/24 has disappeared! Why is that? Looking on the route reflector gives us the answer:


RR#show bgp vpnv4 uni all 192.168.3.0/24
BGP routing table entry for 100:2439:192.168.3.0/24, version 14
Paths: (2 available, best #1, no table)
  Advertised to update-groups:
     3
  Refresh Epoch 1
  Local, (Received from a RR-client)
    10.255.255.1 (metric 3) from 10.255.255.1 (10.255.255.1)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      mpls labels in/out nolabel/25
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 1
  Local, (Received from a RR-client)
    10.255.255.3 (metric 3) from 10.255.255.3 (10.255.255.3)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Extended Community: RT:100:200
      mpls labels in/out nolabel/28
      rx pathid: 0, tx pathid: 0
RR#

Both 192.168.3.0/24 prefixes are being advertised with the same RD but different route targets. The route reflector has, therefore, seen the two "identical" prefixes and has chosen a best path - for want of a better metric it has chosen based on lowest next hop IP (PE1, VRF A):

Since the route reflector only advertises best paths to its clients, that means nobody gets to hear about the route from PE 3, VRF B. The route advertised from PE1 has a route target of 100:100, which doesn't match any VRFs on PE4 so it just discards the route leaving it with no way to reach the 192.168.3.0/24 network.

This proves that:

If you fail to apply globally unique route distinguishers on at least a per VRF basis, changes in one VRF can impact on another. This is irrespective of whether there are devices common to the two VRFs and occurs even when their route targets are completely different.

Policy at the PE

A similar but more subtle example of where globally unique route distinguishers are a benefit is in the case where you have a multi-homed network connected or routed via two PEs for resilience.
We want to use the the purple link to reach this prefix when it's available for administrative reasons (say the purple link is cheaper, or faster). Routes learned over the purple link are tagged with community 100:123 to allow upstream PEs to recognise this. Let's compare the case where both PEs use the same RD vs. the case where each PE uses a unique RD for the same VRF. Firstly, the same RD:

PE1 and PE2 are set to use the same RD. PE3 wants to use purple routes so it is set up with a policy to favour anything with a 100:123 community attached, as follows:

ip vrf A
 rd 100:2512
 import map A-import-map
 route-target export 100:100
 route-target import 100:100
!
route-map A-import-map permit 10
 match community purple
 set local-preference 200
!
route-map A-import-map permit 20
 set local-preference 100
!
ip community-list standard purple permit 100:123

For some reason, though, our traffic all goes out via the orange link. What is happening here is that the route reflector is again receiving  two identical prefixes - this does not cause a reachability problem as both prefixes reside within the same VRF, but it does mean that the route reflector makes a best path determination and discards one of the routes. PE3 only receives one route so its policy has to take what it can get:

PE3#show bgp vpnv4 uni all 192.168.200.0/24
BGP routing table entry for 100:2439:192.168.200.0/24, version 61
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 4
  200
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2512:192.168.200.0/24, version 68
Paths: (1 available, best #1, table A)
  Not advertised to any peer
  Refresh Epoch 4
  200, imported path from 100:2439:192.168.200.0/24 (global)
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0x0
PE3#

Clearly this is not doing what we want. The local VRF table only has one option, and that's the orange route. Let's try the same thing but with unique RDs per VRF per PE:

Now we see this at PE3:

PE3#show bgp vpnv4 uni all 192.168.200.0/24
BGP routing table entry for 100:2439:192.168.200.0/24, version 61
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 4
  200
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2458:192.168.200.0/24, version 62
Paths: (1 available, best #1, no table)
  Not advertised to any peer
  Refresh Epoch 4
  200
    10.255.255.2 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Community: 6553723
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/29
      rx pathid: 0, tx pathid: 0x0
BGP routing table entry for 100:2512:192.168.200.0/24, version 64
Paths: (2 available, best #1, table A)
  Not advertised to any peer
  Refresh Epoch 4
  200, imported path from 100:2458:192.168.200.0/24 (global)
    10.255.255.2 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 200, valid, internal, best
      Community: 6553723
      Extended Community: RT:100:100
      Originator: 10.255.255.2, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/29
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 4
  200, imported path from 100:2439:192.168.200.0/24 (global)
    10.255.255.1 (metric 4) from 10.255.255.200 (10.255.255.200)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Extended Community: RT:100:100
      Originator: 10.255.255.1, Cluster list: 10.255.255.200
      mpls labels in/out nolabel/30
      rx pathid: 0, tx pathid: 0
PE3#

Now we can see that two routes are received (purple and orange) and our route map has taken effect, pushing the purple route up to a better local preference and causing it to be selected into the VRF B table.

In summary, if you use the same route distinguisher at more than one point where the same IP prefix is learned, the best path determination will occur at the route reflector, not the receiving PE. This best path determination is likely to be quite coarse and applying per-VRF policies on route reflectors is inappropriate. Using unique RDs ensures that multiple copies of the same IP prefix can be learned by other PEs, allowing the best path determination to be done by the receiving PE using arbitrary local policies on a per-VRF basis.

Fast Failover

The final example is one of the most widely seen use cases for unique RD per VRF per PE. Let's take a look at the failover times for a route to move between PE1 and PE2 in the following scenario:

In the case of matching RDs, only one route for the destination is learned throughout the network so when a failure occurs a series of BGP updates need to occur before traffic can switch paths. In a real environment this chain of updates may take time. In a scaled environment (and for illustrative purposes in this lab), there may be hierarchical route reflectors and these may be configured with an update delay. Here is an example failover with two-tiered route reflectors and an update delay of 10s:

CE2#ping 1.1.1.1 repeat 100000

Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.......UUUUUUUUUUUUUUUUU
UUUUU.UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU.UUUUUUUUUUUUUUUU
UUUUUUUUUUUUUUUUUUUUUUUUUUU.UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
UUUUU.UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU.!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
Success rate is 37 percent (130/344), round-trip min/avg/max = 4/8/152 ms
CE2#

This failover takes around 30 seconds due to cascading updates being batched and delayed multiple times. The "U" marks above show that the edge PE has no route to the destination, due to having received the withdraw from the primary path but not yet having received the advertisement from the standby. The diagram below shows the BGP updates which need to take place before routing converges to the standby path:

Compare this to the output when unique RDs are set, meaning that the alternate path is already learned throughout the network but is simply not selected by the ingress PE:

 
CE2#ping 1.1.1.1 repeat 100000

Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.......!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
Success rate is 94 percent (131/139), round-trip min/avg/max = 5/9/16 ms
CE2#

As you can see, the failover is much faster, around 10-15s, and there are no unreachables as the PE always has a route in its table (even if it is a stale one for a while).

Super Fast Failover

This can be improved further by using label per VRF mode in addition to unique RDs. Without going into too much detail, the standard mode for Cisco IOS is to generate a label per prefix. The LFIB of the generating PE will have an entry saying "if I receive label X, I will stick encap Y on it and throw it out of interface Z". This can be changed as follows:

PE2(config)#mpls label mode vrf A protocol bgp-vpnv4 per-vrf 

In label per VRF mode, the same label is advertised for all prefixes advertised from within a particular VRF - the corresponding LFIB entry essentially says "rip off the label and route the packet that follows". When in label per VRF mode, we don't wait for any BGP updates at all because the egress PE where the primary link just failed can instantly use the standby route, which it already knows thanks to the unique RDs. Traffic gets U-turned back into the MPLS network while the BGP convergence occurs, but the traffic at least arrives:

Traffic temporarily hops via the primary PE into the secondary, restoring connectivity while BGP takes its sweet time to converge. Once the routing updates have propagated, traffic will go directly to the secondary PE. Failover times here are much more impressive:

CE2#ping 1.1.1.1 repeat 100000

Type escape sequence to abort.
Sending 100000, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (200/201), round-trip min/avg/max = 5/7/12 ms
CE2#

One ping lost / two seconds to fail over. Much better, and only possible with unique RDs!

Using unique RDs allows for much faster failover times, due to decreased numbers of BGP updates being required to converge following failures. This is particularly true when using label per VRF mode, since egress PEs can U-turn traffic without waiting for any BGP convergence at all.

AS-Override and the Importance of SoO

Recently I discovered that AS-override works in the opposite direction to what I thought! Now, this is largely academic as in most cases if you apply it to one peer you apply it everywhere, but I was dealing with a bit of a corner case and it caught me out as I had to mess about with (i.e. clear) a peer that I didn't really want to touch.

Cisco's config guides are a little bit ambiguous, saying:

"To configure a provider edge (PE) router to override the autonomous system number (ASN) of a site with the ASN of a provider, use the as-override command in VRF neighbor address family configuration mode. To restore the system to its default condition, use the no form of this command."

No mention whatsoever of in which direction the override happens. I always thought that a PE configured with AS override just didn't add the peer's AS to the AS_PATH when it received routes from the peer. It turns out I know nothing and that is not how it works at all.

In fact, as-override has no effect at all on received routes; it works only in the outbound direction. This makes sense, really, as the AS_PATH within carrier the carrier remains true (i.e. the service provider still gets to see what AS the routes originally came from). It's only when advertising routes out of the AS that as-override makes a difference, "overriding" the peer's ASN with the provider's.

But what does "overriding" mean? Let's take a look at some scenarios.

Simple Base Case

In a simple case where the AS_PATH (as seen by the provider) only contains a single entry and this corresponds with the peer's ASN, clearly the provider just replaces that with their own ASN. By "replace" I mean the peer's ASN is overwritten by the carrier's and then, as the route is advertised via eBGP, the carrier's ASN is added as normal. The route the peer receives, therefore, has an AS_PATH containing two copies of the carrier's ASN:

CE2#show ip bgp
BGP table version is 5, local router ID is 10.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  10.0.1.0/24      10.2.2.1                               0 100 100 i
 *>  10.0.2.0/24      0.0.0.0                  0         32768 i
CE2#

Prepended

So what if there are multiple copies of the peer's ASN at the start of the path? Well, as you might expect the whole topology doesn't suddenly tumble down. All copies of the peer ASN are replaced wih the carrier's ASN (after all, if we only replaced the first then the peer would still see it's own ASN and drop the update) before, again, adding the carrier's ASN as the route is advertised:

PE1#show ip bgp vpnv4 vrf cust1
BGP table version is 5, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:100 (default for vrf cust1)
 *>  10.0.1.0/24      10.1.1.2                 0             0 65000 65000 65000 i
 *>  10.0.2.0/24      10.2.2.2                 0             0 65000 i
PE1#

As observed on the PE, the route learned from CE1 has been prepended twice giving a total AS_PATH length of 3. All three of these 65000s will all be overridden when advertised towards CE2, creating the three 100s in orange and another copy of the local ASN (in red) will be added on egress as shown:

CE2#show ip bgp
BGP table version is 7, local router ID is 10.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  10.0.1.0/24      10.2.2.1                               0 100 100 100 100 i
 *>  10.0.2.0/24      0.0.0.0                  0         32768 i
CE2#

ASN Arbitrarily Contained in the AS_PATH

Another possibility exists where there are multiple AS involved. What if the customer connects to two different carriers who, in turn, connect to each other? This introduces the possibility that a customer route is learned from the other carrier, which then needs to be advertised out to the customer. The diagram below probably explains things better:

Thankfully, as-override doesn't seem to be too fussy (unlike remove-private-as in earlier IOS) and will replace the ASN wherever it appears in the path. It literally operates like a "find and replace all". Here's the AS_PATH from the PE's perspective showing the customer's ASN, followed by the other carrier's ASN:

PE1#show ip bgp vpnv4 vrf cust1              
BGP table version is 6, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:100 (default for vrf cust1)
 *>  192.168.0.0      172.16.1.2                             0 200 65000 i
PE1#

And here we see the route from CE2's perspective, with the 65000 (customer's) ASN replaced by the provider's ASN (100 shown in orange), followed by the untouched transit AS (in blue) and finally the provider's ASN is added on egress (in red):

CE2#show ip bgp   
BGP table version is 4, local router ID is 10.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.0.0      10.2.2.1                               0 100 200 100 i

CE2# 

Don't forget - the other carrier will also have to use as-override, otherwise CE1 will discard CE2's routes.

In Coordination with "local-as"

Now, as you'd imagine as-override works in conjunction with "local-as" on the PE. The happy news is that when local-as is in use, the ASN specified in the local-as command is used to override the customer ASN (after all, we're pretending to be that ASN). The bad news is you get some funny looking AS_PATHs.

Let's take a simple case where two CEs connect to a single PE, actually configured as AS 50 but masquerading as AS100:

As you can see, on the PE, the route learned from CE1 shows the customer's ASN and also one copy of the pretend AS which is tacked on by default when using the local-as command:

PE#show ip bgp vpnv4 vrf cust1
BGP table version is 4, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:100 (default for vrf cust1)
 *>  10.10.10.0/24    10.1.1.2                 0             0 100 65000 i
PE#

If we look on CE2 we can see:

CE2#show ip bgp
BGP table version is 33, local router ID is 192.168.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  10.10.10.0/24    10.2.2.1                               0 100 50 100 100 i
CE2#

To explain this strange arrangement, we have:

• 100 - the pretend ASN (normal local-as behaviour, added on egress)
• 50 - the real ASN (normal local-as behaviour, added on egress)
• 100 - the pretend ASN (normal local-as behaviour, added on ingress from CE1)
• 100 - the pretend ASN used in place of the customer ASN (as-override)

Now, it's possible to set "local-as" with the "no-prepend" directive. This makes the situation slightly cleaner in that the pretend ASN is no longer added as routes are received by the PE from the relevant peer. In other words you lose the orange ASN out of the path, but notice that in this example the no-prepend has to be applied on the PE's peering with CE1 in order to clean up CE2's BGP table....

Really, though, how many bodges do you want in play at once?

The Importance of SoO

Whenever altering the behaviour of something as important as BGP's loop prevention mechanism it is important to have a safety net. Unless you're very careful it's possible to introduce routing loops, particularly where multiple ISPs / ASNs are involved. Site of Origin, or SoO for short, provides just such a safety net.

The mode of operation is as follows:

• A SoO extended community is allocated for each customer site
• The SoO value is configured against each customer BGP peer within the PE router
• As routes are learned from a neighbour, the SoO extended community is attached to them to indicate their site of origin
• The PE checks any routes that are waiting to be advertised to a BGP peer and handles them according to the following rules:
• Any routes that are found to have the same SoO as the peer are not advertised to that particular peer
• Any routes that have a SoO community different to the peer's are advertised to that peer
• Any routes that do not have a SoO community attached are advertised

Now, SoO is occasionally overlooked as as-override often appears to work without it. Really, though, you are storing up problems for later.

Here's an example of SoO config on the PE:

router bgp 100
!
 address-family ipv4 vrf cust1
  neighbor 10.1.1.2 remote-as 65000
  neighbor 10.1.1.2 activate
  neighbor 10.1.1.2 as-override
  neighbor 10.1.1.2 soo 100:1
  neighbor 10.2.2.2 remote-as 65000
  neighbor 10.2.2.2 activate
  neighbor 10.2.2.2 as-override
  neighbor 10.2.2.2 soo 100:2
 exit-address-family

If there were two links into the same site (or into two sites joined by a backdoor network) then we would set the same SoO on both of its links. Since it uses an extended community (and this is a VRF so extended communities must be turned on) the SoO principle works across sites as well. It's important that different sites use different SoO values, otherwise they will not be able to learn each other's routes.

Problems Without SoO

There's one minor, almost cosmetic, quirk you get if you enable as-override without SoO:

If two CEs attach to the same PE then they will receive a copy of their own routes back from the PE.

This is a strange side-effect of the way update-groups work. Normally the PE would receive routes from both CEs, put together a list of updates and send them to both CEs - at this point each CE would see its own updates but would weed them out due to the AS_PATH containing the local ASN. With as-override enabled on the PE, the customer ASN is overridden with the provider's and the CE has no way to tell that the route was just echoed back.

Normally this doesn't matter as other mechanisms cause the locally injected route to be preferred (weight is set to 32768 for locally originated prefixes unless overridden, static routes generally have a better AD, etc) so it just looks a bit weird. There are cases where this does cause (rather drastic) problems, though. Take the following, not too far-fetched situation:

When the primary feed is up everything is great. The local preference of routes learned over the secondary feed is set to 50 by a route-map to ensure that they are less preferable than those received from the primary.

Let's break the primary feed and see what happens:

The withdrawals ripple through the network until CE2 is aware that the primary has gone away, decides to use the secondary and announces the route upstream. Seems legit so far...

Ah, no... this doesn't look right. The PE has echoed the route back to CE2 and, since the echoed route doesn't contain the local ASN and has a default local-preference of 100 it is now CE2's favourite.

 

Now we're in a right knot. CE2 has told the PE that it has a new route to use, but it contains the carrier ASN in the AS_PATH. The PE drops that as a loop, removes the route from its BGP table and sends a withdrawal message to CE2.

We are effectively back to the start where CE2 only has one option - it will take the route it is learning over the secondary feed and advertise it to the PE. Round and around we go...

There are a few tell-tale signs that this is happening. First of all there will be intermittent connectivity (usually in 30 second steps):

CE2#ping 10.10.10.1 repeat 1000
Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.0.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!................!!!!!!!!
!!!!!!!!!!!!!!!!!!!!

The next big giveaway is that when you run "show ip route" the age of the affected route(s) is always very low, typically under 30 seconds on standard BGP timers, and the path alternates between the same two next hops over and over:

CE2#show ip route 10.10.10.0
Routing entry for 10.10.10.0/24
  Known via "bgp 65000", distance 20, metric 0
  Tag 100, type external
  Last update from 10.2.2.1 00:00:29 ago
  Routing Descriptor Blocks:
  * 10.2.2.1, from 10.2.2.1, 00:00:29 ago
      Route metric is 0, traffic share count is 1
      AS Hops 3
      Route tag 100
      MPLS label: none
CE2#show ip route 10.10.10.0
Routing entry for 10.10.10.0/24
  Known via "bgp 65000", distance 20, metric 0
  Tag 200, type external
  Last update from 192.168.2.1 00:00:02 ago
  Routing Descriptor Blocks:
  * 192.168.2.1, from 192.168.2.1, 00:00:02 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 200
      MPLS label: none
CE2#

Finally, another good indication is that your BGP table version number is through the roof and continuously incrementing:

CE2#sh ip bgp
BGP table version is 51423, local router ID is 192.168.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  10.10.10.0/24    10.2.2.1                               0 100 100 200 i
 *                    192.168.2.1              0     50      0 200 i
 
CE2#

Note that you can also see the genuine and echoed routes in the BGP table (sometimes, re-check periodically).

It's possible to bodge together a route-map or prefix-list to 'fix' this, in fact just applying any unique route-map outbound on the PE will put the peer into a separate update-group which will bodge it into action. Please just use SoO, though - that's what it's there for!

References

Cisco's SoO feature description