Wednesday 3 April 2019

A "real" full Internet table / route server in the lab

A common requirement when lab / stress testing carrier kit is to have at least one copy of a full Internet routing table. While many people will use a network tester to generate, say, 700,000 routes programmatically, this is not really representative. Real Internet routes are not all the same size. They are not contiguous or predictable. They have different AS_PATHs of varying lengths. There are aggregates and specific prefixes, often both covering the same IP space. These factors may make no difference to your system under test, or they might make a huge difference - it's not unusual for TCAM to be partitioned or optimised by prefix length so surely it's best to test with the real thing, right?

This post shows how to build a fully populated and very fast BGP route server based on Ubuntu Bionic Beaver 18.04 LTS.

There are three main elements to this:
  • A daily dump of the RIPE RIB, which we will manipulate and shrink down to a single "view" ready for processing
  • A copy of RIPE's bgpdump and a (slightly tweaked) instance of the bgp_simple perl script which can be used to replay the processed dump file back to a listening BGP instance
  • An instance of BIRD, which is loaded up with routes by bgp_simple and can be used to re-advertise them (in a very fast and resilient fashion) to your systems under test.
This method overcomes a number of horrible issues - Trimming the RIB snapshot massively reduces the time to advertise the table, without compromising the number of prefixes. The bgp_simple instance is modified to reduce the amount of text output, which also increases speed. The BIRD instance sitting between bgp_simple and the system(s) under test increases robustness (BIRD will re-establish dropped BGP sessions whereas bgp_simple does not) and provides much faster update capability.

With these tweaks, a full table can be loaded into BIRD in around 2 minutes and, once fully loaded, BIRD can advertise the full table onward in a few seconds.

Topology


In this setup we will assume our route server will be 10.0.0.1 in AS 65001 and the system under test will be 10.0.0.2 in AS 65002.

Host Setup


This guide starts with a fresh Ubuntu 18.04 LTS instance. This can be a bare metal install, a VM or even a LXC container. You will need to set up this host's networking so that it can access the Internet, at least to begin with. Since this is a lab box, I'm being super sloppy and doing everything as root. You can mentally put "sudo" in front of everything if you'd rather.

First, update the packages and install a few necessary extras:

apt update && apt install bird git perl build-essential libz-dev libbz2-dev screen

Now we need to get some Perl libraries via CPAN. First open CPAN:

cpan

Accepting the auto config should normally do what you want. Within there, run the following:

install CPAN
reload CPAN
install Net::BGP


Exit out by pressing Ctrl-D

Now, edit /etc/bird/bird.conf as follows (replacing the existing "router id") config:

router id 10.0.0.1;
listen bgp address 10.0.0.1;

protocol bgp {
        local 10.0.0.1 as 65001;
        neighbor 10.0.0.2 as 65002;
        multihop;
        source address 10.0.0.1;
        next hop self;
        import all;
        export all;
}

protocol bgp {
        local 10.0.0.1 as 65001;
        neighbor 192.2.0.1 as 64999;
        multihop;
        source address 10.0.0.1;
        next hop self;
        passive;
        import all;
        export none;
}


Now, restart BIRD as follows:

service bird restart

At this point you should have BIRD running (you can check its status by running "birdc" and running some "show" commands - if your system under test is already configured you should see the peer come up). Next, we will set up bgp_simple to load the table into BIRD.

Before we can do that we need to download a few things:

git clone https://github.com/xdel/bgpsimple.git
wget ris.ripe.net/source/bgpdump/libbgpdump-1.6.0.tgz
wget data.ris.ripe.net/rrc00/latest-bview.gz


The daily dump file is pretty big and could take several minutes. Either wait for it or open another tab to build libbgpdump:

tar -xzf libbgpdump-1.6.0.tgz
cd libbgpdump-1.6.0/
./configure && make && make install
cd ..
rm -r libbgpdump-*


Now, the dump file contains many, many views of the full table. We just need one, so let's get a list of neighbours:

zcat latest-bview.gz | bgpdump -m - | head -50000 | cut -d '|' -f 4 | grep -v ':' | sort | uniq

Pick one of the IPs listed at random and filter down for just that neighbour. You can repeat this process using different addresses on different route servers if you'd like to simulate getting a table from two providers. In this example, we'll use 185.210.224.254, but any should be just as good.

zcat latest-bview.gz | bgpdump -m - | grep '185.210.224.254' > bgpsimple/routes.txt

This takes a looooong time, perhaps 25 minutes on a rubbish PC. Grab a cuppa.

Everything up until here will still be there after a reboot, the parts that follow will need to be re-run each time the route server gets rebooted.

Almost there, we need to add the unroutable test address onto the loopback interface:
ip address add 192.2.0.1/32 dev lo

Finally, we will make some minor tweaks to bgp_simple:

cd bgpsimple

Edit bgp_simple.pl and comment out lines 640-649 - this prevents the script echoing all 700,000 routes to the console as they are advertised (complete with their AS_PATH, MED, etc).

Now run an instance of "screen" to keep bgp_simple running after your CLI session ends:

screen -an

And execute the script:

./bgp_simple.pl -myas 64999 -myip 192.2.0.1 -peerip 10.0.0.1 -peeras 65001 -p routes.txt -holdtime 300 -keepalive 30

The script should connect and, after a short pause, tell you that it is advertising its routes to the peer. After a couple of minutes you should see a message to say the advertisement is complete.

At this point you can break out of screen using Ctrl-a, followed by "d" (reconnect later using "screen -r").

Finally, run "birdc" and execute "show route count" to confirm how many routes you are seeing. Once BIRD is loaded up it can blow a table down to a peer in seconds and can be configured for as many peers as you like.

Happy labbing!


Saturday 16 March 2019

How to use 25G ports in 1/10G mode on Cisco NCS (540, 55A2, etc)

I've recently been test driving a Cisco NCS 55A2, which has 24 x 1/10 Gbps and 16 x 1/10/25 Gbps ports. However, I ran into a problem where the 1/10/25 Gbps ports would not rate adapt down to 10 Gbps. There is no "speed" configuration under the port and seemingly nothing useful under controllers relevant to 25G ports.

I eventually got the answer from Cisco engineering but since I couldn't find this documented anywhere on the Internet I thought it might be worth preserving it here for prosperity!

Theory


Unlike 1/10G operation where the device just detects what optic is inserted then presents the appropriate GigabitEthernet or TenGigabitEthernet interface, 1/10/25G ports need to be hard set into either 1/10G or 25G mode. Frustratingly, when it comes to 25 Gbps ports, the NCS platforms set this in groups of four ports, each of which is referred to as a"quad". In another confusing move, Cisco has decided to put the quad config in a completely different place to very similar things such as the config to break out 40/100G ports into 10/25G members.

Configuration


As we can see here, the 1/10/25G ports default to 25G mode, reflected in their TFx/x/x/x naming:

RP/0/RP0/CPU0:55A2(config)#do show interfaces description | begin 0/0/0/24
TF0/0/0/24 admin-down admin-down
TF0/0/0/25 admin-down admin-down
TF0/0/0/26 admin-down admin-down
TF0/0/0/27 admin-down admin-down
TF0/0/0/28 admin-down admin-down
TF0/0/0/29 admin-down admin-down
TF0/0/0/30 admin-down admin-down
TF0/0/0/31 admin-down admin-down
TF0/0/0/32 admin-down admin-down
TF0/0/0/33 admin-down admin-down
TF0/0/0/34 admin-down admin-down
TF0/0/0/35 admin-down admin-down
TF0/0/0/36 admin-down admin-down
TF0/0/0/37 admin-down admin-down
TF0/0/0/38 admin-down admin-down
TF0/0/0/39 admin-down admin-down
Mg0/RP0/CPU0/0 admin-down admin-down

Now, let's configure the first four 25G ports into 1/10G mode. The configuration lives under the "hw-module" branch, and quads are numbered from 0 (up to 3 on this platform). To set the first four ports into 1/10G mode:

RP/0/RP0/CPU0:55A2(config)#hw-module quad 0 location 0/0/CPU0 mode 10g
RP/0/RP0/CPU0:55A2(config)#commit

As we can see, ports 24-27 have transformed into 1/10G ports and now appear with the Tex/x/x/x naming convention:

RP/0/RP0/CPU0:55A2(config)#do show interfaces description | begin 0/0/0/24
Te0/0/0/24 admin-down admin-down
Te0/0/0/25 admin-down admin-down
Te0/0/0/26 admin-down admin-down
Te0/0/0/27 admin-down admin-down
TF0/0/0/28 admin-down admin-down
TF0/0/0/29 admin-down admin-down
TF0/0/0/30 admin-down admin-down
TF0/0/0/31 admin-down admin-down
TF0/0/0/32 admin-down admin-down
TF0/0/0/33 admin-down admin-down
TF0/0/0/34 admin-down admin-down
TF0/0/0/35 admin-down admin-down
TF0/0/0/36 admin-down admin-down
TF0/0/0/37 admin-down admin-down
TF0/0/0/38 admin-down admin-down
TF0/0/0/39 admin-down admin-down
Mg0/RP0/CPU0/0 admin-down admin-down

Let's set the rest to 10G mode, because 25G optics are expensive and I don't have any in the lab:

RP/0/RP0/CPU0:55A2(config)#hw-module quad 1 location 0/0/CPU0 mode 10g
RP/0/RP0/CPU0:55A2(config)#hw-module quad 2 location 0/0/CPU0 mode 10g
RP/0/RP0/CPU0:55A2(config)#hw-module quad 3 location 0/0/CPU0 mode 10g
RP/0/RP0/CPU0:55A2(config)#commit
RP/0/RP0/CPU0:55A2(config)#do show interfaces description | begin 0/0/0/24
Te0/0/0/24 admin-down admin-down
Te0/0/0/25 admin-down admin-down
Te0/0/0/26 admin-down admin-down
Te0/0/0/27 admin-down admin-down
Te0/0/0/28 admin-down admin-down
Te0/0/0/29 admin-down admin-down
Te0/0/0/30 admin-down admin-down
Te0/0/0/31 admin-down admin-down
Te0/0/0/32 admin-down admin-down
Te0/0/0/33 admin-down admin-down
Te0/0/0/34 admin-down admin-down
Te0/0/0/35 admin-down admin-down
Te0/0/0/36 admin-down admin-down
Te0/0/0/37 admin-down admin-down
Te0/0/0/38 admin-down admin-down
Te0/0/0/39 admin-down admin-down
Mg0/RP0/CPU0/0 admin-down admin-down

Easy (when you know how)!