Using QinQ to Build Flexible Lab Topologies

If you're sick and tired of re-cabling your lab every time you want to try out a new topology then you should probably consider using a QinQ tunnelling switch instead. With a QinQ switch at the heart of your lab network it's possible to stand up completely arbitrary topologies with very little effort, no re-cabling and even set it up from a remote location.

In this post I'll walk through setting up a Cisco 3560 to act as a central QinQ switch and how to set up a few example topologies. This can also be done on older / lower spec switches using the same concepts - 802.1Q tunnelling is supported in most of the gear you'll find on eBay.

Theory

Service providers like to aggregate as many customers onto a single link as possible - otherwise they can't be price competitive. Customers want circuits that allow them to not only trunk VLANs but to use and and all of the 4095 possible VLAN IDs without having to check with their service provider first.

One (adequate) way to do this is to extend the notion of VLANs. We all know VLANs separate a LAN into multiple logical partitions using a VLAN identifier tag - QinQ takes that to the next level by stacking 2 VLAN tags on top of each other. If the service provider assigns a VLAN to each customer then they can be used to segregate customers from one another as follows:

Note here that both customers use VLAN 10, however they each get their own VLAN 10 independent of any other customer's. Customers can use any VLAN numbers they like, irrespective of what other customers or the provider has chosen to use.

In service provider parlance, the first (or outer) VLAN tag is the Service Provider VLAN (or S-VLAN). A customer's VLAN which get tunnelled through is called Customer VLAN (or C-VLAN). There is a hierarchy in that one S-VLAN may have multiple C-VLANs, while a C-VLAN can only have one parent S-VLAN. VLAN IDs can be re-used across customers, however customer A's VLAN 10 is different to customer B's VLAN 10.

For our lab setup we will do exactly the same thing, but locally on a single switch. 802.1Q tunnelling not only allows us to connect trunks to each other over a VLAN but also, optionally, control protocols such as LACP, spanning tree, CDP and so on can also be tunnelled through, giving the impression that the end devices are actually attached to each other rather than through a switch.

Physical Topology

Here's a simple lab setup - 2 PCs, 2 routers, 2 firewalls and 2 switches (plus our QinQ switch):

Basically, we just need to plug all the devices into the QinQ switch. Use as many ports as you have available in the QinQ switch and be sure to label them up with descriptions! Sensibly, though, you will really want a minimum of 4 ports for a switch and at least 3 for a firewall. You can get most of the functionality you want with using sub-interfaces on a router so in a pinch you can usually get away with having single links if need be. It's all about giving yourself the maximum flexibility.

Initial Setup

Before we even begin with the configuration proper, we have to work around a dopey default behaviour in Cisco switches. Straight out of the box most Catalyst switches have a layer 2 switching MTU of 1500 for both Fast Ethernet and Gigabit Ethernet ports. This needs to be overridden to allow full size frames to be passed through with a VLAN tag still attached.

Adjust the maximum MTU for Fast Ethernet ports as follows:

Lab-QinQ(config)#system mtu 1998
Changes to the system MTU will not take effect until the next reload is done

Note - different switches have different maximum values. Use the "?" key to see what your device will go to and pick the maximum

Adjust the maximum MTU for Gigabit Ethernet ports as follows:

Lab-QinQ(config)#system mtu jumbo 9000
Changes to the system jumbo MTU will not take effect until the next reload is done

Again, different devices may have different maximum values so use the "?" key to find how far you can set this.

Now, as it says, reboot the switch to make the changes stick.

Once the switch is back up and running, the next job is to create your "provider" VLANs. Note - these VLANs will not be visible to your "customer" topology so it's good to pick a range of consecutive VLAN IDs. I like to start at 100 and work up. Note that each of these VLANs needs to have its MTU increased from the default to allow transport of full size frames with the additional VLAN tag:

Lab-QinQ(config)#vlan 100
Lab-QinQ(config-vlan)# name xconnect100
Lab-QinQ(config-vlan)# mtu 1900
Lab-QinQ(config-vlan)#exit

You will need one VLAN for each virtual connection between devices - I usually throw 20 in and add more later if required.

Finally, for every port where you will attach a device, set the switchport into 802.1Q tunnel mode, and enable all the protocol tunnelling options:

Lab-QinQ(config)#interface range Fa0/1 - Fa0/24
Lab-QinQ(config-if-range)# switchport mode dot1q-tunnel
Lab-QinQ(config-if-range)# l2protocol-tunnel cdp
Lab-QinQ(config-if-range)# l2protocol-tunnel stp
Lab-QinQ(config-if-range)# l2protocol-tunnel vtp
Lab-QinQ(config-if-range)# l2protocol-tunnel point-to-point pagp
Lab-QinQ(config-if-range)# l2protocol-tunnel point-to-point lacpLab-QinQ(config-if-range)# l2protocol-tunnel point-to-point udld

Lab-QinQ(config-if-range)# spanning-tree portfast trunk

Setting Up a Basic Topology

OK so we have the switch set up, let's set up a really simple topology:

The basic process here is to look at your diagram and anywhere you see a line, assign it one of your provider VLAN numbers:

Now, for each port on the topology, go and set the relevant switch port as a member of that VLAN:

Lab-QinQ(config)#int Fa0/1
Lab-QinQ(config-if)#description PC1
Lab-QinQ(config-if)#switchport access vlan 101
Lab-QinQ(config-if)#interface Fa0/6
Lab-QinQ(config-if)#description SW1 Gi1/4
Lab-QinQ(config-if)#switchport access vlan 101

Lab-QinQ(config-if)#interface Fa0/3
Lab-QinQ(config-if)#description SW1 Gi1/1
Lab-QinQ(config-if)#switchport access vlan 102
Lab-QinQ(config-if)#interface Fa0/11
Lab-QinQ(config-if)#description R1 Fa0/1
Lab-QinQ(config-if)#switchport access vlan 102

As you can see, VLAN 101 is used to "connect" PC1 to SW1 port Gi1/4, while VLAN 102 is used to "connect" SW1 port Gi1/1 to R1 port Fa0/1. Thanks to the protocol tunnelling config, SW1 and R1 believe they are directly connected:

R1#show cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW1              Fas 0/1           168             R S I  WS-C6503- Gig 1/1

MAC entries are learned as if the devices were directly attached and the router and PC can ping each other:

R1#ping PC1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
R1#


This is a fairly simple example but as we're about to get into, way more complex topologies can be achieved using the same methods.

Slightly More Complex Setup

Let's mix things up a bit by building a topology with an HA pair of Juniper SRX firewalls, trunking VLANs down to a pair of switches, connected by an LACP link bundle (portchannel):

As with the other example, we simply assign a provider VLAN to each link:

I won't include the config here for each of these as it's a pure repetition of the earlier work - just make sure that your ports go into the right VLANs and everything should be fine.

Now we can see that the firewalls have come up in HA:

{primary:node0}
root@SRX-top> show chassis cluster status
Monitor Failure codes:
    CS  Cold Sync monitoring        FL  Fabric Connection monitoring
    GR  GRES monitoring             HW  Hardware monitoring
    IF  Interface monitoring        IP  IP monitoring
    LB  Loopback monitoring         MB  Mbuf monitoring
    NH  Nexthop monitoring          NP  NPC monitoring
    SP  SPU monitoring              SM  Schedule monitoring

Cluster ID: 1
Node   Priority Status         Preempt Manual   Monitor-failures

Redundancy group: 0 , Failover count: 1
node0  100      primary        no      no       None
node1  50       secondary      no      no       None

Redundancy group: 1 , Failover count: 1
node0  100      primary        no      no       None
node1  50       secondary      no      no       None

The switches have bundled their ports and can see each other's device IDs:

SW-2#show lacp internal
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi0/1     SA      bndl      32768         0x1       0x1     0x112       0x3D
Gi0/2     SA      bndl      32768         0x1       0x1     0x113       0x3D
 
SW-2#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi0/1     SA      32768     3037.a6ca.aa80  10s    0x0    0x1    0x112   0x3D
Gi0/2     SA      32768     3037.a6ca.aa80   9s    0x0    0x1    0x113   0x3D


And, as before, CDP works fine:

SW-2#show cdp neighbor
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW-1             Gig 0/1           152              S I   WS-C3560G Gig 0/1
SW-1             Gig 0/2           164              S I   WS-C3560G Gig 0/2
SW-2#

Even UDLD is active and "sees" the device on the other end as if it were locally connected:

SW-1#show udld Gi0/1

Interface Gi0/1
---
Port enable administrative configuration setting: Enabled / in aggressive mode
Port enable operational state: Enabled / in aggressive mode
Current bidirectional state: Bidirectional
Current operational state: Advertisement - Single neighbor detected
Message interval: 7
Time out interval: 5

    Entry 1
    ---
    Expiration time: 44
    Device ID: 1
    Current neighbor state: Bidirectional
    Device name: FOCABCD0123
    Port ID: Gi0/1
    Neighbor echo 1 device: FOCWXYZ9876
    Neighbor echo 1 port: Gi0/1

    Message interval: 15
    Time out interval: 5
    CDP Device name: SW-2
SW-1#

As a side note, a lot of Cisco kit only supports long LACP timers (90 second failure detection, as opposed to 3 seconds for short) so if you are in this boat then consider using UDLD when configuring bundles over indirect links. This should reduce detection time to 45 seconds, which is still a bit rubbish but better than 90. By default, error-disable recovery is not active for UDLD so once UDLD takes a link down, it stays down - so you probably want to switch recovery on:

SW-1(config)#errdisable recovery cause udld

 Simulating Failures

One of the more common uses for a lab environment is to test failovers. One of the more common failure types to want to simulate is a link failure, however this is not quite as straightforward with a QinQ switch in the middle as taking one port down does not make the other go down, e.g.:

If you want to simulate pulling a link, I find the best way is to use an interface range command. Let's say we want to remove the link between FW1 and SW1, simply specify an interface range on the QinQ switch containing the switch ports facing each of those two devices and shut them down at the same time:

Lab-QinQ(config)#interface range fa0/14, fa0/21
Lab-QinQ(config-if-range)#shut
.May 30 12:55:30 UTC: %LINK-5-CHANGED: Interface FastEthernet0/14, changed state to administratively down
.May 30 12:55:30 UTC: %LINK-5-CHANGED: Interface FastEthernet0/21, changed state to administratively down
.May 30 12:55:31 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/14, changed state to down
.May 30 12:55:31 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/21, changed state to down

The same technique can be used to simulate the failure of an entire cabinet or site, just select all the ports you need in a range and shut them down.

Another common test that you may want to perform is to simulate a "silent failure", i.e. where both ends see the link up but traffic is lost in the middle. This is good for checking how quickly and how well protocol heartbeats detect link problems (think LACP, UDLD, routing protocols, etc) and is definitely worth checking before you put services over carrier circuits or inter-DC links. To achieve this, simply set the provider VLAN on one of your ports to something unused:

Lab-QinQ(config)#interface fa0/14
Lab-QinQ(config-if-range)#switchport access vlan 2

The two ends of the link will remain up but no traffic will pass through.

References

Cisco Documentation on Jumbo Frames
Cisco Documentation on UDLD

Basic Internet Connectivity Setup Using HWIC-3G-GSM Card

In addition to working on some spanking new 4G Cisco 819 devices, I've occasionally had to slum it by providing Internet access with a normal 1800 / 2800 series router and an HWIC-3G-GSM card. Once you know what's involved the config is remarkably simple but it can be difficult to understand what's what at first.

The video below takes a very quick walk through setting this up, some further explanation is given beneath that for those interested.

Building Blocks

While there are a few mobile-specific pieces of configuration, anyone who has previously worked on ISDN, async modems or ADSL on Cisco routers will probably find a lot of familiar concepts. Here are the main elements of a 3G / 4G configuration:

Cellular Profile 

This is where the APN address and authentication mode are configured. These are saved to the modem's NVRAM as soon as they are applied. Here's an example of how to set a cellular profile on the two different platforms:

Router#cell 0/0/0 gsm profile create 1 three.co.uk
Profile 1 will be created with the following values:                            
PDP type = IPv4                                                                 
APN = three.co.uk                                                               
Are you sure? [confirm]                                                         

Profile 1 written to modem                                                      

Router#

The number "1" here indicates which profile slot on the modem will be used to store the details. This is significant later on because there may be multiple APNs configured and the router needs to know which to use when connecting.

This example is for three, a UK mobile carrier which is interesting because the APN uses no authentication. If your APN requires authentication, simply follow the APN with either a pap or chap keyword, the username and finally the password.

Note that this is applied at the exec prompt rather than in config mode.

Cellular Interface

The physical radio interfaces are referred to using the "Cellular" prefix, in this case Cellular0/0/0. The Cellular interface is where the dialer, authentication and IP details are normally configured - I say normally as there are many different ways to configure dialers depending on what kind of load balancing and resilience are required. For a typical 3G deployment, though, you will only have one physical interface and so the simplest way is to forget about pools and put the config straight onto that.

Here is an example configuration showing the key elements:

interface Cellular0/0/0
 ip address negotiated
 encapsulation ppp
 dialer in-band
 dialer string "*98*1#"
 dialer-group 1
 ppp chap refuse
!
dialer-list 1 protocol ip permit

ip route 0.0.0.0 0.0.0.0 Cell0/0/0

The key thing to notice here is the "*98*1#" dialer string. The "*98*" and "#" are fixed, the "1" refers to the profile slot number used earlier. If you used a different slot, refer to it here.

The rest is fairly standard dialer stuff, in this example I've made the dialer-list so that any IP traffic will cause it to connect.

Sundry Config

At this point the router should be able to connect to the cellular network. For most purposes, though, you will need to either set up NAT or some sort of VPN tunnel for the connection to be of any use. These are set up the same way as for any other setup.

Testing and Diagnostics

How can you tell whether the cellular connection is coming up? The first clue is that log entries similar to the following should appear:

%LINK-3-UPDOWN: Interface Cellular0/0/0, changed state to 
up

To check whether the modem is attached to the radio network, use the following commands:

Router#show cell 0/0/0 network                                                  
Current Service Status = Normal, Service Error = None                           
Current Service = Combined                                                      
Packet Service = HSDPA (Attached)                                               
Packet Session Status = Active                                                  
Current Roaming Status = Home                                                   
Network Selection Mode = Automatic                                              
Country = GBR, Network = 3 UK                                                   
Mobile Country Code (MCC) = 234                                                 
Mobile Network Code (MNC) = 20                                                  
Location Area Code (LAC) = 24                                                   
Routing Area Code (RAC) = 24                                                    
Cell ID = 14827                                                                 
Primary Scrambling Code = 81                                                    
PLMN Selection = Automatic                                                      
Registered PLMN = 3 , Abbreviated =                                             
Service Provider =                                                              
Router#show cell 0/0/0 radio                                                    
Radio power mode = ON                                                           
Current Band = WCDMA 2100, Channel Number = 10564                               
Current RSSI = -76 dBm                                                          
Band Selected = Auto                                                            
Number of nearby cells = 1                                                      
Cell 1

        Primary Scrambling Code = 0x51
        RSCP = -77 dBm, ECIO = -0 dBm           
                                                                                
Router#

Note that the band and channel need to be populated, the network should display the expected carrier name and the packet service should show as attached. The actual band and service type will vary depending on carrier, coverage, area and equipment used.

If the network status remains in "Emergency Only" and you get no MSISDN showing in your show cell 0/0/0 hardware command, particularly if it is accompanied by messages saying "%CELLWAN-2-SIM_LOCKED: [Cellular0/0/0]: SIM is locked", then you have probably locked the SIM (i.e. by setting up a startup PIN on a phone handset) and will need to unlock it as follows:

Router #cell 0/1/0 gsm sim unlock 1234
!!!WARNING: SIM will be unlocked with pin=1234(4).
Do not enter new PIN to unlock SIM. Enter PIN that the SIM is configured with.
Call will be disconnected!!!
Are you sure you want to proceed?[confirm]

*Dec  7 22:15:38.035: %LINK-3-UPDOWN: Interface Cellular0/0/0, changed state to up
Router#

If the radio interface is up but a data connection cannot be established then all the usual debugs may be used:

debug dialer (to verify it is trying to dial)
debug chat (sometimes useful to deduce whether APN is configured correctly)
debug ppp negotiation (shows the PPP negotiation process from agreeing basic link properties and authentication type, through the authenticating stage and up to IP being allocated)

A full deep-dive into these debugs wouldn't really be appropriate for this post, in any case it's usually fairly evident where the problem lies. UPDATE: The promised dialer / PPP debugging guide is available here - it's written for PPPoE but the vast majority of it is applicable to cellular interfaces as well.

References

Video accompanying this blog post

Configuring Basic 4G LTE Connectivity on the Cisco 819 Router

I've recently had the mixed fortune to have set up a couple of Cisco routers for 3G and 4G data services. It turns out to be surprisingly simple, although I found myself having to flit around between a handful of different documents to work out how to get it working. Luckily I was sat next to someone who previously worked in one of the UK's biggest mobile carriers while I was working on it, which saved me a bit of head scratching at times.

As a little bonus I had two different devices to work on - the first being an old Cisco 1841 router with a 3G WIC installed, the second being a Cisco 819 (4G LTE model). As it turns out the concepts are pretty similar but the syntax is moderately different between the two platforms, so in time I'll write up the process for each. This post and its accompanying video will explain the 4G version.

Building Blocks

While there are a few mobile-specific pieces of configuration, anyone who has previously worked on ISDN, async modems or ADSL on Cisco routers will probably find a lot of familiar concepts. Here are the main elements of a 3G / 4G configuration:

Cellular Profile 

This is where the APN address and authentication mode are configured. These are saved to the modem's NVRAM as soon as they are applied. Here's an example of how to set a cellular profile on the two different platforms:

Router#cell 0 lte profile create 1 three.co.uk none

PDP Type = IPv4
Access Point Name (APN) =
Username =
Password =
Authentication = NONE

Profile 1 already exists with above parameters. Do you want to overwrite? [confirm]

Profile 1 will be overwritten with the following values:

PDP type = IPv4
APN = three.co.uk
Username =
Password =
Authentication = NONE

Are you sure? [confirm]
Profile 1 written to modem
Router#

Note that this example is for three, a UK mobile carrier which is interesting because the APN uses no authentication (and barfs if you try to authenticate with it). I found during my testing that an 819 router running IOS 15.2 does not have the option to use authentication type "none". Under 15.3 the option is there and works fine - luckily I had another 819 with 15.3 installed which worked and so a) I knew that's what the problem was and b) I could copy the image across!

Also note that this is applied at the exec prompt rather than in config mode.

Most carriers use CHAP authentication, these just require the authentication type and credentials added to the command, for example:

cell 0 lte profile create 1 everywhere chap eesecure secure

The number "1" here indicates which slot on the modem will be used to store the profile. This is significant later on because there may be multiple APNs configured and the router needs to know which to use when connecting.

Cellular Interface

The physical radio interfaces are referred to using the "Cellular" prefix, in this case Cellular0. The Cellular interface is where the dialer, authentication and IP details are normally configured - I say normally as there are many different ways to configure dialers depending on what kind of load balancing and resilience are required. For a typical 3G / 4G deployment, though, you will only have one physical interface and so the simplest way is to forget about pools and put the config straight onto that.

Here is an example configuration showing the key elements:

interface Cellular0
 ip address negotiated
 encapsulation slip
 dialer in-band
 dialer-group 1
!
dialer-list 1 protocol ip permit

ip route 0.0.0.0 0.0.0.0 Cell0


The key thing to notice here is that unlike the old 3G config there is no "*98*1#" type diaper string. If you want to use alternative profiles you have to mess around with config under the "controller cellular 0" context.

The rest is fairly standard dialer stuff, in this example I've made the dialer-list so that any IP traffic will cause it to connect.

Note that the encapsulation specified on this 4G interface is SLIP. When 4G is not available it will fall back to 3G (which uses PPP encapsulation) - it does this transparently and does not need the encap changed.

Sundry Config

At this point the router should be able to connect to the cellular network. For most purposes, though, you will need to either set up NAT or some sort of VPN tunnel for the connection to be of any use. These are set up the same way as for any other setup.

Testing and Diagnostics

How can you tell whether the cellular connection is coming up? The first clue is that log entries similar to the following should appear:

%LINK-3-UPDOWN: Interface Cellular0, changed state to up

If the above log entries don't appear it could be because the modem is not ready yet. The modems in the 819 routers I was playing with took an incredibly long time to boot. The following logs indicate that the modem has (finally) booted up:

%CISCO800-2-MODEM_UP: Cellular0 modem is now UP.
%CISCO800-6-SIM_STATUS: SIM in slot 0 is present

However at this point the modem will still need to attach to the cellular network, which can take a little time. To check whether the modem is attached to the radio network, use the following commands:

Router#show cell 0 radio
Radio power mode = ON
Channel Number = 1667
Current Band = LTE
Current RSSI = -60 dBm
Current RSRP = -84  dBm
Current RSRQ = -4  dB
Current SNR = 10.6  dB
LTE Technology Preference = AUTO
LTE Technology Selected = LTE

Router#show cell 0 network
Current System Time = Sun Jan 6 0:1:30 1980
Current Service Status = Normal
Current Service = Packet switched
Current Roaming Status = Home
Network Selection Mode = Automatic
Network = 3
Mobile Country Code (MCC) = 234
Mobile Network Code (MNC) = 20
Packet switch domain(PS) state = Attached
Registration state(EMM) = Registered
Location Area Code (LAC) = 107
Cell ID = 9150878
Primary Scrambling Code = 65535

Note that the band and channel need to be populated, the network should display the expected carrier name and the packet service should show as attached. The actual band and service type will vary depending on carrier, coverage, area and equipment used.

If the radio interface is up but a data connection cannot be established then all the usual debugs may be used:

debug dialer (to verify it is trying to dial)
debug chat (sometimes useful to deduce whether APN is configured correctly)
debug cellular 0 messages callcontrol (shows the cellular network assigning the IP and DNS)

Note on Earlier Releases

Prior to IOS 15.3 you had to define your own chat string and apply it to the line - later releases do this automatically. If your "debug chat" output shows anything about "ATDT" or expecting "CONNECT" then this probably applies to you. Making and applying the chat script is pretty simple:

chat-script lte "" "AT!CALL" TIMEOUT 20 "OK"

line 3

 script dialer lte

!

Basically this says to define a script called "lte" which waits for nothing, sends "AT!CALL" to the modem and expects to get "OK" in return within 20 seconds. Then that script gets attached to line 3, which in the 819 router is the 4G cellular modem.

References

Cisco 4G configuration guide
YouTube clip accompanying this post