Building a simple Linux multipath router

2022-05-03

I am occasionally running LAN parties. The available internet bandwidth is often an issue. I wanted a simple way to balance traffic over multiple internet uplinks. It is assumed that there will be NAT on or after the egress interface of the router.

This first version tries to be simple, while not being optimal in all cases.

The basic feature we will be using is multipath routing. Instead of a simple default route our router will have a default route with multiple nexthops. This can be added via iproute2:

ip route add 0.0.0.0/0 \
    nexthop via 172.21.0.1 dev bond0.300 weight 3 \
    nexthop via 172.21.101.1 dev eno1 weight 1

This route will balance the traffic to the nexthops 172.21.0.1 and 172.21.101.1 over the specified interfaces. The weight setting will balance the flows (statistically) 3 to 1 between the links. You can adjust that according to your available bandwidth.

You also might want to check, that there is no other default route with a lower metric, because that will be preferred.

L4 Hashing

By default the load balancing is only done on L3 (IP) headers. A hash is calculated over these fields and that is then used to assign the flow to a nexthop. So given a big enough number of flows you should see a flow distribution according to your weights. To use the L4 headers (which will make it possible to balance flows of the same client over different uplinks) we have to set the sysctl

net.ipv4.fib_multipath_hash_policy = 1

Caveats

There are some issues with this setup:

  1. It is not tested in production yet.
  2. There might be issues with protocols that use multiple ports in parallel (e.g. ftp) or services that need to track users by IP address. This might be "solved" by sticking to L3 header hashing only.
  3. This setup in this naive form does not consider link latency and special traffic classification
  4. This does not handle traffic shaping. If you want to rate limit the traffic, you have to do this via tc, nft or whatever you prefer.
  5. In the case that you are not running NAT but have a public address space that is reachable via all of your uplinks you can only control the way packets are leaving your network. The ingress way is out of your control.
  6. Keep in mind that different flows might cause different amounts of traffic. If you have 100 flows and a 1:1 split, then you will see ~50 flows per nexthop. If you are unlucky the 50 flows for one nexthop might need a lot of traffic and fill up that link while the other 50 are not enough to fill the other link. Although this scenario is rather unlikely.

Testing the setup

To test the setup you have to send traffic from an address that is not on the transit networks to your nexthops, because otherwise the route selection algorithm might influence the route decision

You can use mtr -I $iface 9.9.9.9 to check which nexthop you get. Just select a appropriate interface or use a second computer behind the router. In the later case you can skip the -I $iface parameter. When you use several different destination addresses you should see the different nexthops you specified.

Alternatively ip route get $target from $address should return you different next hops for different targets.

References

Booting Cisco 3560E switches with IOS 15.2 does not work

2021-09-01

TL;DR: You can not boot IOS 15.2 from the 3560X switches on 3560E switches.

The why and what happens if you try it

I have a couple of 3560E switches. Sadly they only run IOS 15.0. I wanted to have IPv4 address families over OSPFv3. This is not supported in IOS 15.0. But it is in IOS 15.2. And the IOS 15.2 image for the successor model to the 3560E, the 3560X, is called c3560e-universalk9-mz.152-4.E10.bin. And the IOS 15.0 firmware for the 3560X switches is the same file as for the 3560. So I tried to boot the 15.2(4)-E10 image. The switch itself boots the image but hits a malloc error during the boot process, crashes and reboots. So no OSPFv3 with IPv4 address family on those switches.

Here is the output of the boot process:

<...snip...>
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

Waiting for Port download...Complete
SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!



%Software-forced reload


 00:01:01 UTC Mon Jan 2 2006: Unexpected exception to CPUvector 2000, PC = 36828B0
-Traceback= 0x36828B0z 0x2AFD98Cz 0x2B05F5Cz 0x3678A24z 0x367DF60z 0x2B2A9E0z 0x31D2AE0z 0x3118FDCz 0x3206400z 0x3209A2Cz 0x2AE5D30z 0x2AE5EFCz 0x2AE604Cz 0x6D921Cz 0x6D9454z 0x3683BA0z

Writing crashinfo to flash:/crashinfo_ext/crashinfo_ext_4

=== Flushing messages (00:01:03 UTC Mon Jan 2 2006) ===

Buffered messages:
Queued messages:
*Jan  2 00:01:03.929: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Mar  1 00:00:07.012: Read env variable - LICENSE_BOOT_LEVEL =
*Jan  2 00:00:03.456: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c3560e Next reboot level = ipservices and License = ipservices
*Jan  2 00:01:01.329: %SYS-2-MALLOCFAIL: Memory allocation of 60000 bytes failed from 0x3678A20, alignment 0  <<< This is the relevant part.
Pool: Processor  Free: 38104  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool
 -Process= "Init", ipl= 0, pid= 3
-Traceback= 6C9738z 2AFD8F8z 2B05F5Cz 3678A24z 367DF60z 2B2A9E0z 31D2AE0z 3118FDCz 3206400z 3209A2Cz 2AE5D30z 2AE5EFCz 2AE604Cz 6D921Cz 6D9454z 3683BA0z
Cisco IOS Software, C3560E Software (C3560E-UNIVERSALK9-M), Version 15.2(4)E10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 31-Mar-20 21:44 by prod_rel_team

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x031BB534  SRR1 = 0x00029230  SRR2 = 0x036828B0  SRR3 = 0x00029230
ESR = 0x00000000  DEAR = 0x00000000  TSR = 0x84000000  DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000  PC = 0x036828B0  MSR = 0x00029230  CR = 0x35000053
LR = 0x0368284C  CTR = 0x006B4068  XER = 0xE0000075
R0 = 0x0368284C  R1 = 0x06357288  R2 = 0x00000000  R3 = 0x04F897A8
R4 = 0x00000000  R5 = 0x00000000  R6 = 0x06357258  R7 = 0x05850000
R8 = 0x00029230  R9 = 0x05850000  R10 = 0x00008000  R11 = 0x00000000
R12 = 0x35000059  R13 = 0x079817A8  R14 = 0x053AFDD8  R15 = 0x00000000
R16 = 0x038B2550  R17 = 0x00000004  R18 = 0x00000020  R19 = 0x05850000
R20 = 0x00000000  R21 = 0x00000000  R22 = 0x0000EA60  R23 = 0x00000000
R24 = 0x00000000  R25 = 0x053AFDD8  R26 = 0x03678A20  R27 = 0x0000EA60
R28 = 0x00000000  R29 = 0x05EE80C0  R30 = 0x05BA0978  R31 = 0x00000000

Stack trace:
PC = 0x036828B0, SP = 0x06357288
Frame 00: SP = 0x06357298    PC = 0x0368284C
Frame 01: SP = 0x063572C8    PC = 0x02AFD98C
Frame 02: SP = 0x06357360    PC = 0x02B05F5C
Frame 03: SP = 0x06357378    PC = 0x03678A24
Frame 04: SP = 0x06357390    PC = 0x0367DF60
Frame 05: SP = 0x063573C0    PC = 0x02B2A9E0
Frame 06: SP = 0x063573E8    PC = 0x031D2AE0
Frame 07: SP = 0x06357400    PC = 0x03118FDC
Frame 08: SP = 0x06357418    PC = 0x03206400
Frame 09: SP = 0x06357458    PC = 0x03209A2C
Frame 10: SP = 0x06357468    PC = 0x02AE5D30
Frame 11: SP = 0x06357488    PC = 0x02AE5EFC
Frame 12: SP = 0x063574A0    PC = 0x02AE604C
Frame 13: SP = 0x063574D0    PC = 0x006D921C
Frame 14: SP = 0x063575D8    PC = 0x006D9454
Frame 15: SP = 0x063575E0    PC = 0x03683BA0

<...switch reboots...>

The other interesting problem is now: How to reset the switch? Normaly the switch would try the next image on disk if the first one fails, but this image works "good enough" for that mechanic not to kick in. This means that the switch is stuck in an infinite boot loop.

Resetting the switch to the old image

Disconnect the power, connect to the console port, press and hold the Mode button. Then plug the power cord back in. After 10-20 seconds release the Mode button. After a short time the switch: prompt should appear and you are in the bootloader.

To boot the old image you have to initialize the filesystem (flash_init), find the image with dir flash: and delete it with delete flash:/c3560e-universalk9-mz.152-4.E10.bin. After that boot the switch with boot. The switch tries to boot the image that is in the config file, fails and tries the first available image on the device which is probably your old working image. If no working image is on the flash have fun with xmodem or tftp.


Using driver version 1 for media type 2
Base ethernet MAC Address: 08:1f:f3:39:8c:80
Xmodem file system is available.
The password-recovery mechanism is enabled.

The system has been interrupted prior to initializing the
flash filesystem.  The following commands will initialize
the flash filesystem, and finish loading the operating
system software:

    flash_init
    boot

switch: ?
           ? -- Present list of available commands
        boot -- Load and boot an executable image
         cat -- Concatenate (type) file(s)
        copy -- Copy a file
      delete -- Delete file(s)
         dir -- List files in directories
  flash_init -- Initialize flash filesystem(s)
      format -- Format a filesystem
        fsck -- Check filesystem consistency
        help -- Present list of available commands
      memory -- Present memory heap utilization information
       mkdir -- Create dir(s)
        more -- Concatenate (display) file(s)
      rename -- Rename a file
       reset -- Reset the system
       rmdir -- Delete empty dir(s)
         set -- Set or display environment variables
      set_bs -- Set attributes on a boot sector filesystem
   set_param -- Set system parameters in flash
       sleep -- Pause (sleep) for a specified number of seconds
        type -- Concatenate (type) file(s)
       unset -- Unset one or more environment variables
     version -- Display boot loader version

switch: flash_init
Initializing Flash...
mifs[2]: 10 files, 1 directories
mifs[2]: Total bytes     :    2097152
mifs[2]: Bytes used      :     614400
mifs[2]: Bytes available :    1482752
mifs[2]: mifs fsck took 2 seconds.
mifs[3]: 3 files, 1 directories
mifs[3]: Total bytes     :    4194304
mifs[3]: Bytes used      :     949248
mifs[3]: Bytes available :    3245056
mifs[3]: mifs fsck took 2 seconds.
mifs[4]: 5 files, 1 directories
mifs[4]: Total bytes     :     524288
mifs[4]: Bytes used      :       9216
mifs[4]: Bytes available :     515072
mifs[4]: mifs fsck took 0 seconds.
mifs[5]: 5 files, 1 directories
mifs[5]: Total bytes     :     524288
mifs[5]: Bytes used      :       9216
mifs[5]: Bytes available :     515072
mifs[5]: mifs fsck took 1 seconds.
 -- MORE --
mifs[6]: 15 files, 3 directories
mifs[6]: Total bytes     :   57671680
mifs[6]: Bytes used      :   49069056
mifs[6]: Bytes available :    8602624
mifs[6]: mifs fsck took 26 seconds.
...done Initializing Flash.

switch: dir

List of filesystems currently registered:

                  bs[0]: (read-only)
               flash[6]: (read-write)
              xmodem[7]: (read-only)
                null[8]: (read-write)

switch: dir flash:
Directory of flash:/

    2  -rwx  20310016  <date>               c3560e-universalk9-mz.150-2.SE11.bin
    3  drwx  512       <date>               crashinfo_ext
    8  -rwx  1560      <date>               express_setup.debug
    9  -rwx  916       <date>               vlan.dat
   10  -rwx  64        <date>               ztp.py
   11  -rwx  26771456  <date>               c3560e-universalk9-mz.152-4.E10.bin
   12  -rwx  1920      <date>               private-config.text
   13  -rwx  5144      <date>               multiple-fs
   14  -rwx  6223      <date>               config.text
   15  drwx  512       <date>               crashinfo

8602624 bytes available (49069056 bytes used)

switch: delete flash:c3560e-universalk9-mz.152-4.E10.bin
Are you sure you want to delete "flash:c3560e-universalk9-mz.152-4.E10.bin" (y/n)?y
File "flash:c3560e-universalk9-mz.152-4.E10.bin" deleted

switch: boot
Loading "flash:c3560e-universalk9-mz.152-4.E10.bin"...flash:c3560e-universalk9-mz.152-4.E10.bin: no such file or directory

Error loading "flash:c3560e-universalk9-mz.152-4.E10.bin"

Interrupt within 5 seconds to abort boot process.
Loading "flash:/c3560e-universalk9-mz.150-2.SE11.bin"...@@@@@@@@@@@@@@@<... switch booting image as usual ...>

Compressing IPv6 Addresses with Regular Expressions

2021-03-06

This post was lying around for too long, time to finish it and get it off my todo list.

The Goal

Convert from a full IPv6 address like 2001:0db8:0000:0000:0000:0023:4200:0123 to a compressed one like 2001:db8::23:4200:123 with a regular expression. Why? Because I can. And because some prometheus exporters only give you the uncompressed addresses.

Details regarding IPv6 address compression can be found in RFC 5952 Section 4.

Preprocessing

The first step is to get from the full form to a form where the leading zeros are removed/reduced to just a single zero per block.

This could be done like this:

^0{0,3}(.*:)0{0,3}(.*:)0{0,3}(.*:)0{0,3}(.*:)0{0,3}(.*:)0{0,3}(.*:)0{0,3}(.*:)0{0,3}(.*)$

and $1$2$3$4$5$6$7$8 as a replacement string. This results in 2001:db8:0:0:0:23:4200:123

The Common Case

Now the funny part begins. Basically we have to find the first longest sequence of zero-blocks that is longer than 1 block. If we make a group with everything to the left and right of that (including the : ) and combine the 2 groups we get the result. There are some corner cases, those will be handled later.

To find a sequence of length N we can build a very simple expression like

0:0:< in total N zeros >:0:0

Everything to the left of that must have less N consecutive zero blocks. A expression to match that could be:

((0:){0,$N-1}[1-9a-f][0-9a-f]{0,3}:)*

Everything to the right of the zero block sequence can have at most N consecutive zero blocks. The expression for that looks like this.

(:[1-9a-f][0-9a-f]{0,3}(:0){0,$N})*

Combined they result in this expression:

(((0:){0,$N-1}[1-9a-f][0-9a-f]{0,3}:)*)0:0:< in total N zeros >:0:0((:[1-9a-f][0-9a-f]{0,3}(:0){0,$N})*)

Combining the first and fourth group of that expression results in the compressed representation of the address, if the longest zero block sequence is N blocks long. We can build the expression for all possible block lengths.

The Corner Cases

As mentioned earlier there are several corner cases.

Compression at the left or right side

If the longest sequence is at the left or right end (e.g. 0:0:0:0:0:0:1:2 and 2:1:0:0:0:0:0:0) then we need a special expression. For the left side it looks like this:

0(:)0:< N-1 times 0 in total >:0:0((:[1-9a-f][0-9a-f]{0,3}(:0){0,$N})*)

This solves 2 problems:

  1. The group that would be on the left side with the expression for the common case needs to be removed, because there is nothing to match there.
  2. we have to find a : to build the :: in the compressed representation. This is done by taking one from the N zero blocks of the longest sequence.

the right side is constructed the same way.

Compressing 7 consecutive zero blocks

When there are 7 consecutive zero blocks then the compression will happen at either the left or right side, because there is only one non zero block left. for the 7 only the two expressions for the sides are needed, not the common one.

Compressing 8 zero blocks

All the other expressions don't work for the one case of 8 consecutive zeros. But we have to get 2 : for the :: from somewhere. This could look like this:

0(:)0(:)0:0:0:0:0:0

Combining it all

In total we get 3 expressions each for 2, 3, 4, 5 and 6 consecutive zero blocks, 2 for the 7 consecutive zero blocks and 1 for the 8 zero blocks.

Those 18 expressions can be combined like this:

^(($EXPR1)|($EXPR2)|...)$

Building all of this by hand is shitty and annoying. Here is some code to do it.

zero = "0"
non_zero = "[1-9a-f]"
all_chars = "[0-9a-f]"
non_zero_chunk = f"{non_zero}{all_chars}{{0,3}}"

def max_n_zero_block_left(n: int) -> str:
    return f"((({ zero }:){{0,{n}}}{ non_zero_chunk }:)*)"

def max_n_zero_block_right(n: int) -> str:
    return f"((:{ non_zero_chunk }(:{ zero }){{0,{n}}})*)"

def n_length_zero_block(n: int) -> str:
    return ":".join(zero*n)

left_zero_prefix = f"0(:)"
right_zero_suffix = f"(:)0"

patterns = list()
replacement_positions = list()
next_pattern_group_start = 1

for n in range(7,1,-1):
        # special case for the longest continuos zero string at the left
        pattern = f"({ left_zero_prefix }{ n_length_zero_block(n - 1) }({ max_n_zero_block_right(n) }))"
        patterns.append(pattern)
        replacement_positions.append(next_pattern_group_start + 2)
        replacement_positions.append(next_pattern_group_start + 3)
        next_pattern_group_start += 6

        # special case for ending with 0
        pattern = f"(({ max_n_zero_block_left(n-1) }){ n_length_zero_block(n - 1) }{ right_zero_suffix })"
        patterns.append(pattern)
        replacement_positions.append(next_pattern_group_start + 2)
        replacement_positions.append(next_pattern_group_start + 6)
        next_pattern_group_start += 6

        if n == 7:
            continue  # the regular case does not exist for n=7

        # regular case
        pattern = f"(({ max_n_zero_block_left(n-1) }){ n_length_zero_block(n) }({ max_n_zero_block_right(n) }))"
        patterns.append(pattern)
        replacement_positions.append(next_pattern_group_start + 2)
        replacement_positions.append(next_pattern_group_start + 6)
        next_pattern_group_start += 9

patterns.append("(0(:)0(:)0:0:0:0:0:0)")
replacement_positions.append(next_pattern_group_start + 2)
replacement_positions.append(next_pattern_group_start + 3)

all_patterns = "|".join(patterns)
all_patterns = "^(" + all_patterns + ")$"
print("Expression:")
print(all_patterns)

replacement = "".join(f"${{{i}}}" for i in replacement_positions)
print("Replacement:")
print(replacement)

And the output of the script:

Expression:
^((0(:)0:0:0:0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,7})*)))|(((((0:){0,6}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0:0:0:0(:)0)|(0(:)0:0:0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,6})*)))|(((((0:){0,5}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0:0:0(:)0)|(((((0:){0,5}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0:0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,6})*)))|(0(:)0:0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,5})*)))|(((((0:){0,4}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0:0(:)0)|(((((0:){0,4}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,5})*)))|(0(:)0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,4})*)))|(((((0:){0,3}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0(:)0)|(((((0:){0,3}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,4})*)))|(0(:)0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,3})*)))|(((((0:){0,2}[1-9a-f][0-9a-f]{0,3}:)*))0:0(:)0)|(((((0:){0,2}[1-9a-f][0-9a-f]{0,3}:)*))0:0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,3})*)))|(0(:)0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,2})*)))|(((((0:){0,1}[1-9a-f][0-9a-f]{0,3}:)*))0(:)0)|(((((0:){0,1}[1-9a-f][0-9a-f]{0,3}:)*))0:0(((:[1-9a-f][0-9a-f]{0,3}(:0){0,2})*)))|(0(:)0(:)0:0:0:0:0:0))$
Replacement:
${3}${4}${9}${13}${15}${16}${21}${25}${27}${31}${36}${37}${42}${46}${48}${52}${57}${58}${63}${67}${69}${73}${78}${79}${84}${88}${90}${94}${99}${100}${105}${109}${111}${115}${120}${121}

Please keep in mind that I am just an idiot on the internet, don't use this expression to burn your production environment down.

Interesting Observations of IOS(-XE) ACL CLI and Command Syntax

2020-11-26

My initial contact for the things shown in this article comes from trying to parse and generate ACLs. Many of the things here may not bother you, if you are a pure CLI user and are not generating configs. This article does not claim to be the ultimate source for all the weird details of ACL syntax behaviour. This is based on my experience which primarily comes from IOS 15.7, IOS-XE 03.16, IOS-XE 16.09 and 16.12. IOS-XE 16.12 changed a lot in regards to sequence numbers. So there will be several sections that will be divided into a pre IOS-XE 16.12 part and a post IOS-XE 16.12 part. the regular IOS can be considered a part of the pre IOS-XE 16.12 sections, because I have not found differences between those.

General things

An Access Control List is a series of entries. Each entry matches some parts of a the packet headers. Each entry is either a remark or a permit or deny action. (I will ignore reflexive ACLs and the like here.)

Sequence Numbers

Each entry in an ACL has a sequence number. By default the first entry starts at 10 and every further entry has a number that is 10 higher.

When modifying entries you can specify a sequence number at which position you want to add something. So if you want to insert something between entry 10 and 20 you could choose a unused number between 10 and 20, e.g. 15.

Resequencing

But what happens if you want to insert between two entries where there is no free sequence number? For Legacy IP ACLs you can do a ip access-list <type> <name> resequence which renumbers the entries so that each entry is now 10 numbers apart again.

But that is not implemented for IPv6... A workaround is to recreate the ACL. But that is annoying.

IPv4

Pre IOS-XE 16.12

On IOS and IOS-XE the IPv4 sequence numbers are not a part the config. This means that you have to do show ip access-list ... to see the sequence numbers. They are generated at runtime. This also means that the sequence numbers change after a reboot.

Post IOS-XE 16.12

Sequence numbers are now a part of the configuration. The implications will be discussed later.

IPv6

Pre IOS-XE 16.12

IPv6 sequence numbers can be a part of the config. They are shown on a per entry basis if the sequence number is not exactly 10 bigger than the previous one.

If you enter the commands:

ipv6 access-list test
sequence 10 permit ipv6 host 2001:db8::1 any
sequence 15 permit ipv6 host 2001:db8::2 any
sequence 20 permit ipv6 host 2001:db8::3 any
sequence 25 permit ipv6 host 2001:db8::4 any
sequence 35 permit ipv6 host 2001:db8::5 any
sequence 40 permit ipv6 host 2001:db8::6 any

Then the ACL in the config is this:

ipv6 access-list test
 permit ipv6 host 2001:DB8::1 any
 sequence 15 permit ipv6 host 2001:DB8::2 any
 sequence 20 permit ipv6 host 2001:DB8::3 any
 sequence 25 permit ipv6 host 2001:DB8::4 any
 permit ipv6 host 2001:DB8::5 any
 sequence 40 permit ipv6 host 2001:DB8::6 any

The 2001:db8::2, 2001:db8::3 and 2001:db8::4 have numbers because their distance to the sequence number of the previous entry is 5. 2001:db8::5 has no number because the distance is 10, and 2001:db8::6 has a distance of 5 and therefore has a sequence number.

The show ipv6 access-list test command shows you all sequence numbers, but at the end of the entry, not at the beginning like your config file does it.

IPv6 access list test
    permit ipv6 host 2001:DB8::1 any sequence 10
    permit ipv6 host 2001:DB8::2 any sequence 15
    permit ipv6 host 2001:DB8::3 any sequence 20
    permit ipv6 host 2001:DB8::4 any sequence 25
    permit ipv6 host 2001:DB8::5 any sequence 35
    permit ipv6 host 2001:DB8::6 any sequence 40

Post IOS-XE 16.12

In IOS-XE 16.12 the sequence numbers are always shown in the config. So there is at least one point where IOS got more consistent.

ipv6 access-list test
 sequence 10 permit ipv6 host 2001:DB8::1 any
 sequence 15 permit ipv6 host 2001:DB8::2 any
 sequence 20 permit ipv6 host 2001:DB8::3 any
 sequence 25 permit ipv6 host 2001:DB8::4 any
 sequence 35 permit ipv6 host 2001:DB8::5 any
 sequence 40 permit ipv6 host 2001:DB8::6 any

But sadly the output of the show ipv6 access-list command still puts the sequence number at the end of the entries.

IPv4 Remarks

IPv4 ACL Remarks do not have their own sequence numbers.

Pre IOS-XE 16.12

This means inserting remarks in IPv4 ACLs at a specific position does not work. Only permit and deny are allowed.

asr920(config-ext-nacl)#42 ?
  deny    Specify packets to reject
  permit  Specify packets to forward

If you want to add a remark somewhere in the middle you have to delete the whole ACL, and insert the remark at the correct position while recreating the ACL.

But if you want to insert a new entry with a remark you can work your way around this issue. First you insert a remark without a sequence number and insert an entry with a sequence number and that remark will end up at the same position as the other entry.

ip access-list extended abcd
10 permit ip host 10.0.0.0 any
20 permit ip host 10.0.0.1 any
remark test
15 permit ip host 10.0.0.2 any

results in

ip access-list extended abcd
 permit ip host 10.0.0.0 any
 remark test
 permit ip host 10.0.0.2 any
 permit ip host 10.0.0.1 any

Post IOS-XE 16.12

In IOS-XE 16.12 every entry got a sequence number. But as mentioned earlier, IPv4 remarks did not have sequence numbers. So Cisco "solved" that. An ACL now looks like this in the config:

ip access-list extended remark-seq-numbers
 10 remark foo
 10 permit ip host 10.0.0.0 any
 20 remark bar
 20 permit ip host 10.0.0.42 any

Also inserting remarks at a specific sequence number works now (to some degree). If you enter 15 remark test for the ACL above you get this:

ip access-list extended remark-seq-numbers
 10 remark foo
 10 permit ip host 10.0.0.0 any
 20 remark bar
 20 remark asdf
 20 permit ip host 10.0.0.42 any
 15 remark test

Our remark is now at the end, where it does not belong. This change in IOS-XE 16.12 does result in sequence numbers appearing multiple times and being out of order.

Reusing the same sequence number

When you want to replace an entry in an IPv6 ACL with an other one you can just use the same sequence number and the entry will be replaced.

If you try that with an IPv4 permit/deny entry you get this response:

% Duplicate sequence number
%Failed to add ace to access-list

(yes, the space after the % in the first line and the lack thereof in the second is not a copy+paste error.)

However if you do this in IOS-XE 16.12 with a remark line it works just fine and it replaces the remark.

Singular/Plural in the ACL show commands

While the config commands for ACLs are ip[v6] access-list the show-commands are show ip access-lists with a s at the end and show ipv6 access-list without a s at the end.

c1111-lab#show ip access-lists
Standard IP access list 2
 ...
c1111-lab#show ipv6 access-list
IPv6 access list test
 ...
c1111-lab#show ipv6 access-lists
                               ^
% Invalid input detected at '^' marker.

ACL naming: numbers vs names

There is not only the difference between standard and extended ACLs, there is also a difference between ACLs with a name and ACLs with a number. Which numbers can be given to an ACL depends on their type.

asr920(config)#access-list ?
  <1-99>            IP standard access list
  <100-199>         IP extended access list
  <1300-1999>       IP standard access list (expanded range)
  <2000-2699>       IP extended access list (expanded range)
  <2700-2799>       MPLS access list
<snip>

To make things more complicated the numbers for standard and extended ACLs each have 2 seperate ranges.

A little tip at this point: do not use numbered ACLs. Use ACLs with names, because you can give them a name that hopefully tells you and others what this ACL is for. Or can you remember what ACL 42 was for? and where it is used? (good luck with sh run | incl 42 for large configs)

Standard vs Extended Numbered ACL Config format

Pre IOS-XE 16.12

If you use an extended ACL with a number (please don't) you probably enter

ip access-list extended 101
permit ip host 10.0.0.0 any

and that is exactly what will end up in your config. However if you want to use a standard ACL with a number (pls dont) you enter

ip access-list standard 1
permit host 10.0.0.0

And you will end up with this in your config:

access-list 1 permit 10.0.0.0

post IOS-XE 16.12

This format is no longer used in IOS XE 16.12, it has been changed to the same format as named ACLs are using and it looks like this:

ip access-list standard 1
 10 permit 10.0.0.0

This is nice, because it is less inconsistent. But unless you have absolutly no legacy gear you have to support it anyways.

Remarks

Remarks are very handy when trying to understand lengthy ACLs (lengthy in my case sometimes means hundreds of lines, but others might laught at that). Sadly IOS does not show remarks with show ip access-list, for those you have to take a look into your config.

Interface-ACLs that do not exist

If you delete an ACL that is configured on an interface, the config line is not deleted from the interface. But the interface now accepts all traffic.

asr920(config)#ip access-list extended delete-test
asr920(config-ext-nacl)#deny ip any any
asr920(config-ext-nacl)#do sh run int Gi0/0/0
interface GigabitEthernet0/0/0
 ip address dhcp
 ip access-group delete-test in
 negotiation auto
end

asr920(config-ext-nacl)#do sh ip access-list delete-test
Extended IP access list delete-test
    10 deny ip any any
asr920(config-ext-nacl)#do ping  10.0.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
asr920(config-ext-nacl)#exit
asr920(config)#no ip access-list extended delete-test
asr920(config)#do sh ip access-list delete-test
asr920(config)#do sh run int Gi0/0/0
interface GigabitEthernet0/0/0
 ip address dhcp
 ip access-group delete-test in
 negotiation auto
end

asr920(config)#do ping  10.0.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

IPv4 ACLs that only consist of remarks

But what if the ACL exists but there is no action statement in an ACL at all? We can build such an ACL that is only a remark. The ACL shows up in the config, but the default deny does not deny all traffic. But when we add an action the implicit deny suddenly works.

asr920(config)#ip access-list extended noentry
asr920(config-ext-nacl)#remark test1
asr920(config)#do sh run | section ip access-list extended noentry
ip access-list extended noentry
 remark test1
asr920(config-std-nacl)#in Gi0/0/0
asr920(config-if)#ip access-group noentry in
asr920(config-if)#do ping 10.0.0.42
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.42, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
asr920(config-if)#exit
asr920(config)#ip access-list extended noentry
asr920(config-ext-nacl)#permit ip host 192.168.0.0 any
asr920(config-ext-nacl)#do ping 10.0.0.42
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.42, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Default settings for config sections that can have an ACL

Things like SSH, NTP, SNMP, NETCONF, ... may (and probably should) be secured by ACLs. But by default they are not protected and answer to everything. This means you want to have an ACL for all of them. For IPv4 and IPv6. If you add the first IPv6 address to a device and you dont have an IPv6 ACL for those services configured then your device might be reachable via IPv6. So even if you haven't started with IPv6 for management and monitoring you have to keep in mind that your router listens on every address and if you have IPv6 enabled on any interface your router might be reachable via IPv6.

Port numbers

The router converts some port numbers to names. This is annoying if you want to parse and compare that. At least they match up with the services list published by IANA, altough in some cases the service-name column does not work and you have to use one of the aliases (e.g. port 80 is www instead of http) It's just an other piece that makes your parser a bit more complex.

Order of entries in standard ACLs

Standard ACLS are funny. They automagically put single addresses in front of larger prefixes. It looks like those changes are done in a way that does not influence what is permitted and denied by the ACL, but it makes some things hard to read, because now everything is out of order.

Entering

ip access-list standard order1
permit 10.0.0.0 0.0.0.255
permit host 10.0.42.0

results in this config:

ip access-list standard order1
 permit 10.0.42.0
 permit 10.0.0.0 0.0.0.255

But because that is not confusing enough, you can always add remarks to make things easier to understand...

These commands

ip access-list standard order2
permit 10.0.0.0 0.0.0.255
remark test1

Result in the expected config:

ip access-list standard order2
 permit 10.0.0.0 0.0.0.255
 remark test1

Add an other address, e.g. this:

permit 10.0.42.0

And your config now looks like this:

ip access-list standard order2
 remark test1
 permit 10.0.42.0
 permit 10.0.0.0 0.0.0.255

This is because the remark is always attached to the line after it (not adding a line after it is probably a corner case) and when the single address is added it is attached to this line. But single addresses are put to the top and the remark with them.

Putting one remark in front of several entries can screw you up beautifully. In this example we have 2 groups, called group1 and group2 Each group has a single address and a prefix in it. One could now build an ACL that looks like this.

ip access-list standard out-of-order
remark group1
permit 10.0.1.0 0.0.0.255
permit 10.0.0.0
remark group2
permit 10.0.2.0
permit 10.0.3.0 0.0.0.255

What ends up in the config is this:

ip access-list standard out-of-order
 remark group2
 permit 10.0.2.0
 permit 10.0.0.0
 remark group1
 permit 10.0.1.0 0.0.0.255
 permit 10.0.3.0 0.0.0.255

It looks like the groups have switched positions and 10.0.0.0 and 10.0.3.0 0.0.0.255 have changed groups. Good luck reverse engineering ancient firewall rules...

The sequence numbers in IOS-XE 16.12 can explain what happens here.

ip access-list standard out-of-order
 30 remark group2
 30 permit 10.0.2.0
 20 permit 10.0.0.0
 10 remark group1
 10 permit 10.0.1.0 0.0.0.255
 40 permit 10.0.3.0 0.0.0.255

Connecting to a Device via SSH from Netbox

2020-11-08

Connecting via SSH to a device in netbox would be nice. So I build a thing.

Keep in mind that you have to adjust almost every config or script to your environment.

First of all you need a custom link for devices in netbox that somehow encodes the hostname of your device in a URL.

So your URL could look like this:

ssh://{{ obj.name }}.example.com

An when you give it a name like {% if obj.device_role.name == "Router" %}SSH Login{% endif %} the link is only shown to devices with the router role.

When you click on the link your browser should ask you how to open it.

Now you can build a little script that parses that url, opens your favorite terminal and starts ssh in there. My script is called sshterminal. Dont forget to set execute permissions on the script.

#!/bin/bash

HOST=$(echo "$1" | cut -d"/" -f 3)

alacritty -e ssh $HOST

alacritty is currently the terminal of my choice, but any terminal that has an option to directly execute a programm should work. (thats what the -e option does.)

If something else opens that url you have to set up your browser to use the script. In firefox that is somewhere in the "applications" menu in about:preferences#general.

Now you have a terminal with an ssh connection to the device.

You probably also want to have some options like jump hosts, ancient ciphers for shitty routers, a username and an ssh key. So put something like this in your ssh config:

host *.example.com
    user admin
    Ciphers=+aes256-cbc
    KexAlgorithms=+diffie-hellman-group14-sha1
    IdentityFile=~/.ssh/my_private_key
    ProxyJump my_jump_host