Zyxel NR7101 Performance Testing
2023-05-11
Introduction
The Zyxel NR7101 is a PoE-powered 5G router. It is often used to gain internet connectivity for LAN parties. This leads to large volumes of traffic going through the device. Some events reported reliability issues with this device. So we will investigate where the limitations of this device are.
Test-Setup
We connect our test client (in this case a computer running Linux) via Ethernet to the Zyxel router. That router then connects via LTE or 5G to the Broadband ISP (in this case Deutsche Telekom). For some of the tests we require a test server with internet connectivity to send packets from the internet to our test device.
It must be acknowledged, that some of the tests might be influenced by the network of the broadband ISP, however in some cases we can clearly show, that there is a limitation on the Zyxel router.
Accessing the Web-UI
The device can be configured via a Web UI. This can be done via the wireless network that this device creates or via a Ethernet connection.
From there we can configure many settings. Especially interesting for our purposes is setting the APN under Network Setting -> Broadband -> Cellular APN to a APN which gives us a public IPv4 address so that we do not have Carrier Grade NAT. An other important option is to enable IP Passthrough, so that that our test client actually gets the public IPv4 address and the router does not introduce an additional layer of NAT.
The web UI also offers an overview over many of the signal characteristics. An explanation of these values can be found in the Zyxel support portal
Finally the device can also be accessed via SSH. To do so one has to activate SSH via the Maintenance -> Remote Management Menu. Then we can log into the device via SSH with the user admin and the same password that is also used for the web UI. Your SSH client might be very new, so you might have to enable ssh-rsa as a the host key algorithm to log in.
ssh root@192.168.1.1 -o HostKeyAlgorithms=+ssh-rsa
However the SSH login only gives you access to a very minimalistic, proprietary CLI interface.
Gaining root access
Since the regular SSH login is very limited we can get a root login. The router generates it's root password from it's serial number. More details on how to generate that password can be found at the OpenWrt Wiki.
Equipped with this root password we have access to the Linux OS running on the device.
BusyBox v1.20.1 (2022-11-29 09:28:07 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.
_______ ___ ___ _______ _____
|__ |.--.--.| | || ___|| |_
| __|| | ||- -|| ___|| |
|_______||___ ||___|___||_______||_______|
|_____|
-------------------------------------------
Product: NR7101
Version: 1.00(ABUV.7)C0
Build Date: 2022/11/29
-------------------------------------------
root@NR7101:~#
It is a small Linux system, that runs the standard Linux networking stack. So you will find netfilter, iptables, iproute2, conntrack, tcpdump, etc.
General commands
Here is an overview of the available commands:
root@NR7101:~#
8021xd date gunzip mailsend pure-ftpd syslog-ng wan
8021xdi dbclient gzip makedevlinks.sh pwd syslogd watch
AutoSpeedTest dd halt md5sum qimeitool sysupgrade watchdog
Ethctl depmod head microcom qmi-network tail wc
QFirehose df hexdump mii_mgr qmicli tar wget
SpeedTest dhcp6c hostid mkdir quectel-CM taskset which
[ dhcp6relay hostname mkfifo quectel-DTool tc wifi
[[ dhcp6s hotplug-call mknod quectel_qlog tcpdump wifi_led.sh
ac diag_start.dat httpdiag mktemp radvd tee wifi_off_timer.sh
acl diff hwclock modprobe ramonitor telnet wlan
agetty dirname hwnat more readlink telnetd wlan_wps
arping dmesg hwnat-disable.sh mosquitto_pub reboot test xargs
ash dns hwnat-enable.sh mosquitto_sub redirect_console tftp yes
atcmd dnsdomainname id mount reg time zcat
ated dnsmasq ifconfig mpstat reset top zcmd
atftp drop_caches.sh init mtd_write restoredefault touch zebra
awk dropbear insmod mtr rilcmd tr zhttpd
basename dropbearkey ip mv rilcmd.sh traceroute zhttpput
bcm_erp.sh du ip6tables nc ripd traceroute6 zpublishcmd
blkid ebtables ip6tables-restore ndppd rm true zstun
brctl echo ip6tables-save netstat rmdir tty2tcp zsuptr69
btnd egrep ipcalc.sh nice rmmod tty_log_echo zsuptr69cmd
bunzip2 env iperf3 nslookup route ubiattach ztr369cmd
busybox esmd iptables ntpclient rs6 ubiblock ztr69
bzcat eth_mac iptables-restore ntpd scp ubicrc32 ztr69cli
cat ethwanctl iptables-save nuttcp sed ubidetach ztr69cmd
cfg expr iwconfig nvram self_check.sh ubiformat ztzu
chgrp ez-ipupdate iwlist nvram-factory sendarp ubimkvol zupnp
chmod false iwpriv obuspa seq ubinfo zupnp.sh
chown fdisk kill obuspa.sh setsmp.sh ubinize zybtnchk
chpasswd fgrep killall openssl sh ubirename zycfgfilter
chroot find klogd opkg sleep ubirmvol zycli
clear firstboot led.sh passwd smp.sh ubirsvol zyecho
cmp flash_mtd less pgrep snmpd ubiupdatevol zyecho_client
config.sh free ln pidof sort udhcpc zyledctl
conntrack fsync logger ping speedtest udpst zysh
conntrackd ftpget login ping6 ss umount zywifid
cp ftpput login.sh pings start-stop-daemon uname zywifid_run.sh
crond fuser logread pingsvrs switch uniq zywlctl
crontab fwwatcher logrotate pivot_root switch_root updatedd
curl genXML ls poweroff swversion uptime
cut getty lsmod pppoectl sync vcautohuntctl
dalcmd gpio lsof printf sys vconfig
dat2uci grep lte_srv_diag ps sysctl vi
Interfaces
The interface configuration looks like this:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ifb0: <BROADCAST,NOARP> mtu 2048 qdisc noop state DOWN group default qlen 32
link/ether 2e:c1:bf:ac:dd:3a brd ff:ff:ff:ff:ff:ff
3: ifb1: <BROADCAST,NOARP> mtu 2048 qdisc noop state DOWN group default qlen 32
link/ether c6:b6:cb:33:53:e7 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1 state UNKNOWN group default qlen 1000
link/ether 4c:c5:3e:a3:79:e2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4ec5:3eff:fea3:79e2/64 scope link
valid_lft forever preferred_lft forever
5: usb0: <NOARP,UP,LOWER_UP> mtu 2048 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::50:f4ff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
6: wwan0: <NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::50:f4ff:fe00:0/64 scope link
valid_lft forever preferred_lft forever
7: wwan1: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
8: wwan2: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
9: wwan3: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
10: wwan4: <NOARP> mtu 2048 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
11: wwan5: <NOARP> mtu 2048 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
12: wwan6: <NOARP> mtu 2048 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
13: wwan7: <NOARP> mtu 2048 qdisc noop state DOWN group default qlen 1000
link/ether 02:50:f4:00:00:00 brd ff:ff:ff:ff:ff:ff
14: teql0: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 100
link/void
15: ra0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether 4c:c5:3e:a3:79:e3 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4ec5:3eff:fea3:79e3/64 scope link
valid_lft forever preferred_lft forever
16: eth3: <BROADCAST,MULTICAST> mtu 2048 qdisc noop state DOWN group default qlen 1000
link/ether 00:0c:43:28:80:03 brd ff:ff:ff:ff:ff:ff
17: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4c:c5:3e:a3:79:e2 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::4ec5:3eff:fea3:79e2/64 scope link
valid_lft forever preferred_lft forever
18: apcli0: <BROADCAST,MULTICAST> mtu 2048 qdisc noop state DOWN group default qlen 1000
link/ether 4e:c5:3e:03:79:e3 brd ff:ff:ff:ff:ff:ff
19: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4c:c5:3e:a3:79:e2 brd ff:ff:ff:ff:ff:ff
inet 37.83.118.29/30 brd 37.83.118.31 scope global br1
valid_lft forever preferred_lft forever
inet6 fe80::4ec5:3eff:fea3:79e2/64 scope link
valid_lft forever preferred_lft forever
And since there are also bridges, here is the bridge configuration.
root@NR7101:/usr/sbin# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.4cc53ea379e2 no ra0
br1 8000.4cc53ea379e2 no eth2
The important parts here are:
- wwan0: The first WWAN interface.
- eth2: The Ethernet link that we are using to connect to the device. This is connected to br1.
- br1: The bridge interface over which we access the device via eth2.
- br0: The bridge for the wireless LAN network of the device.
- ra0: the actual wireless interface that is connected to br0.
Routing Table
The routing table is also rather straight forward:
root@NR7101:/usr/sbin# ip route
default dev wwan0 scope link src 37.83.118.29
37.83.118.28/30 dev br1 proto kernel scope link src 37.83.118.29
127.0.0.0/16 dev lo scope link
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.1
239.0.0.0/8 dev br0 scope link
First we have the default route as a onlink-route. Then the local networks on the bridges br1 and br0 and the loopback interface. And somehow it also adds a multicast network to the bridge for the wireless networks.
Connection Tracking
Regarding connection tracking there is conntrack running with support for 20480 connections and 4096 buckets for the hash-table.
The other settings for conntrack appear to be reasonable at first glance:
root@NR7101:/usr/sbin# sysctl -a | grep net.netfilter.nf_conntrack
sysctl: error reading key 'net.ipv4.route.flush': Permission denied
sysctl: error reading key 'net.ipv6.route.flush': Permission denied
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 4096
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 17
net.netfilter.nf_conntrack_expect_max = 60
net.netfilter.nf_conntrack_frag6_high_thresh = 4194304
net.netfilter.nf_conntrack_frag6_low_thresh = 3145728
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 20480
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
The conntrack tool is also available.
conntrack -L lists the tracked sessions while conntrack -C returns the
amount of tracked sessions.
IPTables
The router is running iptables as a firewall. The firewall is very lengthy, a dump of the rules can be found in a GitHub repository.
In general there are a lot of tables that are empty by default. In IP-Passthrough mode the FORWARD chain simply accepts everything. The INPUT chain allows the protocols defined for remote management and everything that has a RELATED/ESTABLISHED state.
Benchmarks
Bandwidth-Test
We will start with simple bandwidth test to get an estimate for the maximum bandwidth available.
For this we are running iperf3 for 60 seconds, once with the client as sender, receiver and one in bidirectional mode.
We repeat the test with 10 parallel sessions to check, that that we are not limited by a single flow.
The throughput depends a lot on the signal strength and the general usage of the cell we are in. During daytime the tests were in the range of 60-120Mbit/s download speed, however during the early morning hours on a Sunday we were able to achieve the 350Mbit/s that are advertised by the ISP in the area we tested in.
During this test the CPU utilization never got beyond 10%.
new-sessions per second test
To test how many new sessions the device can handle we generate them via trafgen and this package description:
{
# --- ethernet header ---
eth(sa=aa:bb:cc:dd:ee:ff, da=gg:hh:ii:jj:kk:ll)
# --- ip header ---
ipv4(id=drnd(), ttl=64, sa=A.B.C.D, da=E.F.G.H)
# --- UDP header ---
udp(sport=48054, dport=dinc(10000, 50000), csum=0)
# payload
'A', fill(0x41, 11),
}
The MAC and IP addresses have to be adjusted accordingly. This generates a new session by increasing the UDP destination port. The test is then executed with the following command.
trafgen -o enp0s31f6: -i packet.dsc -P 4 -b 300pps
In this case we are testing with 300 packets (which is equivalent to 300 sessions in this case).
Somewhere between 250 and 300 new sessions per second the Zyxel device starts sending Pause-frames and has one of the four cores completely utilized with interrupt handling. This is due to all interrupts being handled on one core.
root@NR7101:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
3: 0 2115968 0 0 MIPS GIC eth2
Since irqbalance is not installed we can't easily redistribute these interrupts.
Reducing the amount of allowed conntrack sessions
When we reduce the amount of conntrack sessions to something very low we can test what happens when the conntrack table is full.
We set it to 80 with echo "80" > /proc/sys/net/netfilter/nf_conntrack_max.
Then we ran a iperf test with 100 sessions in parallel.
iperf could not open all of these sessions, because the conntrack session table
was full and the router started dropping packets and informed us in dmesg:
...
[14008.628000] nf_conntrack: table full, dropping packet
[14009.636000] nf_conntrack: table full, dropping packet
[14011.904000] nf_conntrack: table full, dropping packet
[14012.916000] nf_conntrack: table full, dropping packet
...
There are now stateful matches in the iptables rules for the path these packets take, but they are dropped regardless.
disabling conntrack for sessions that don't go through the device
We tried to disable connection tracking for all flows that only go through the
device, however the firmware that is running on the device does not allow for
that. The NOTRACK target and the --notrack option for the CT
target are unknown to the device:
root@NR7101:/# iptables -t raw -A PREROUTING -d 1.2.3.4 -j CT --notrack
iptables v1.4.16.3: unknown option "--notrack"
Try `iptables -h' or 'iptables --help' for more information.
root@NR7101:/# iptables -t raw -A PREROUTING -d 1.2.3.4 -j NOTRACK
iptables v1.4.16.3: Couldn't find target `NOTRACK'
Try `iptables -h' or 'iptables --help' for more information.
So disabling connection tracking is not easily possible.
Increasing the amount of conntrack sessions
We can adjust the maximum amount of conntrack sessions and the conntrack bucket size, if we run into the packet-drop problem.
echo "40960" > /proc/sys/net/netfilter/nf_conntrack_max
echo "8192" > /sys/module/nf_conntrack/parameters/hashsize
For a detailed discussion of those values take a look this Conntrack Parameter Tuning wiki page.
Conclusion
The NR7101 has some limitations regarding the amount of sessions it can handle.
There is no direct solution available, however increasing the nf_conntrack_max
can help a bit.
There is no easy workaround for the limited amount of new sessions per second.
An other option could be to use the OpenWrt firmware instead.
Remarks regarding the NR7102
The NR7102 comes from the same series of devices as the NR7101. From the outside and the Web UI this device looks very similar to the NR7101. However the method to gain a root password does not work here. So we can't take a deeper look into this device.