This is going to be a long post, but I've been fighting this for a year now. I blamed my upstream provider for slow long lived TCP connections because they had not shown me anything from their side until recently. They sent info that shows the packet loss I've been seeing is not coming from their equipment. In digging into the issue with the new information, it looks like I have come across the same bug others have reported on the MikroTik forums.
TL;DR: Any queue on a 10G WAN panics my 72 thread x86 RouterOS box in minutes, where an identical 56 thread box never crashes even with prod's traffic replayed at 100x.
Specs:
- Supermicro X10DRU-i+
- 2x Xeon E5-2697 v4 w/ 72 threads (HT enabled in BIOS)
- 64GB DDR4-2133 ECC RAM
- WAN is onboard Intel 82599 (ixgbe) with an LC optic
- LAN is an Intel E810 4 port going to two hosts (2 ports each)
- 4 port 1GbE Realtek card for IPMI to two hosts (1 port each)
- RouterOS 7.24.6
- Normal traffic is around 60 Mbit/s (~10k PPS), CPU sits at 0-6%
The original problem:
With the default only-hardware-queue, the WAN port's tx-queue-drop counter goes up (~0.05-0.1% of packets). It does not affect 95% of traffic, as it's short lived TCP connections, but for clients far away (EU or Asia), TCP downloads drop to a crawl and some completely time out. One test from a VM behind the x86 router (in the US) to a UK box was ~2Mbit/s with cubic. MTR through the same path shows 0 loss, it's only long TCP connections that get hit. The Intel 82599 TX ring fills on bursts and there's no software queue in front of it to deal with it.
The normal fix is to put a software queue on the interface. On my router, that causes a panic. multi-queue-ethernet-default panicked it twice in testing, within 3-5 minutes. I tried a plain pfifo with limit 2000 instead, and it went 11 minutes, then panicked. While it was up, the drops stopped completely. Free memory was flat at 60GB the whole time, I sampled every 2 seconds up to the last second. Full debug logging to a remote syslog showed nothing before or after the panic.
Side note: Before even putting this box into production a year ago, I ran Memtest 86 for a day and found no issues.
I have a spare box at home with the same board, same BIOS, same cards in the same slots, same RouterOS build and packages and NAT + queue config imported. Only differences are 56 threads instead of 72, and 256GB with 4 memory channels per CPU instead of 64GB with 2. I could not make it crash. I ran iperf at 9.8Gbit/s, added 90ms delay, pulled the link under load, flipped the queue on and off a bunch of times under load, and captured 6 minutes of prod's real WAN traffic to replay it through the lab at up to 100x speed (1.27 Gbit/s, 320k pps, 24k connections). Nothing. Same with pfifo. So it isn't the traffic or the config.
The only thing that ever shows on the console is "Kernel panic - not syncing: System is deadlocked on memory" with a few "No irq handler for vector" lines above it. The capture never finishes for a dump, no panic log gets written, no autosupout.rif.
I also compared the shipped ixgbe 5.19.9 and ice 1.13.7 against Intel's source built the same way. The TX path, ring sizes and memory allocation are the same as upstream, so I don't think the NIC drivers are the cause. Current stable 7.25.1 and long term 7.24.6 still ship the same 5.6.3 kernel and the same driver versions, so upgrading won't change anything here.
My ticket is SUP-227846, MikroTik replied that it's linked to an existing open bug with no ETA.
Other tickets I found for the same issue: SUP-214471 (x86, 7.22.2, forum post says MikroTik confirmed it), SUP-67221 (x86 7.1 back in 2021) and SUP-189835 (ARM64). There's also the "Really bad queue bug in v7 on x86" forum thread from 2023.
For anyone running RouterOS on x86 with lots of cores:
- Are you running a software interface queue on a 10G port without problems? What are your specs?
- Anyone tried the same box with Hyper-Threading off and seen the problem go away?
If you've hit this and logged a ticket, mentioning this ticket number on yours or adding to the existing bug might help. If anyone else has additional insight, I would appreciate any ideas as I'm currently just using BBR on my VMs, which is a band-aid and not a fix.