The Center Cannot Hold: Porting RSS to a Router That Hangs at Boot

#

Things fall apart; the centre cannot hold; Mere anarchy is loosed upon the world, — W. B. Yeats, The Second Coming

This is the story of a single kernel driver, ported by hand onto a stable tree, that builds perfectly and then refuses to boot. Not a compile error, not a link error — a hard hang, forty seconds deep in module loading, with not a single line of driver output before the machine goes catatonic. And the fix wasn’t where I first looked.

It’s also the story of why you desk-check a kernel module’s probe path in the serial console before you trust the patch that “builds clean.”

TL;DR

#

• The MediaTek MT7988 ethernet MAC in upstream Linux is single-NAPI: one poll, one ring, one CPU. The bridged/routing path is CPU-bound around ~1.6 Gbit/s.
• frank-w’s RSS series spreads flows across four PDMA RX rings and measured ~7.3 Gbit/s upstream-internal (a forum report even quotes 4.5 → 8.4 with RSS). But it was written against his own heavily-diverged tree and does not apply to 6.18.44.
• I hand-ported the multi-ring NAPI + RSS onto the vanilla 6.18.44 single-NAPI driver. It compiled clean and produced a loadable module (=m) so the driver could be swapped out for a known-good copy if things went sideways.
• Why it’s worth it: per-ring NAPI + per-ring IRQ + per-ring DIM turn the CPU-bound ~1.6 Gbit/s single-NAPI RX path into a four-queue, four-CPU pipeline — the difference between a box that’s the bottleneck on a multi-gig WAN/LAN bridge and one that’s incidental to it. frank-w measured ~7.3 Gbit/s; a forum A/B saw 4.5 → 8.4 with RSS on the same silicon.
• First TFTP boot test into the recovery image: hard hang inside mtk_eth’s init_module — no “frame engine” print, then five consecutive 60-second RCU stalls on CPU 1, then the watchdog reset. NAND untouched (the whole point of booting recovery over TFTP).
• My first suspect (a rewritten DIM delay-IRQ routine) was ruled out by a second boot after reverting it — it hung identically.
• The instrumented boot tagged every probe step with pr_err markers. The last marker to print before the freeze pointed at a two-line window… and “the fix didn’t hold” because I’d blamed the wrong lines.

Status: still hanging, exactly localized, fix under construction. Every boot is a single, safe, reproducible, NAND-free experiment.

The Setup

#

• SoC: MediaTek MT7988 (BPI-R4 Pro 8X), 4-core, PCIe Gen3
• Kernel: OpenWrt on 6.18.44, our own branch cut from bpi-r4pro-8x-v2
• Network stack: Mediatek mtk_eth_soc — upstream single-NAPI RX (one rx_napi, one poll, ring 0 drained by the IRQ handler)
• The port: frank-w’s RSS series, 6.18-main / BPI-Router-Linux release lineage
• Flashing: out-of-band — TFTP boot the -initramfs-recovery.itb from U-Boot menu “Boot system via TFTP”, zero writes to NAND
• Console: USB-UART serial into minicom, everything captured to a log file

Act I: The 1.6 Gbit Ceiling

#

The BPI-R4 Pro 8X has eight 2.5/10G ports behind an MxL86252 switch. The switch is more than fast enough. The problem is what happens to a packet once it reaches the SoC’s ethernet block.

In upstream Linux, the MT7988’s RX is single-NAPI: the PDMA hardware writes every flow into one ring, a single NAPI polls it, and one of the four CPU cores does all the driver/flow/packet work. That serializes the hot path. Over and above the silicon’s own classification (PPE flow offload already joins flows), the remaining traffic that has to go up the stack — bridge-local routing, non-offloadable flows, anything the PPE table misses — all funnels through one poller on one core.

Measured result: ~1.6 Gbit/s on the paths that matter, on a box with eight multi-gig ports. The CPU is the wall.

RSS (“Receive Side Scaling”) is the standard answer: hash each flow (Toeplitz on the tuple), use a 128-entry indirection table to spread flows onto up to four RX rings, give each ring its own NAPI and its own PDMA IRQ, and let four cores chew in parallel. That is the entire architecture of the fix, and it’s why the throughput headline numbers exist.

What we’re actually buying, and by which mechanism

#

Before walking through the port, it’s worth being precise about why this helps and where the cycles actually go. The single-NAPI design’s ceiling isn’t some arbitrary software cap — it’s the arithmetic of one CPU doing everything, serialized:

1. One ring, one poller, one CPU — the serialization wall. In the vanilla driver, the PDMA writes every received descriptor into ring 0. The lone IRQ handler schedules the lone NAPI; the lone NAPI runs mtk_poll_rx, walking descriptors, allocating pages, delivering skbs up the stack, then re-arming the ring. Every packet, even from wildly different flows between different ports, traverses that exact single-threaded path. The ethernet block is literally multi-port capable — eight of them — but the driver makes it a one-lane road. At ~1.6 Gbit/s the CPU is saturated running the poll loop itself, not doing anything user-meaningful. That’s the whole problem in one sentence.

2. NAPI per-queue ≠ just “more workers”. The subtlety people miss: NAPI isn’t just a loop, it’s a deferral and batching contract with the IRQ and the softirq path. Putting four rings behind one NAPI still serializes them — the poll function is still one function on one CPU. You don’t get parallelism by adding rings alone; you get it by giving each ring its own NAPI instance, bound to its own IRQ, so each can be scheduled independently onto (potentially) a different CPU. That’s what the struct mtk_napi rx_napi[] array buys: four independent scheduling domains instead of one.

3. The indirection table is the load balancer. The RSS IP in the PDMA computes a Toeplitz hash over the packet tuple and looks the result up in the 128-entry indirection table, whose entries say “put this flow on ring N”. We populate it round-robin (i % rss_num) so flows land on rings 0-3. The hardware then writes each flow’s descriptors to its assigned ring. This is real affinity, done in silicon, zero per-packet software cost — the single most important thing RSS adds versus a software hash-to-CPU scheme: the classification is free.

4. Per-ring interrupts, one IRQ stream per ring. Each ring’s done-bit is routed to its own PDMA IRQ (pdma0..pdma3, already in the device tree), which schedules that ring’s NAPI. mtk_handle_irq_rx becomes per-ring aware: it checks its ring’s done bit, disables just that bit, schedules that NAPI. Four rings, four IRQ→NAPI→poll→rearm chains, each delivered as a separate interrupt source the IRQ core can dispatch to different CPUs.

5. Interrupt coalescing (DIM) applies per ring too. The driver forgets nothing here — the deferred-interrupt / dynamic-interrupt-moderation machinery (mtk_dim_rx/mtk_dim_tx) is expected to run per ring so each queue can use its own burst profile instead of sharing one global coalescing knob. (More on how that specific part tried to sabotage the whole port shortly.)

6. The concrete, measurable payoff. The headline number frank-w measured is ~7.3 Gbit/s on the MT7988 (4-6 Gbit/s practical for real office/gateway mixes), versus the ~1.6 Gbit/s single-NAPI wall. A forum user’s A/B report — literally RSS on vs. off on the same silicon — showed 4.5 → 8.4 Gbit/s. In a router, that’s the difference between the box being the bottleneck on a multi-gig WAN/LAN bridge and being completely incidental to it. The UDP µs-latency, the number of concurrent flows a NAT/firewall can track, the ability to run WireGuard + filtering + full-bridge at line rate simultaneously — all of it rides on having >1 core able to service RX.

Why the architecture matters more than the number

#

It’s tempting to file RSS under “performance feature.” Two reasons it’s structural here:

• The 8-port box is oversubscribed by design. Eight 2.5/10G ports behind one switch, to a SoC with one ethernet MAC — but the MAC itself has four RX queues available in silicon. Leaving three of them idle while one core is pegged at 100% is leaving hardware-purchased capability on the table, not a tuning shortcut.
• Router workloads aren’t one big TCP flow. The headline 7.3 Gbit/s is really the flow-concurrency win: a firewall/NAT doing thousands of flows, or several multi-gig devices simultaneously, benefits from four queues far more than a single honking iperf. RSS is what makes “my router can actually route” rather than “my router can saturate one core.”

Upstream 6.18.44 has none of it. So this is a hand port, not a “pick and apply.”

Act II: Why the Patch Won’t Apply

#

frank-w’s RSS/LRO series is real and works on his tree. It also touches a mtk_eth_soc.c that has drifted considerably from vanilla — ring-ownership, mtk_update_rx_cpu_idx, the mtk_hwlro renames, whole RX-path reshapes.

Trying to feed the RSS patch onto 6.18.44:

14 out of 57 hunks FAILED
1 out of 17 hunks FAILED

The register-definitions patch applied clean. The core RSS patch didn’t — not because of syntax, but because the code around it doesn’t exist in vanilla 6.18.44. It wasn’t a mechanical pick; it was a port.

The desk-check that matters

#

Before writing any code, I verified the one external dependency that would make or break a per-ring-IRQ design: the device tree already has the named interrupts. In vanilla 6.18.44, the MT7988 ethernet node carries:

interrupt-names = "fe0", "fe1", "fe2", "fe3", "pdma0", "pdma1", "pdma2", "pdma3";

So the four per-ring PDMA IRQs the port needs are already wired in the board’s DTS, even though upstream’s driver only consumes fe0..fe3. The hardware was designed for this; the driver just never grew the code path. That was green light.

The port, in one breath

#

• Swap struct napi_struct rx_napi → struct mtk_napi rx_napi[] (one NAPI per RX ring, each bound to its ring).
• Make MTK_RX_DONE_INT a function of ring number (BIT(24+ring) on netsys-v3, BIT(30)/BIT(24+ring) on older).
• Wire mtk_get_irqs_pdma() to fetch pdma0..pdma3, request each with IRQF_SHARED, dev_id = &rx_napi[i].
• mtk_napi_rx acks the owning ring’s done bit; mtk_poll_rx drains only rx_napi->rx_ring and advances that ring’s CPU index.
• RSS core: 40-byte Toeplitz key, 128-entry indirection table (i % rss_num), PSE ring mode, int-group routing, and ethtool rxfh get/set.
• Fix the 6.18 drift: upstream uses desc_shift not desc_size; add reg_map locals where the base hadn’t needed them.

The mechanism, traced end to end

#

If a bored packet flows through each piece, the chain is:

packet from switch → PSE (packet switch engine) → PDMA
   → RSS: Toeplitz hash(tuple) = h
   → indirection_table[h & 127]  = ring ∈ {0,1,2,3}
   → descriptors written to that ring
   → ring's RX-done bit asserts in PDMA IRQ status
   → routed to its int-group / PDMA IRQ (pdma0..pdma3)
   → mtk_handle_irq_rx(dev_id=&rx_napi[ring]) → schedule THIS ring's NAPI
   → mtk_napi_rx: ack MTK_RX_DONE_INT(ring), mtk_poll_rx drains ONLY rx_ring[ring]
   → mtk_update_rx_cpu_idx(eth, ring): publish this ring's index back to HW

Each of those is a distinct, deliberate mechanism, and the port has to get all of them right or the ring it drained silently isn’t the ring hardware wrote to:

• Per-ring NAPI binding — mtk_napi_init assigns each rx_napi[i] its eth and its &rx_ring[i]. If this is wrong, the NAPI for ring 3 could be draining ring 2 — and the packets stay stuck.
• Per-ring IRQ ownership — mtk_get_irqs_pdma reads pdma0..3 from DT; each is requested with IRQF_SHARED and dev_id = &rx_napi[i], so the handler for pdma2 knows it owns ring 2’s NAPI. No global “which ring had data?” scan needed — the ring is implied by the IRQ that fired.
• Per-ring done-bit semantics — the ack in mtk_napi_rx must be MTK_RX_DONE_INT(ring_no), not a global mask, otherwise you ack ring 3’s interrupt while ring 1 still has work — and the PHY never warns you about the hung queue.
• Per-ring CPU index publication — mtk_update_rx_cpu_idx(eth, ring) writes that ring’s calc_idx to that ring’s crx_idx_reg (the MTK_QRX_OFFSET-spaced registers). Get this wrong and the hardware thinks you’re done with ring 0 when you’ve actually been freeing ring 3 → descriptor reuse → corruption.
• RSS enablement in the PDMA — mtk_rss_init writes the 40-byte hash key, programs the 128-entry indirection table, sets the hash-type bits, and does the “pause → enable → release” dance so the classification becomes active in hardware. Only then does the chip actually spread flows.
• IRQ routing to groups — rings 1/2/3’s done bits are routed into their interrupt groups (int_grp, int_grp+4, int_grp3) so the GIC can deliver them as separate IRQ lines, and mtk_hw_init reprograms the FE interrupt grouping (MTK_FE_INT_GRP) to tell the frame engine where the PDMA interrupts live in the new multi-ring world.

Every one of those is a “if you get it wrong, silently, the box stops forwarding but still looks alive” — which is precisely the class of bug this whole investigation is chasing: it doesn’t fail loudly, it fails as a hang or as “one port’s traffic sits in a ring nobody reads.”

It compiled. Clean. mtk_eth.ko linked; RSS symbols were in the binary; vmlinux no longer bundles the driver.

But nothing, at this point, had executed on real silicon. So I made the sensible, defensive choice:

Build it as a module — so I can un-build it

#

Instead of baking RSS into the kernel image, CONFIG_NET_MEDIATEK_SOC=m. Now the whole driver (including its WED offload parts) is one loadable mtk_eth.ko. I kept a known-good v2 module on the TFTP server: if RSS broke the box, a serial-console rmmod mtk_eth; insmod <v2.ko> would be a five-second rollback instead of a reflash.

Why the module indirection buys this: with the driver built-in, a broken mtk_probe is part of the kernel image — it executes before userspace, there is no “unload it”, and the only way out of a hang is reflashing. As a module, mtk_eth.ko loads on demand from /etc/modules.d/; the kernel itself comes up clean either way, so a hung driver leaves a live kernel you can pull the module out from under. Vermagic (6.18.44 SMP mod_unload aarch64) is identical across both builds, so the v2 .ko is a drop-in on the RSS kernel. That turns a “bricked router” recovery into a copy-a-file + two commands operation:

rmmod mtk_eth
insmod /tmp/mtk_eth_v2.ko        # known-good, 0 RSS symbols

There’s a cost (the driver comes up a few seconds later at boot, and there’s an autoload ordering dance with the DSA/MxL stack), but for an experimental branch it’s the single best insurance policy in the box: a driver you can swap is a driver you can afford to break.

Then I booted it. Over TFTP. Into the recovery initramfs. The whole point of recovery-over-TFTP is that nothing touches NAND — if it hangs, you power-cycle and you’re exactly where you started, having learned something.

Act III: First Boot — Nothing

#

The recovery image booted, userspace came up, kmodloader started pulling /etc/modules.d/*… and then silence. No driver message. Not even “mediatek frame engine”. Just a frozen console, and ~60 seconds later, the first of several RCU stall dumps:

[   12.418832] Loading modules backported from Linux version v7.2-0-g8d3ae59288f1
[   72.603432] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   72.609363] rcu:   0-...0: (1 GPs behind) idle=09bc/... softirq=329/330

A hard hang in a driver’s module init, with interrupts wedged on a CPU, is one of the least informative failure modes in kernel development: no backtrace you can read (the NMI dump only shows the healthy idlers), no log, no “where did it stop?” — since the module prints nothing between calling init_module and catastrophe.

But note that beautiful early line in the boot:

Cannot find device "eth0"

That appears at preinit, several seconds before the module is even loaded. It looked like a symptom. It was, in fact, a red herring — and proving that was the first win.

Red herring: Cannot find device "eth0"

#

OpenWrt’s lib/preinit/05_set_preinit_iface runs before /etc/modules.d/* loads. The BPI-R4 Pro 8X isn’t in its per-board list, so it falls through to the default ip link set eth0 up; ifname=lan1. In a built-in build, eth0 exists by ~3.9s (kernel probes it during driver init), long before preinit → no error. In a module build, eth0 doesn’t exist until ~18s, after preinit → ip: Cannot find device "eth0".

It’s benign. netifd config comes from board.d (lan1..lan6), not eth0. The default-iface fails gracefully. The error is only a marker that the driver is a module instead of built-in — which is exactly what we intended.

Ruling the red herring out matters, because otherwise you’d chase a boot-ordering “fix” (move mtk_eth to modules-boot.d) that would actually cause the real problem: probing the eth MAC before its DSA/MxL/phy dependents are loaded. Wrong layer.

The tool that pinned it: initcall_debug

#

The initramfs boots with a modified command line, and that’s when the log becomes readable:

setenv bootargs 'console=ttyS0,115200n1 loglevel=8 initcall_debug'
run boot_tftp

initcall_debug prints every calling initcall ... @ pid / initcall ... returned line. The hang became a one-liner:

[   18.473135] calling  init_module+0x0/0xfdc [mtk_eth] @ 1131
[   78.635968] rcu: INFO: rcu_sched detected stalls ...

mtk_eth’s module init was called at 18.47s and never returned — no initcall ... returned, no driver printk, then five consecutive 60-second RCU stalls on CPU 1 before the watchdog finally reset. The DSA/MxL switch never even got to load.

module_init → platform_driver_register → the DT node exists → really_probe → mtk_probe runs synchronously and hangs before mtk_add_mac (which is why there’s no “frame engine” line). The RSS port touched exactly these probe-time steps: mtk_get_irqs_pdma, per-ring request_irq, mtk_napi_init, and the mtk_hw_init FE-int-group changes.

Now: which one?

Act IV: The Marker Wave

#

I have one expensive resource (a physical board, boots that take four minutes of hang each) and one cheap one (recompile, wall-clock ordered, background). So the strategy was: maximize information per boot, don’t care about minutes. Add a pr_err marker at every probe step, rebuild once, let one boot tell me everything.

pr_err specifically: it’s KERN_ERR, level 3, which prints to the serial console regardless of loglevel. When a CPU is mid-hang with interrupts off, the serial UART is the one thing that still drains — so the last marker that appears is the next statement that never executed. That’s the mind-marker contract.

First pass, ~31 markers covering every probe step. Result:

[   16.793304] MTKDBG: hw_reset -> CHK_IDLE_EN done      ← last printed
[   16.797914] MTKDBG: probe request_irq block DONE       ← misnamed; actually "after reset"
              ... hang ... RCU stall at 78.7

The hang is in a ~4-line window of mtk_hw_init between “after reset” and “FE int grouping”: v3 FE_GLO_MISC read/write, pctl GPIO, the MAC-MCR loop, CDMQ, the DIM calls, irq-disable.

The false prime suspect

#

The only RSS-specific code in that window was the rewritten mtk_dim_rx/mtk_dim_tx. Upstream writes interrupt-delay coalescing to a single pdma.delay_irq (0x6a0c). My port split it into netsys-v3 tx_delay_irq (0x6ab0) / rx_delay_irq (0x6ac0) with a val << 16 “ring” bit duplication. At probe time, before any IRQ handler is registered, writing a garbage-derived value into an IRQ configuration register could trip an unhandled interrupt → CPU 1 spin → RCU stall. It fit perfectly.

I reverted the DIM functions to pristine — single 0x6a0c write, no split, no ring bits — rebuilt, and booted again.

THE FIX DIDN’T HOLD. Same exact last marker, same hang, same stall on CPU 1. Disassembly of the rebuilt module confirmed mtk_dim_rx was doing a single register write. So the DIM rewrite was a red herring all along — cleanly ruled out by a second 4-minute boot, which is exactly what cheap NAND-free boots are for. This was the moment the Yeats line both titled and described this post: I’d believed the center (the “obvious” RSS-only change) held the weight, and it didn’t.

Act V: Where I Actually Stand

#

Marker set now: 36 markers, fine-grained, bracketing every statement between the reset and the FE int-grouping:

#8  probe request_irq block DONE
#9  after FE_GLO_MISC
#10 after pctl
#11 after CDMQ
#12 after DIM calls       ← DIM confirmed not the cause (reverted, still hangs)
#13 after irq_disable

The instrumented image is restaged on TFTP. The next boot will show exactly which statement never completes — I expect after FE_GLO_MISC to be the last one, pointing at the FE-domain register read/write that happens immediately after mtk_hw_reset (and which, in the previous boot, that 0x6f8ff CHK_IDLE_EN reset write is the sole new-adjacent step to). But I’ve been wrong once this evening, so I’m letting silicon tell me rather than my theories.

Everything is committed, every boot is reproducible, and the physical risk per boot is zero.

What “success” will look like, mechanically

#

Once the probe stops hanging, the payoff is honestly checkable — not from a benchmark license, but from the same four mechanisms in reverse:

1. ethtool -x must show 128 indirection-table entries mapping across rings 0-3 (not all-0), and ethtool -n ... rx-flow-hash tcp4 should report Toeplitz — proof the classification is live in the PDMA.
2. /proc/interrupts should show pdma0..pdma3 each incrementing on a different IRQ (and, with future IRQ-affinity tweaks, landing on different CPUs) — proof the per-ring IRQ→NAPI chains are real, not decorative.
3. iperf3 reverse (-R) with -P4 on the 8X is the throughput headline: single-NAPI-wall ~1.6 Gbit/s should give way to multi-Gbit, staying CPU-under-utilized instead of pegging one core.
4. The latency signature that exposed the single-NAPI wall last time (hundreds-of-ms unicast RTT while broadcast flows fine) should be gone — proof the flow-hash→ring→CPU path actually drains, rather than stalling.

That last one is worth flagging: the symptom class a failed RSS port produces — flows hashed to a ring nobody drains, broadcast/ICMP fine but unicast crawling — is precisely the kind of “looks alive but isn’t forwarding” bug I mentioned in the mechanism trace. If the driver ever boots but traffic dies, the indirection table is pointing at a ring with no reader. Heads-up for whenever we get past the probe hang.

What I Learned (so far)

#

1. A hang that prints nothing is still findable — initcall_debug turns “module init never returned” into a one-line smoking gun, and pr_err markers flush to serial even with a CPU pinned.
2. Rule out the red herring before you fix it. Cannot find device "eth0" at preinit is the =m-build footnote, not the failure; “fixing” boot order would have broken the real boot.
3. Make the driver a module if you’re going to break it. Swappable driver = affordable experiment. This whole investigation ran with a rmmod-able escape hatch and never risked the NAND.
4. “Fix didn’t hold” is data, not failure. The second, same-hang boot after reverting DIM eliminated an entire hypothesis class cheaply. A four-minute hang per hypothesis is a luxury budget only because recovery-over-TFTP makes each one free of consequence.
5. Belt-and-suspenders the probe path. When you touch probe-time code, expect to instrument probe-time code. The marker wave is not optional scaffolding; it’s the primary instrument.
6. Mechanism-first framing pays off. Knowing exactly which mechanism each touched line serves (ring binding vs. done-bit vs. CPU-index vs. IRQ routing) turned “is the patch even architecture-right?” into a concrete checklist of failure modes — and it’s why I can already predict what both the hang and a post-boot traffic stall would look like in the console.

the centre cannot hold — sometimes it never was the centre you were looking at.

Follow-up: next boot either names the hanging statement (and I fix the real register step), or it boots, and the DIM was somehow still implicated despite my revert — either way, one more free boot will tell. Full port details, module-swap rollback plan, and every error log are tracked in the project notes.