The Centre Held: From a Silent Boot to 9.9 Gbit/s and a Safer Jumbo Path

#

Things fall apart; the centre cannot hold; Mere anarchy is loosed upon the world, — W. B. Yeats, The Second Coming

Six days ago I published the story of a hand-ported RSS series that built clean and then refused to boot — init_module called, never returned, CPU pinned, five RCU stalls, watchdog reset. I ended it mid-investigation: the hang localised (by a wave of pr_err markers) to a two-line window, a false suspect (the DIM rewrite) ruled out, the next boot ready to name its sentence.

This is the follow-up. It is a much longer journey than “and then it worked,” because it did work — and then the box led me into a fresh rabbit hole that took the driver series from a single-NAPI CPU wall to 9.9 Gbit/s at 9K MTU, and finally back to a runtime kernel path that, until now, would panic the box on a single command. Spoiler: the hang that started all this wasn’t even the interesting bug.

TL;DR

#

• The localized hang turned out to be a module-build artifact: the exact frame-engine register read that hung at ~18 s probe in the =m build completes fine in a built-in =y build (value 0x8000c016). The real boot blocker for RSS was the ordering of NAPI registration on the shared dummy netdevice.
• With a device-tree bootconf fix (the TFTP boot path was silently leaving the WAN/SFP/power overlays off) and a one-hunk reorder in the driver (allocate eth->dummy_dev and call netif_napi_add() before register_netdev()), all four PDMA RX rings lit up — pdma0..3 all incrementing in /proc/interrupts.
• RSS then delivered what the silicon allows. Honest numbers: banana egress 9.4 Gbit/s; banana ingress is capped by a PDMA ingress wall (~4.2–4.9 Gbit/s) no matter how many rings or IRQ affinities you throw at it. RSS’s real gain is CPU spreading (four NAPI kthreads vs one pegged core), not raw single-NIC throughput.
• Jumbo escalation: MTU 2000 (+20–25 %) → full 9K (new MTK_NETSYS_RX_9K capability, dynamic rx_buf_len, PDMA SDL programming, page_pool off for jumbo RX). This killed a recurring skb_over_panic and took 4-stream TCP forward from 4.43 → 9.89 Gbit/s (+126 %) and UDP forward from 2.33 → 9.63 Gbit/s (+313 %).
• The 9K mystery that looked like our driver was actually a host NIC driver bug: the client’s Aquantia AQC113 “atlantic” driver never programs the TX max-frame register, so ip link set mtu 9000 changed the Linux MTU while the silicon silently refused to transmit frames >1518 bytes. The banana’s 9K stack was correct all along.
• Finally, the footgun behind the original crash — ip link set mtu on a running interface doesn’t resize the RX rings — got a proper fix: a dedicated light rx_buf_len_work worker (rings-only, no link flap, no full FE reset), replacing my earlier coarse reset-worker reuse. Flashed to production and verified live: 9216 -> 1536, then 1536 -> 9216, zero panics, zero carrier flaps, 8K ping and iperf3 at ~9.9 Gbit/s both directions afterwards.

Status: rework flashed to production (r195-eb8f58aedf), live-verified, CI green. Runtime MTU switching at 9K is now a non-event.

Act I: The Hang That Wasn’t the Bug

#

Last post’s cliffhanger: 36 fine-grain pr_err markers, hang narrowed to a single unreturned register read. The 43-marker build delivered the verdict:

[   18.759320] MTKDBG: probe request_irq block DONE
[   18.763925] MTKDBG: FE_GLO_MISC read try1          <- mtk_r32() invoked
[   78.765776] rcu: INFO: rcu_sched detected stalls ... CPU 0

mtk_r32(eth, MTK_FE_GLO_MISC) — a frame-engine-domain register read, ~24 µs after mtk_hw_reset released the FE — never returns. The AXI bus hangs, a CPU wedges, RCU stalls, watchdog resets. ethsys_reset itself finished cleanly (ASSERT/DEASSERT/DONE markers all printed). Everything pointed at the FE domain not being settled or clocked at probe time.

Then the story bent. I had prepared a built-in (=y) build to cross-check the module-vs-builtin hypothesis — and it booted:

MTKDBG: FE_GLO_MISC read try1 OK (0x8000c016)

Same code, same register, same memory map — but probe ran at ~8.7 s during kernel init instead of ~18 s from /etc/modules.d. The hang was real, but it was a symptom of module-context probe timing, not of the RSS port. The proof that it wasn’t porting logic was already sitting in a diff table: mtk_hw_init, mtk_hw_reset, ethsys_reset in my tree are byte-identical (modulo the markers) to frank-w’s tree that boots RSS on this same board.

But the built-in build had its own, nastier problem: eth2 didn’t exist. Probe walked only two add_mac calls; the 8X DTS has three mediatek,eth-mac children. mac@2 — the MxL switch / combo-port MAC, the one RSS actually matters on — was not-okay at probe time.

Two failures, one shared root: probe timing versus the boot sequence

#

• Module build (=m): probe at ~18 s. The FE-domain read hangs the bus.
• Built-in build (=y): probe at ~8.7 s, right in the middle of the FIT boot applying DT overlays. mac@2 isn’t okay yet → skipped → no eth2, no lan1–6, no 10G path to test RSS against.

The second failure has a clean, satisfying fix, and it isn’t in the driver at all. The device carried a saved U-Boot environment (env.txt, Sep 3) that predated bootconf_extra. The production boot path applies the combo overlays through that variable — the recovery/TFTP boot path did not. So every TFTP boot silently used only the base fdt-1: no WAN PHY, no SFP overlays, no combo ports.

boot_tftp=tftpboot $loadaddr $bootfile && bootm $loadaddr#$bootconf#$bootconf_emmc#$bootconf_extra

fw_setenv boot_tftp, and suddenly every TFTP boot nails all six overlays in order (-8x → -emmc → -cn13 → -cn14 → -8x-lan-phy → -8x-wan-phy). eth2 registers, four RX rings allocate, the MxL86252 switch comes up with its firmware banner, and ethtool -x eth2 shows a round-robin indirection table instead of “operation not supported.”

The actual RX fix was one small reorder

#

There was still a maddening “frames counted at the MAC, never delivered to the stack” period — tcpdump -i lan6 showing zero packets while ethtool -S eth2 counted RX packets climbing. Ring, IRQ, and NAPI code all byte-equivalent to the reference tree. That is where the attention goes; it was not where the fix was.

The fix was a one-hunk reorder (patch 980): allocate eth->dummy_dev and call netif_napi_add() for tx + rx[0] + RSS rings before register_netdev(). With threaded NAPI (dummy_dev->threaded = 1), RX work runs on the dummy netdevice’s kthreads. In the old order, the instant netifd saw a netdev registered it called ndo_open, which hit mtk_dma_init() -> __xdp_rxq_info_reg() with eth->dummy_dev == NULL → WARNING: ... Missing net_device from driver → -ENODEV, wedging the shared RX path while eth0/eth1/eth2 all still looked up. The per-ring NAPIs never registered, so the four RSS rings never drained.

After that one reorder — all four PDMA IRQs fire, ethtool -x reports Toeplitz on, and RSS works. The meatiest bug of the whole investigation was a registration-order race, not RSS.

Act II: Honest Throughput Accounting

#

The RSS headline arrived fast: iperf3 -R -P4 → 9.4 Gbit/s. Then I re-ran it properly and got the part nobody likes to write: numbers are directional.

Banana egress is line rate; banana ingress tops out around 4.5–4.7 G. I chased this through everything: IRQ affinity (spreading IRQs 106–109 across CPUs 0–3 → no gain), NAPI kthread affinity, per-ring counters, switch MIBs. All four RSS IRQs fire and distribute; the new NAPI kthreads run at 5–16 % CPU each; no rx overflows, no pause/flow-control drops, no switch drops; a bidirectional test saturates the full 10 G link. Conclusion: the limit is a PDMA/SoC RX ingress ceiling on this silicon, not a porting fault. Anything quoting “9.4 Gbit/s” as a bidirectional number is wrong; it is egress-only.

The A/B that shows what RSS is actually for

#

Counterfactual on banana ingress (iperf3 -P4: all-RX-on-ring0 via ethtool -X weight 1 0 0 0 vs 4-ring -X equal 4):

Raw ingress is wall-limited regardless of ring count, so the headline delta is small (+~0.6 G). The real win of multiring/RSS is headroom: single-ring pegs one core for all RX; four rings distribute the kthread work across four CPUs. That matters for the actual router workload (lots of concurrent flows across NAT/bridge), not for a raw endpoint iperf.

The real ingress knob: threaded-NAPI kthread affinity

#

On a fresh boot every napi/mtk_eth-0 kthread spawns with affinity CPU0 (it inherits the creating context, probe on CPU0 — IRQ masks don’t move threaded-NAPI work). Moving the kthreads across CPUs 0–3 (taskset) moved the dial:

Reproducible, ~0.5 G either way. Persisted in /etc/rc.local: pin the four PDMA IRQs, then taskset each napi/mtk_eth-0 kthread round-robin. IRQ affinity alone does nothing, because with threaded NAPI the work lives in the kthreads, not the IRQ handler.

Act III: The 9K Jumbo Escalation

#

RSS gave the CPU room. The next wall was packets, not bytes: at MTU 1500 the RX path is packet-rate-capped. MTU 2000 (the max the stock driver accepted) already bought +20–25 % ingress (~5.3 → 6.4–6.7 G) because every NAPI poll moves more bytes. The obvious next step was 9K — and it took three patches.

760-27: the 9K capability

#

Ported the frank-w / MTK jumbo series: a new MTK_NETSYS_RX_9K capability on MT7988, dynamic eth->rx_buf_len (1536/2048/9216) recomputed from the largest GMAC MTU, mtk_set_mcr_max_rx learned to program the XGMII XMAC_RX_CFG2 max-rx encoder for MTU up to 9216, per-open netdev->max_mtu (9K on the 10G MAC, 2K elsewhere), and mtk_max_buf_alloc(size) for order-aware jumbo buffers. ip link set eth2 mtu 9000 went from -EINVAL to accepted.

Act IIIb: the crash that made 760-28 necessary

#

With MTU 9000 persisted and a reboot (rings allocate 9K at open), the escape route appeared again:

skbuff: skb_over_panic: len:8046 put:8046 tail:0x206e end:0xec0 dev:<NULL>
Kernel BUG at skb_panic+0x4c/0x50
Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt

An 8046-byte frame into a ~3776-byte buffer. Two related root causes surfaced:

1. PDMA SDL was never programmed. The hardware’s max-RX-frame register stayed at its default, so jumbo frames >~1518 were either silently dropped or — after the MTU/length paths disagreed — delivered into undersized buffers.
2. The page_pool RX path is order-0. A single-page buffer cannot hold a 9K frame; skb_put() overran it and the kernel panicked.

760-28 fixed both: program MTK_PDMA_LRO_SDL + rx_buf_len into the PDMA RX cfg on netsys-v3+, and make mtk_page_pool_enabled() return false when any netdev runs an MTU above page_pool capacity so jumbo RX falls back to 9K-capable frag buffers. After that, an 8K raw blast into the CPU path no longer panicked: 0 crash counts, stable uptime.

The throughput numbers that made the work feel worth it (iperf3 -P4, 10G link, r182):

At 9K both directions approach the 10 G line rate for TCP and UDP; at 1500 the box is packet-rate- and CPU-limited. The 8-port router stopped being the bottleneck.

Act IV: The 9K Mystery Wasn’t the Router

#

Once the driver was crash-safe, end-to-end 9K still failed: 8K pings to the client, 100 % loss. The RCA went on a walk that ended, embarrassingly, at somebody else’s driver.

• The banana’s own switch-to-switch 8K worked (self-wire to a second host on lan6: 0 % loss both ways) — so the mtk RX/TX path, the MxL switch, and the CPU handling are all fine with 8K frames.
• Client → banana 8K: nothing physically left the client. Its TX byte counters climbed, the banana’s port MIBs stayed flat (no FCS/mtu-exceed), switch drop counters zero.
• Client detail: raw 1580-byte frames did arrive; ~8028-byte frames never did. A hard cliff at ~1518.
• The first diagnosis blamed the physical segment: the serial cable path “carries only ≤1518-byte frames” — the AS21010 PHY / SFP+ copper combo. My own notes say it with a straight face.

Then the operator corrected me mid-post: “wtf @ physically. i’m not using SFP, i’m using normal 6-grade twisted pair cable.”

Re-derivation with the correct facts, all measured:

• banana self-wire 8K: works.
• client AF_PACKET 8K (20 frames): 0 arrive; client counters say the bytes leave the NIC — the wire says they don’t.
• client can TX 1580 B but not ~8028 B: the client’s Aquantia AQC113 (“atlantic”) aq_nic_set_mtu never programs a TX max-frame/TPSMT register — disassembly literally moves mtu into a struct field and returns. ip link set mtu 9000 changed the Linux netdev MTU; the silicon still refused to transmit >1518 B, silently. TSO/GSO didn’t help because the check is at the MAC.

So: the banana’s 9K port is correct (MTU 9000 accepted, rings sized 9K, switch port max-packet-len 9022, no drops on the router); end-to-end 9K was blocked by a host NIC driver bug on the box I was testing from. The interesting hardware lesson: when a link says 10G and both ends say 9000 and the frame still doesn’t move, disassemble the driver before you blame the cable.

(For the record, the 9K-to-1G direction did surface one real switch quirk: CPU-port egress to a 1 G speed port drops >1518 frames even though peer-to-peer 1G↔10G jumbo works. That one is the MxL86252’s per-port egress limit, not mtk_eth_soc.)

Act V: The Footgun That Caused It All

#

Go back to Act IIIb’s crash. skb_over_panic, 8046 into 3776, in mtk_poll_rx. The page_pool and SDL fixes stopped it for boot-time-9K configs — but the trigger was a command that is supposed to be boring:

ip link set dev eth2 mtu 9000      # on a live, running interface

The rings are sized once, at first ndo_open, from eth->rx_buf_len. Changing the MTU on a running interface updates rx_buf_len, the PDMA SDL, and the MAC max-rx — but the already-allocated ring buffers stay at the old (smaller) size. Any jumbo frame larger than the old buffer overruns in skb_put(). frank-w’s own tree had the same limitation (“9K only works if MTU is set before open”); the forum’s answer was “set mtu via uci and reboot.”

My first fix was honest but crude: when the buffer size actually changed on a running interface, schedule mtk_pending_work — the full frame-engine reset worker (stop every netdev, FE/WED reset, reopen everything). It worked, and it was the wrong tool: heavyweight, and it left a grow-window race where the MAC was widened before the rings could take the larger frames.

Meanwhile frank-w’s 7.3-jumbo branch — and, beneath it, MTK’s own feed patches (999-eth-33/999-eth-50, Mason Chang) — had landed the right design: a dedicated, light-weight worker that reallocates only the DMA rings. I reworked 760-29 to match it:

The rework, mechanically

#

rx_buf_len stops being something mtk_change_mtu() writes. A helper mtk_rx_buf_len() derives it from the largest GMAC MTU, and mtk_dma_init() stores it at ring-allocation time — the single source of truth, so eth->rx_buf_len always describes the buffers the rings actually hold, and no path can let a MAC accept what the rings can’t take.

The ring bring-up/teardown is factored out of mtk_open/mtk_stop into mtk_rings_start() / mtk_rings_stop() — rings only, no phylink, no link flap. The worker runs under rtnl_lock + the existing MTK_RESETTING flag:

rtnl_lock()
if MTK_RESETTING: bail                      # a full FE reset will re-derive anyway
new_len = mtk_rx_buf_len(eth)
if !dma_refcnt or new_len == old: bail
set_bit(MTK_RESETTING)
for running devs: netif_tx_disable; mark   # no TX into rings about to be freed
if new_len < old_len: narrow MACs first    # SHRINK order
mtk_rings_stop(); mtk_rings_start()        # realloc at the new size
if grow: widen MACs after rings fit        # GROW order
clear_bit(MTK_RESETTING)
for running devs: netif_tx_wake_all_queues
rtnl_unlock()

The ordering contract is the whole point: shrinking narrows the MACs first (so no oversized frame can arrive while the smaller rings are installed), growing widens them only after the rings fit the larger frames. And there’s a trap specific to us: frank and MTK re-sync the PDMA SDL inside mtk_hwlro_rx_init() — which only runs on HW-LRO. Their MT7988 chips enable HWLRO; we deliberately didn’t (the forum-confirmed packet-ordering bug on this silicon). So the port must program SDL on a no-HWLRO path — I set it in mtk_dma_init() from the just-derived rx_buf_len, covering boot and every ring realloc, grow and shrink.

Flashed, then hammered

#

Built (make -j24 target/linux/compile, then the full image), staged, sysupgraded retaining config (no -n), pstore cleared, and the banana came back as r195-eb8f58aedf with br-lan still at 9000. Then the fun part — a live shrink and grow on a running interface:

[  571.829069] mtk_soc_eth 15100000.ethernet: rx buffer length 9216 -> 1536, reallocating the rx rings
[  577.421990] mtk_soc_eth 15100000.ethernet: rx buffer length 1536 -> 9216, reallocating the rx rings

…plus the boot-time grow (1536 -> 9216, netifd raising the conduit to jumbo live). Across it all:

• skb_over_panic / kernel WARNING count: 0.
• No link flap: conduit carrier_changes stayed at the single boot event; the worker never goes near phylink.
• No tx/rx errors on the conduits.
• Raise the ends to jumbo (client NIC 9000, DSA conduit 9004 — the conduit doesn’t self-elevate on this tree) and the functional test passes: 8K DF ping 3/3 both directions, iperf3 ~9.88 Gbit/s (banana→client) and ~9.83 Gbit/s (client→banana) at 9K MTU.
• CI: Check Kernel patches (mediatek, filogic) on the pushed series is green; second quilt refresh is a no-op.

That last outcome is the one worth sitting with: the command that used to panic the kernel is now a non-event. Rings resize live, links don’t drop, traffic keeps flowing.

What I Learned

#

1. A hang that prints nothing is findable, but a hang that boots might be the wrong hang. The “obvious” RSS probe hang was a module-context artifact; the actual blocker turned out to be a netdev registration-order race decoupled from everything I was instrumenting.
2. Boot-infra bugs masquerade as driver bugs. The missing eth2 / “RSS not testable” chapters were U-Boot bootconf variables and overlay-application timing — a fdt-1-only TFTP boot has no WAN, no combo ports, and no clue it’s missing them.
3. Verify the test rig before you trust a negative. “9K doesn’t work” was true; “the router’s 9K driver is broken” was not. The packet wasn’t leaving the other host’s NIC. ip link set mtu changing the Linux MTU ≠ the silicon accepting the frames.
4. Honesty in benchmarks is a feature. The RSS headline was egress-only; the ingress wall (~4.5–4.8 G) survived every software lever, and the honest A/B is what proved RSS’s actual value is CPU headroom, not a bigger iperf number.
5. When you can’t raise the speed, raise the MTU. 9K jumbo bought +126 % TCP / +313 % UDP on the previously packet-rate-capped RX path — sometimes the cheapest win is fewer, bigger packets.
6. “Fix it with a reset” is a smell. The full-FE-reset answer to a buffer-resize problem worked; the dedicated rings-only worker is what survived review against frank-w’s and MTK’s own designs. Stop/start only what changed and order the boundaries so nothing bigger than the rings can arrive.

the centre held — it was never the busy register read, nor the RSS rings, nor the cable. It was the one-liner that moves a NETDEV’s MTU while the DMA is live, and the discipline to rebuild the thing that line touches until it’s boring.

Follow-up: the full series — 760-21..26 (multi-ring NAPI/RSS + ethtool surface), 760-27 (9K/rx_buf_len), 760-28 (SDL + page_pool gate), 760-29 (dedicated rx-ring-realloc worker), 979 (stable MAC via nvmem), 980 (NAPI dummy-device reorder) — lives on the bpi-r4pro-8x-v2-multiring-napi branch of the fork, quilt-canonical and CI-green. Design docs, the marker-wave archaeology, the honest throughput tables, and the console boot audit are all in the project notes. Next on the list: a newer on-device ethtool, optionally mounting the QDMA-TX rework if that ever becomes relevant to this base, and re-validating stable-MAC/nvmem if we bump the kernel.