TL;DR

#

Every SSH handshake from a LAN host through my OpenWrt router died the moment the key exchange began — GitHub, GitLab, legacy kex algorithms, even SSH-over-443. Plain TCP to port 22 connected fine and HTTPS worked perfectly. Packet captures initially screamed “ISP deep-packet-inspection is forging RSTs.” Adding MAC-address capture flipped the diagnosis: the killer RST carried the router’s own WAN MAC address, meaning the router generated it itself. The culprit was the vendor’s software nftables flowtable (flow add @f in the forward hook), which offloaded NAT’d SSH flows and then lost their state mid-KEX, causing the kernel to answer GitHub’s final key-exchange packet with an RFC-793 reset — as if nothing was listening on :57700. The router’s own SSH always worked because it never crosses the forward path. Removing the flowtable offload fixed it instantly, 3-for-3.

The Setup

#

• Router: a Banana Pi-class board running OpenWrt with firewall4 (fw4) + nftables
• WAN: public IP 203.0.113.17/23 on br-wan, gateway 203.0.113.1 (MAC 02:00:00:00:ff:01)
• LAN: 10.222.1.0/24 on br-lan, hosts NAT’d out via masquerade
• Client: a workstation at 10.222.1.22 (MAC 02:00:00:00:00:0a)
• The failing thing: ssh git@github.com, git push over SSH, any SSH to GitLab — all reset mid-handshake. Long-lived SSH sessions also died periodically.

The Symptom

#

ssh -T git@github.com → Connection reset by 140.82.121.3 port 22.

git push over SSH → fatal: Could not read from remote repository.

The sessions weren’t refused and never timed out — they died with a hard reset, always at the same spot. Reconnecting always failed the same way. If you’ve ever had a carrier that mangles SSH, this feels exactly like that.

Hypothesis 0: It’s the ISP

#

The first instinct is always the carrier. The WAN is a single public IP behind NAT; SSH on port 22 is the classic thing middleboxes stab. I wrote a quick test matrix before capturing anything:

That last row was the smoking gun nobody noticed at first. If an ISP middlebox were resetting SSH by protocol content, it would kill the router’s own SSH too. It doesn’t. But port 22 and 443 both dying pointed at content-based filtering, not port-based — which kept the ISP hypothesis alive longer than it deserved.

First Capture: The Forged-RST Red Herring

#

I ran tcpdump on the router’s WAN bridge (br-wan) while re-triggering the failure:

20:30:10.606477 IP 203.0.113.17.45060 > 140.82.121.3.22: Flags [S], seq 1627035582, ...
20:30:10.632563 IP 140.82.121.3.22 > 10.222.1.22.45060: Flags [S.], seq 4009941603, ack 1627035583, ...
20:30:10.633228 IP 203.0.113.17.45060 > 140.82.121.3.22: Flags [P.] ... length 33: SSH: SSH-2.0-OpenSSH_10.4p1 Debian-4
20:30:10.843331 IP 140.82.121.3.22 > 10.222.1.22.45060: Flags [P.] ... length 817: SSH: SSH-2.0-feb815a
20:30:11.234786 IP 140.82.121.3.22 > 10.222.1.22.45060: Flags [P.] ... length 1232
20:30:11.237635 IP 203.0.113.17.45060 > 140.82.121.3.22: Flags [P.] ... length 16
20:30:11.361847 IP 140.82.121.3.22 > 203.0.113.17.45060: Flags [P.], seq 2066:2662, ack 2929, ... length 596
20:30:11.361893 IP 203.0.113.17.45060 > 140.82.121.3.22: Flags [R], seq 1627038512, win 0, length 0

Study that last pair. The TCP handshake completes. Both SSH banners are exchanged. The KEXINIT goes both ways. Then GitHub sends its final 596-byte key-exchange reply — and 46 microseconds later a reset arrives from our own public IP back toward GitHub, with win 0, no TCP options, and seq 1627038512 — which is exactly the ACK number GitHub just used. In one other capture a reset even showed a corrupted source IP (140.28.121.4, clearly not GitHub’s 140.82.121.x).

For a network-security engineer this is the textbook signature of TCP RST injection: a middlebox watching the stream that guesses/derives the sequence number and sends a spoofed reset from one side to tear the connection down. The corrupted source IP, the bare RST header, the 46µs reaction time — I was ready to write “the ISP’s DPI is resetting SSH.” I was wrong, and the thing that saved me was listening to the user’s instinct: capture MAC addresses.

The Decisive Capture: MAC Addresses

#

Redoing the same capture with -e (link-layer headers) on both br-wan and br-lan changes the story completely. First, the same moment on the WAN bridge:

1786989274.478061 02:00:00:00:ff:01 > 02:00:00:00:00:01 ... 140.82.121.3.22 > 203.0.113.17.57700: Flags [P.] seq 1183093451:1183094047, ack 3215364687, ... length 596
1786989274.478082 02:00:00:00:00:01 > 02:00:00:00:ff:01 ... 203.0.113.17.57700 > 140.82.121.3.22: Flags [R] seq 3215364687, win 0, length 0

The reset at .478082 has source MAC 02:00:00:00:00:01 — the router’s own br-wan MAC — and it’s leaving the router toward the gateway. An ISP-injected RST would arrive at the router from the gateway MAC (02:00:00:00:ff:01), not depart from the router’s own interface. This packet was generated by the router itself.

And its seq 3215364687 is again exactly the ACK GitHub just sent in the 596-byte packet it’s answering. Per RFC 793, when a TCP stack receives a segment for a socket it doesn’t have, it replies with a reset whose sequence number is taken from the incoming segment’s ACK field:

If the incoming segment has an ACK field, the reset takes its sequence number from the ACK field of the incoming segment.

RST.seq = incoming.ACK — that is the kernel talking, not a middlebox guessing.

Now the LAN side, same flow, at the same instant:

1786989274.388837 02:00:00:00:00:0a > 02:00:00:00:00:02 ... 10.222.1.22.57700 > 140.82.121.3.22: Flags [P.], seq 3215364687:3215364731, ... length 44
1786989274.626068 02:00:00:00:00:0a > 02:00:00:00:00:02 ... 10.222.1.22.57700 > 140.82.121.3.22: Flags [P.] ... length 44   <- retransmission
1786989274.654250 02:00:00:00:00:02 > 02:00:00:00:00:0a ... 140.82.121.3.22 > 10.222.1.22.57700: Flags [R], seq 1183093451, win 0, length 0

The client never sent an RST — on the whole LAN capture there is exactly one reset, and it’s GitHub’s reply reset (the .654250 line, src = github). The client just sits there waiting and retransmitting its 44-byte push, which is never acknowledged. Meanwhile GitHub’s final 596-byte SSH packet never reaches the client at all — it arrives at the WAN bridge, the router eats it, and the router immediately answers GitHub with a reset for a port it doesn’t own.

Reconstructing the full story:

1. The client’s 44-byte SSH push leaves the LAN at .388837 … and never appears on the WAN until the retransmission at .626081. The router dropped its own client’s outbound packet.
2. GitHub’s 596-byte key-exchange reply arrives on the WAN at .478061 … and is never forwarded to the client.
3. Instead the packet is treated as if it were destined to the router itself (203.0.113.17:57700), no socket is listening there, and the kernel sends the RFC-793 reset.

So the NAT binding for this flow vanished mid-handshake. The flow stopped being translated; it started being answered.

Why the Router Itself Was Immune

#

The router’s own SSH (source = 203.0.113.17, no masquerade) always worked, even to GitLab, which failed from the LAN. That asymmetry is the key architectural clue: router-originated traffic takes the kernel output path, while LAN client traffic takes the forward path. Something in the forward path was breaking SSH and nothing else. A DPI middlebox can’t see “which path” — it just sees SSH bytes. This narrows it to local forwarding machinery.

The Flowtable

#

Dumping the nftables ruleset surfaced a guest-star that had been hiding in plain sight:

table inet filter {
    flowtable f {
        hook ingress priority filter + 1
        devices = { eth0, eth1, eth2, lan0, lan3, mxl_lan0, mxl_lan1, mxl_lan2, mxl_lan3, mxl_lan5, phy0.0-ap0, phy0.1-ap0 }
        flags offload
        counter
    }

    chain forward {
        type filter hook forward priority filter; policy accept;
        meta l4proto { tcp, udp } flow add @f
    }
}

A software flowtable (flags offload) sits at forward priority, and the forward chain feeds every TCP/UDP flow into it with flow add @f. Once a flow is offloaded, packets are steered through the flowtable’s fast path instead of the normal conntrack/NAT pipeline. And crucially — this rule only exists in the forward chain. Router-local traffic never hits it. That matches the asymmetry exactly: forwarded SSH dies, local SSH lives.

The flowtable is a known category of troublemaker: the offloaded fast path has limited awareness of TCP sequence tracking, retransmissions, and state transitions, and when a connection’s state gets lost, packets fall out of the translated path and get treated as local traffic — producing precisely the “kernel answers with a reset using the incoming ACK as its sequence number” behavior we saw. The client’s dropped 44-byte packet is the fastpath swallowing an in-flight segment.

Where does it come from on this firmware? A vendor script chain, not stock OpenWrt UCI:

/etc/hotplug.d/net/99-flowtable_update  →  /sbin/smp-mt76.sh  →  /sbin/flowtable.sh  →  /etc/flowtable.conf

flowtable.sh generates /etc/flowtable.conf by enumerating every eth*/lan*/phy* interface, then does nft delete table inet filter + nft -f /etc/flowtable.conf. smp-mt76.sh calls nftables_flowoffload_enable at boot (init.d/smp) and the hotplug script re-runs it on every interface-add event. Any reboot or link flap re-arms it.

The Fix

#

Live removal first, to confirm the hypothesis before touching anything persistent:

nft delete rule inet filter forward handle 4

Then re-running the SSH test from the NAT’d client:

$ ssh -o BatchMode=yes git@github.com
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Hi <username>! You've successfully authenticated, but GitHub does not provide shell access.

Three for three. No reset, full authentication, the exact command that had failed every single time minutes earlier.

Then make it survive reboot and hotplug by neutralizing the generator (backup kept):

cp -a /sbin/flowtable.sh /sbin/flowtable.sh.bak
# nftables_flowoffload_enable()  →  no-op (return 0)
nft delete table inet filter          # purge the live table
sh /sbin/smp-mt76.sh                  # re-run loader, confirm nothing re-adds
nft list ruleset | grep -c flowtable  # → 0

Verification

#

• SSH to GitHub from the NAT’d client: authenticated 3/3.
• nft list tables shows only the normal inet fw4 table; the offloaded flowtable is gone.
• The loader was re-run and did not re-arm the offload (count stayed 0).
• The tradeoff: software flowtable offload was presumably there to reduce CPU on the router under routing load. Disabling it costs some headroom on heavy WAN traffic — a fair price for SSH, git, and every long-lived TLS connection working.

Lessons

#

1. MAC addresses are the tiebreaker in RST-injection questions. An ISP/on-path forge arrives from somebody else’s MAC; a router-generated reset departs from its own. tcpdump -e is the difference between blaming the carrier and fixing your own box.
2. RST.seq = incoming.ACK means the kernel answered, not a middlebox. A stack with no socket for a tuple sends exactly this. Corrupted source IPs and bare headers look like DPI — don’t be seduced; check who can emit a packet from that L2 source.
3. Test both NAT’d and local paths. The router’s own SSH working while LAN SSH dies isn’t noise — it’s a routing of the cause to the forward path.
4. Read the asymmetric test results before blaming the ISP. The matrix had the answer in it all along; the flowtable just made the mechanism invisible until we looked at L2.
5. flow add @f offload and SSH do not mix on this firmware. If you see this vendor flowtable, consider disabling it — SSH/TLS flows are exactly the kind that get eaten.

All IPs/MACs in the captures are redacted placeholders; the public WAN IP is shown as 203.0.113.17 (TEST-NET) and MACs as 02:00:00:00:00:xx. The remote GitHub addresses are GitHub’s public addresses.