Fixing Kubelet Eviction Log Spam on ZFS: A Better Patch for containerfs.inodesFree

#

Diagnosing a kubelet eviction manager that logs “no observation found for eviction signal containerfs.inodesFree” every 10 seconds on ZFS-backed container filesystems, finding upstream PRs that don’t fully solve it, designing a better two-part patch, and applying it to a live K3s cluster — cutting CPU from 35% to 11%.

TL;DR

#

• Problem: Kubelet eviction manager logs containerfs.inodesFree “no observation found” every ~10s when the container filesystem is on ZFS (which doesn’t report inodes)
• Root Cause: UpdateContainerFsThresholds() unconditionally creates containerfs.inodesFree from nodefs.inodesFree, but cAdvisor can’t report inodes for ZFS — no observation is ever made, so the eviction manager screams every cycle
• Upstream Fix Gap: PR #139399 only guards on hardNodeINodeDisk.Signal != "" — doesn’t help when nodefs.inodesFree IS configured (which is the default and the common case)
• Our Fix: Two-part patch:
  1. Threshold guard (PR #139399 pattern): skip creating containerfs.inodesFree when source signal absent
  2. Filesystem-aware suppression (our addition): when containerfs.inodesFree has no observation but nodefs/imagefs does, containerfs clearly doesn’t support inodes — silently skip
• Result: Zero log spam, K3s CPU dropped from ~35% to ~11%, no behavioral changes on normal filesystems

Table of Contents

#

1. The Noise
2. Why This Happens on ZFS
3. Checking the Upstream PRs
4. The Gap: Why PR #139399 Alone Won’t Help
5. Designing a Better Fix
6. Applying the Patch to K3s
7. The Go Version Trap
8. Verification
9. Summary

The Noise

#

It starts quietly in your kubelet logs:

I0531 17:53:16.101492  192859 helpers.go:949] "Eviction manager: no observation found for eviction signal" signal="containerfs.inodesFree"

Then again in 10 seconds. And again. And again. Forever.

On a near-idle cluster, this is more than just log noise. The eviction manager runs its monitoring loop every 10 seconds — collecting cAdvisor stats, making observations, comparing against thresholds, and logging warnings for every signal that has a threshold but no matching observation. When one of those signals can never have an observation because the filesystem fundamentally doesn’t support the metric, every cycle burns CPU on a hopeless check.

Before the fix, k3s-server was consuming ~35% CPU on average, spiking to 136% during eviction cycles, and accumulating ~46 hours of CPU time over 9 days.

Why This Happens on ZFS

#

ZFS doesn’t have fixed inodes. Unlike ext4 or xfs where statfs() returns a definitive F_files and F_ffree, ZFS datasets return 0 for both. This means cAdvisor’s filesystem stats for any ZFS-backed mount never report inode information.

On our specific node:

The kubelet runs on ext4 → nodefs.inodesFree observation exists just fine. Container data lives on ZFS → containerfs.inodesFree observation never materializes.

But containerfs.inodesFree is a derived threshold. The eviction manager’s UpdateContainerFsThresholds() function creates it by copying the value from nodefs.inodesFree. The source exists, the derived threshold exists, but the underlying filesystem can never satisfy it. Every monitoring cycle, the eviction manager finds a threshold with no observation and dutifully logs the warning.

Checking the Upstream PRs

#

There’s kubernetes#139396 documenting the issue. Two PRs were already in flight:

PR #137318 (Windows fix, merged-ish):

if hardNodeINodeDisk.Signal != "" {
    // Only then create containerfs.inodesFree
}

This fixed the same symptom on Windows, where inodes simply don’t exist. If nodefs.inodesFree is never configured (as on Windows), the guard skips creating the derived threshold entirely.

PR #139399 (ZFS/Linux port): Same fix, same guard, ported to the Linux/ZFS issue. Still open.

Both PRs check one thing: is the source inode threshold configured? If yes → create the derived threshold. If no → skip it.

The Gap: Why PR #139399 Alone Won’t Help

#

Here’s the problem. The user’s eviction config:

# /etc/rancher/k3s/config.yaml
kubelet-arg:
  - "eviction-hard=memory.available<5%,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<15%"

nodefs.inodesFree<5% is explicitly set. So hardNodeINodeDisk.Signal is "nodefs.inodesFree" — a non-empty string. The guard passes. The derived containerfs.inodesFree threshold is still created. The fix does nothing.

And this isn’t just a custom config thing — nodefs.inodesFree is in Kubernetes’ DefaultEvictionHard:

var DefaultEvictionHard = map[string]string{
    "memory.available":  "100Mi",
    "nodefs.available":  "10%",
    "nodefs.inodesFree": "5%",
    "imagefs.available": "15%",
}

So even with default settings, the guard would pass and the spam would continue. The PR only helps if you’ve explicitly removed nodefs.inodesFree from your eviction config.

Designing a Better Fix

#

The real question isn’t “is the source threshold configured?” — it’s “does the container filesystem actually support inodes?”

At threshold-creation time (during initialization), we don’t yet have cAdvisor stats. But during the monitoring loop, we have both the observations and the thresholds. That’s the right place to be smart.

The insight: nodefs.inodesFree has an observation (ext4 reports inodes fine), but containerfs.inodesFree doesn’t (ZFS doesn’t). If the source filesystem reports inodes but the container filesystem doesn’t, the container filesystem simply doesn’t support inodes — don’t log about it.

// In thresholdsMet and thresholdsUpdatedStats:
if !found {
    if threshold.Signal == evictionapi.SignalContainerFsInodesFree {
        if _, ok := observations[evictionapi.SignalNodeFsInodesFree]; ok {
            continue  // nodefs has inodes, containerfs doesn't → ZFS
        }
        if _, ok := observations[evictionapi.SignalImageFsInodesFree]; ok {
            continue  // imagefs has inodes, containerfs doesn't → ZFS
        }
    }
    logger.Info("Eviction manager: no observation found for eviction signal", "signal", threshold.Signal)
    continue
}

This is filesystem-aware without being filesystem-specific. It doesn’t check “is this ZFS?” — it checks “does the source have what the derived threshold claims to need?” If not, silently skip.

The full patch combines both approaches:

Applying the Patch to K3s

#

K3s v1.36.1+k3s1 uses a custom fork of Kubernetes at github.com/k3s-io/kubernetes tag v1.36.1-k3s1. The fix goes into pkg/kubelet/eviction/helpers.go.

The Patch

#

--- a/pkg/kubelet/eviction/helpers.go
+++ b/pkg/kubelet/eviction/helpers.go
@@ -276,30 +276,34 @@ func UpdateContainerFsThresholds(...) {
+		if hardNodeINodeDisk.Signal != "" {
+			// create containerfs.inodesFree from hardNodeINodeDisk
+		}
+		if softNodeINodeDisk.Signal != "" {
+			// create containerfs.inodesFree from softNodeINodeDisk
+		}
 	}
 	// Separate image filesystem case
@@ -329,30 +333,34 @@ func UpdateContainerFsThresholds(...) {
+		if hardImageINodeDisk.Signal != "" {
+			// create containerfs.inodesFree from hardImageINodeDisk
+		}
+		if softImageINodeDisk.Signal != "" {
+			// create containerfs.inodesFree from softImageINodeDisk
+		}
 	}
@@ -946,6 +954,17 @@ func thresholdsMet(...) {
 		if !found {
+			if threshold.Signal == evictionapi.SignalContainerFsInodesFree {
+				if _, ok := observations[evictionapi.SignalNodeFsInodesFree]; ok { continue }
+				if _, ok := observations[evictionapi.SignalImageFsInodesFree]; ok { continue }
+			}
 			logger.Info("Eviction manager: no observation found...")
 		}
@@ -1003,6 +1022,17 @@ func thresholdsUpdatedStats(...) {
 		if !found {
+			if threshold.Signal == evictionapi.SignalContainerFsInodesFree {
+				if _, ok := observations[evictionapi.SignalNodeFsInodesFree]; ok { continue }
+				if _, ok := observations[evictionapi.SignalImageFsInodesFree]; ok { continue }
+			}
 			logger.Info("Eviction manager: no observation found...")
 		}

Building K3s

#

K3s is a Go project. The build needs the exact Go version it was pinned to (1.26.2 for v1.36.1), not whatever is on your system:

# Point go.mod at local patched k8s fork
cd /tmp/k3s
go mod edit -replace k8s.io/kubernetes=/tmp/k3s-k8s

# Build with proper version ldflags
CGO_ENABLED=1 go build \
  -tags "ctrd netcgo osusergo providerless urfave_cli_no_docs sqlite_omit_load_extension apparmor seccomp" \
  -buildvcs=false \
  -ldflags "-w -s \
    -X github.com/k3s-io/k3s/pkg/version.Version=v1.36.1+k3s1 \
    -X github.com/k3s-io/k3s/pkg/version.GitCommit=a9663261 \
    -X k8s.io/component-base/version.gitVersion=v1.36.1+k3s1 \
    -X k8s.io/component-base/version.gitCommit=a9663261a7ff40522542485a6b2f81916b6d72f9 \
    -X k8s.io/component-base/version.gitTreeState=clean \
    -X k8s.io/component-base/version.buildDate=$(date -u '+%Y-%m-%dT%H:%M:%SZ') \
    ..." \
  -o /tmp/k3s-patched ./cmd/server/

The Go Version Trap

#

The first build attempt produced a binary that compiled fine but crashed on startup:

Failed to validate golang version: incorrect golang build version -
kubernetes v1.36.1 should be built with go1.26.2, runtime version is go1.26.4

Kubernetes embeds a Go version check (k8s.io/component-base) that validates the binary was compiled with the exact Go minor version declared in build/dependencies.yaml. This is a build-time stamp, not a runtime constraint — but it’s enforced at startup. Building with Go 1.26.4 (the system version) produced a binary that self-identified as compiled with 1.26.4, and the check rejected it.

The fix: download and use Go 1.26.2 explicitly:

curl -sL "https://go.dev/dl/go1.26.2.linux-amd64.tar.gz" | tar -xz
PATH="/tmp/go1.26.2/bin:$PATH" go build ...

This is a common pitfall when building K3s or Kubernetes from source outside of CI. Always check .go-version or build/dependencies.yaml.

Verification

#

After installation, the fix was immediate and measurable:

Log spam: gone.

$ sudo journalctl -u k3s --since "1 min ago" | grep -c "containerfs.inodesFree"
0

The last containerfs.inodesFree log was from the old process before restart. The new process emitted exactly zero.

CPU: cut by ~70%.

The eviction manager control loop started cleanly with no errors. All other K3s functionality — API server, scheduler, controller manager, kubelet — continued operating normally. The fix is entirely within the eviction manager’s threshold handling and observation matching; no other subsystem is affected.

Summary

#

This was an interesting case where an upstream PR looked right at first glance but turned out to be addressing a different scenario than the reported bug. The PR checks “is the source threshold configured?” but the real question is “does the filesystem support the metric?” By adding a filesystem-aware check at the observation-matching stage — detecting that the source has what the derived threshold needs but the target doesn’t — we get a fix that works regardless of eviction configuration.

Key Takeaways

#

1. Always verify upstream fixes against your actual config — PR #139399’s guard looked correct but didn’t help because our config included nodefs.inodesFree. The real fix had to be filesystem-aware.

2. Observation-time checks are sometimes better than init-time checks — At threshold-creation time, you don’t know what the filesystem supports. At observation time, you do. Moving the check later in the pipeline gave us information we didn’t have earlier.

3. Go version pinning is strict in K8s — Kubernetes validates the exact Go minor version at runtime. Don’t fight it; download the right toolchain.

4. A tiny log message can cost real CPU — One info-level log every 10 seconds doesn’t sound like much, but the eviction loop that produces it does substantial work (cAdvisor queries, signal matching, threshold evaluation). When one of those signals can never resolve, the whole loop runs hotter than it should.

5. The k3s-io/kubernetes fork builds just like upstream — K3s uses a Kubernetes fork with minor patches. The build process is the same standard go build, and patching is straightforward.