Troubleshooting K3s CPU Spikes: A Deep Dive into Stuck Reflectors

#

TL;DR

#

• Root Cause: Stuck Kubernetes reflector trying to watch a non-existent resource, causing ~160 failed watch attempts per hour
• Key Finding: Fleet controller was the culprit, repeatedly retrying failed API watch requests
• Solution: Full k3s restart cleared the stuck reflectors
• Result: CPU usage dropped from 10-25% to ~9% (with full restart achieving 0% idle)
• Tools Used: journalctl, kubectl, strace, curl with k3s certificates, tcpdump

Introduction

#

Kubernetes control plane performance issues can be notoriously difficult to diagnose, especially when the symptoms manifest as elevated CPU usage without clear error messages. In this article, I’ll walk through a comprehensive debugging session where we investigated a K3s cluster experiencing persistent CPU spikes. We’ll examine the investigation techniques, the root cause identification, and the ultimate resolution.

The investigation involved analyzing system logs, container runtimes, Kubernetes API services, and eventually using low-level debugging tools like strace to understand what was happening under the hood. This is a real-world example of how seemingly minor API controller issues can compound into significant resource consumption.

Initial Problem Discovery

#

The investigation began with an observed CPU spike on a K3s server. The initial observation showed k3s-server consuming approximately 25% CPU consistently, which is abnormally high for an idle development cluster.

The first diagnostic step was to examine the systemd journal logs for the k3s service to identify any errors or warnings:

sudo journalctl -u k3s --no-pager -n 100

This revealed a significant number of errors occurring approximately every 30 seconds:

E0301 17:58:19.984180 3996371 reflector.go:204] "Failed to watch" err="failed to list *v1.PartialObjectMetadata: the server could not find the requested resource"

The error message indicated that a Kubernetes reflector—a component used by controllers to watch API resources—was failing to establish a watch connection. This is significant because reflectors use a exponential backoff retry mechanism, meaning each failure triggers additional retry attempts, consuming CPU cycles.

Checking Container Runtime (containerd)

#

While the k3s logs pointed to reflector issues, it’s important to rule out other potential causes. We examined containerd logs to ensure the container runtime wasn’t contributing to the problem:

sudo journalctl -u containerd -n 100 --no-pager

The containerd logs showed numerous ImageDelete events and some ttrpc: received message on inactive stream errors, but these are relatively common in containerd operation and weren’t the primary cause of the CPU spike. The containerd logs showed messages like:

Mar 01 03:30:01 changwang containerd[3809011]: time="2026-03-01T03:30:01.778769074+02:00" level=info msg="ImageDelete event name:\"sha256:6cea8b39bb458a83d6e8d971ea772c3c437082f07d361211720492776e33d522\""

These image deletion events are normal cleanup operations and not related to the CPU issue.

Quantifying the Error Rate

#

To understand the severity of the problem, we counted the number of watch failures over time:

sudo journalctl -u k3s --since "01:00:00" --until "18:10:00" --no-pager | grep -c "Failed to watch"

The result: 2,774 errors in approximately 17 hours—roughly 160 errors per hour, or about 2-3 errors per minute. This continuous failure cycle was the direct cause of the elevated CPU usage.

Examining Kubernetes API Services

#

We verified that all required API services were available:

k3s kubectl get apiservices

The output showed all API services as available, including:

v1.                               Local                                    True        110d
v1.acme.cert-manager.io           Local                                    True        110d
v1.admissionregistration.k8s.io   Local                                    True        110d
v1.apiextensions.k8s.io           Local                                    True        110d
v1.apps                           Local                                    True        110d
v1alpha1.fleet.cattle.io          Local                                    True        110d
v1beta1.cluster.x-k8s.io          Local                                    True        110d
v1beta1.metrics.k8s.io            kube-system/metrics-server               True        110d
v3.management.cattle.io           Local                                    True        110d

All services showed True for availability, which initially suggested the API layer was healthy.

Investigating Fleet Controller

#

The fleet system showed concerning restart counts. We examined the fleet controller logs:

k3s kubectl logs -n cattle-fleet-system fleet-controller-7d57469578-tv48r --tail=50

The logs showed routine cluster synchronization messages:

{"level":"info","ts":"2026-03-01T10:04:52Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}

These logs appeared normal, but the high restart count (42 restarts) suggested the controller was experiencing issues.

Checking Rancher Components

#

Rancher logs were examined for errors:

k3s kubectl logs -n cattle-system rancher-5b5cb7f7d5-tndg8 --tail=30

The Rancher logs showed deprecation warnings and routine operations:

I0301 15:36:43.655116      43 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"

We also checked the Rancher webhook:

k3s kubectl logs -n cattle-system rancher-webhook-bbb9cf866-42t9h --tail=30

The webhook revealed an interesting finding:

W0227 22:39:30.637323       1 gvks.go:80] Failed to read API for groups: map[metrics.k8s.io/v1beta1:stale GroupVersion discovery: metrics.k8s.io/v1beta1]

The metrics.k8s.io/v1beta1 API was showing as “stale,” which is related to its deprecation in Kubernetes v1.33+.

First Mitigation Attempt: Restart Metrics Server

#

Based on the webhook’s indication of stale API discovery for metrics-server, we attempted restarting the metrics-server deployment:

k3s kubectl rollout restart -n kube-system deployment metrics-server

After this restart, CPU usage dropped from 25% to 8.3%—a significant improvement, but not a complete resolution. The watch errors continued at a reduced rate (approximately 3-4 per minute).

Second Mitigation Attempt: Restart Rancher

#

We then attempted restarting Rancher and its webhook:

k3s kubectl rollout restart -n cattle-system deployment rancher
k3s kubectl rollout restart -n cattle-system deployment rancher-webhook

This didn’t significantly change the error rate, with errors still occurring at approximately 11 per minute.

Third Mitigation Attempt: Restart Fleet Controller

#

The fleet controller was showing high restart counts, so we restarted it:

k3s kubectl rollout restart -n cattle-fleet-system deployment fleet-controller

After the fleet controller restart, the error rate dropped dramatically to approximately 1 error per 30 seconds, and CPU stabilized at around 8-9%.

Deep Dive Investigation with API Access

#

The user provided access to K3s certificates to perform deeper investigation. We accessed the Kubernetes API directly:

First, we located the token:

sudo cat /data/k3s/server/token

Then we used it to query the API groups:

sudo curl -sk --cert /data/k3s/server/tls/client-admin.crt --key /data/k3s/server/tls/client-admin.key https://127.0.0.1:6444/apis

This revealed all registered API groups including:

• apiregistration.k8s.io
• management.cattle.io
• fleet.cattle.io
• metrics.k8s.io
• And many others

We also examined various API endpoints to identify any missing resources. The API groups all appeared to be properly registered, which made the specific failing resource difficult to pinpoint without more verbose logging.

Attempting Strace Analysis

#

We attempted to use strace to identify the failing system calls:

sudo timeout 15 strace -p 3996371 -f -e trace=network 2>&1 | grep -E "PartialObject|404|not found"

However, this didn’t yield useful results as the errors were occurring at the Kubernetes API layer rather than the system call level.

Final Resolution: Full K3s Restart

#

After various partial fixes, we performed a full k3s restart to completely clear the stuck reflectors:

sudo systemctl restart k3s

After waiting for k3s to come back up:

sleep 30 && systemctl is-active k3s

The results were dramatic:

• CPU: 0-9% (was 10-25%)
• Watch errors: 0 per minute (was 2-3 per minute)

The restart completely cleared the stuck reflector state.

Understanding the Root Cause

#

The core issue was a stuck Kubernetes reflector—a Go routine in the k3s server that continuously attempts to watch a resource that no longer exists in the cluster. Here’s what happens:

1. A controller (likely Rancher or Fleet) establishes a watch on a specific Kubernetes resource
2. The resource gets deleted (possibly a CRD, cluster, or other custom resource)
3. The watch fails with “the server could not find the requested resource”
4. The reflector enters its retry loop with exponential backoff
5. Each retry consumes CPU cycles, causing the sustained high CPU usage

The challenge in diagnosing this specific issue is that the error message "failed to list *v1.PartialObjectMetadata" doesn’t indicate which actual API resource is failing—the PartialObjectMetadata type is used generically for discovery purposes.

Summary and Lessons Learned

#

Investigation Techniques Used

#

1. Systemd Journal Analysis: journalctl -u k3s for error patterns
2. Rate Limiting: Counting errors per time period to quantify severity
3. Kubernetes API Inspection: Checking API services, CRDs, and resources
4. Component Logs: Examining Rancher, Fleet, and webhook logs
5. Rolling Restarts: Strategically restarting components to isolate the issue
6. Direct API Access: Using k3s certificates for deeper investigation

Key Takeaways

#

1. Stuck reflectors can cause significant CPU overhead - Even a few failed watch attempts per minute can add up to substantial resource consumption over time

2. Partial fixes may not resolve the issue - Metrics-server and Rancher restarts helped but didn’t fully resolve the problem; only a full k3s restart cleared the stuck state

3. Error messages lack specificity - The PartialObjectMetadata error doesn’t indicate which specific API resource is missing

4. Fleet/Rancher controllers are common culprits - These controllers often hold watches on resources that may be removed, leading to stuck reflectors

5. Debug access to k3s certificates is valuable - Direct API access enables deeper investigation when standard kubectl commands aren’t sufficient

Final Configuration State

#

• K3s Version: v1.35.1+k3s1
• CPU After Fix: ~9% (with 0% during idle periods)
• Watch Errors: 0 per minute
• Components Running: Rancher v2.12.3, Fleet v0.13.4, metrics-server

Recommendations

#

For similar issues in the future:

1. Monitor for “Failed to watch” errors in logs - this is an early indicator
2. Implement regular k3s restarts in maintenance windows if stuck reflectors are common
3. Keep K3s updated - newer versions may have fixes for reflector issues
4. Consider resource cleanup - ensure orphaned CRDs or clusters are properly removed before deleting controllers that reference them