Optimizing a Rust etcd KV Store: From 61% Overhead to 3% CPU in ~2 Weeks

#

The full journey of replacing the k3s kine+SQLite storage stack with a purpose-built Rust etcd v3 gRPC server and systematically eliminating every category of waste until 83% overhead turned into 97% useful work.

Table of Contents

#

1. Background: The Problem with kine
2. Architecture: What Kudurru Is
3. The Starting Point: 61% Infrastructure Overhead
4. Phase 1: Eliminate the SQL Tax
5. Phase 2: Reduce Lock Contention
6. Phase 3: Eliminate Allocations
7. Phase 4: Eliminate Polling
8. Phase 5: Reduce Serialization (Zero-Copy gRPC)
9. Phase 6: Stop Doing O(n) Work
10. Phase 7: Production Learnings
11. Final Results
12. Key Lessons

Background: The Problem with kine

#

k3s uses kine as an etcd-to-SQL abstraction layer. By default, kine backs into SQLite — a full ACID SQL database — to store what is fundamentally an append-only key-value log. For a 30-pod cluster writing ~2.4 operations per second (mostly lease heartbeats), this stack consumes 17% of a CPU core on the control-plane node.

That number was the canary in the coal mine. A CPU profile told the story:

Total CGo + GC + SQL overhead: ~83% of CPU cycles are not doing useful work.

This is a classic architectural mismatch. kine ships 5 SQL query templates that never change — they’re fmt.Sprintf’d at startup. Yet every query goes through SQLite’s full pipeline: lexical analysis, AST construction, cost-based optimization across 6 indexes, and bytecode VM execution. SQLite builds 6 separate B-tree secondary indexes for what is functionally a single ordered key-value map. ACID transactions, which kine never uses (every write is a single-row insert with a unique constraint), add _txlock=immediate and its associated locking overhead. And the 1-second poll-based watch loop — a SELECT ... WHERE id > ? running every second — adds up to 1000ms worst-case notification latency.

The Full List of Mismatches

#

Architecture: What Kudurru Is

#

Kudurru replaces the entire kine + SQLite stack with a single Rust binary that speaks native etcd v3 gRPC. No kine, no SQLite, no Go runtime, no garbage collector.

Data Flow

#

k3s ── gRPC etcd v3 ──▶  Tonic Server
(kube-apiserver)        │  (KV, Watch, Lease, Cluster,
                        │   Maintenance, Auth — 6 services)
                        │
                        │  Store (Arc<RwLock<StoreState>>)
                        │  ┌────────────────────────────┐
                        │  │ keys: BTreeMap<Vec<u8>,    │
                        │  │        KeyState>           │
                        │  │ leases: BTreeMap<i64,      │
                        │  │         LeaseState>        │
                        │  │ watchers: Vec<WatchReg>    │
                        │  │ wal: WalFile               │
                        │  └────────────────────────────┘
                        │
                        │  Background tasks:
                        │  ├─ fsync (every 50ms when dirty)
                        │  ├─ lease expiry (event-driven)
                        │  └─ WAL compaction (>64 MB)
                        └──────────────────────────────────┘

The Storage Model

#

The WAL is the single source of truth — an append-only binary file:

WAL file (append-only, sequentially written):
┌──────┬──────┬──────┬──────┬──────┬────────┐
│ Rec1 │ Rec2 │ Rec3 │ Rec4 │ Rec5 │  ...   │
└──────┴──────┴──────┴──────┴──────┴────────┘

Record format:
┌──────────┬──────────┬────────┬───────┬──────────┬──────────┬──────────┐
│ revision │ key_len  │ val_len│ flags │   key    │  value   │  crc32   │
│   u64    │   u32    │  u32   │  u8   │ [u8;N]   │ [u8;M]   │   u32    │
└──────────┴──────────┴────────┴───────┴──────────┴──────────┴──────────┘

On startup, the WAL is replayed into a BTreeMap<Vec<u8>, KeyState>. No checkpoint, no separate database file, no SQL parsing. The in-memory structure is rebuilt from scratch.

Operation Mapping

#

The Starting Point: 61% Infrastructure Overhead

#

The initial benchmark of the first working Kudurru prototype (P7 baseline, everything working):

Already a 6× CPU improvement over kine/SQLite just by eliminating the Go runtime, CGo, and SQL parsing. But the perf profile showed there was much more to extract — and more importantly, the architecture had bottlenecks that would limit reliability and latency at production scale.

The perf profile told us where to look:

Category           Share
──────────────────────────
Tokio runtime       ~50%  (poll, schedule, park, IO driver, timers)
clock_gettime       ~12%  (libc + vdso, used by tokio)
Kernel syscalls      ~4%  (epoll_wait 2%, write 1.4%)
Kudurru app          ~3%  (expiry task, RwLock)
Other                ~31%

The server was Tokio-runtime-bound at high concurrency, not CPU-bound. The storage layer itself was only ~3.8% of CPU. The bottlenecks were structural: lock contention, allocation churn, polling, and serialization overhead.

Phase 1: Eliminate the SQL Tax

#

This was the foundational decision, made before a single line of storage code was written. The entire architecture flows from rejecting the premise that a SQL database belongs between k3s and its key-value store.

What Changed

#

A binary WAL format replaced SQLite. The in-memory index is a BTreeMap<Vec<u8>, KeyState>. The “query planner” is a match on five RangeBound variants:

enum RangeBound {
    Point(Vec<u8>),
    From(Vec<u8>),
    Prefix(Vec<u8>),
    Range(Vec<u8>, Vec<u8>),
    All,
}

Each maps directly to a BTreeMap::range() call — O(log n + k) instead of O(n) SQL table scans with GROUP BY.

Why It Still Works

#

The five access patterns cover everything etcd’s protocol requires. The resolve_range function decodes etcd’s key + range_end encoding once per request, producing a RangeBound that’s cached in WatchRegistration for the lifetime of the watch. No per-event recomputation.

Why It’s Faster

#

SQLite’s 33% CPU share becomes exactly 0%. The 6 B-tree indexes become one BTreeMap. The ACID transaction layer becomes an append + atomic u64 increment. The SQL parser that never varied its 5 templates is replaced by a match statement with 5 arms.

Result

#

Phase 2: Reduce Lock Contention

#

The single RwLock<StoreState> serializes all writes. With the initial tokio::sync::RwLock, every write lock acquisition allocated a waker and went through the async runtime — even though the lock was never held across .await.

2a: Deferred WAL fsync (Critical)

#

Before: Every put/delete/lease_revoke called sync_all() while holding the write lock. fsync is ~1-10ms on typical SSDs, blocking ALL readers and writers during that time.

After: A background tokio task calls sync_all() every 50ms when the WAL is dirty. The write lock is held for write_all(1-10µs) only.

// Before: inside write lock
state.wal.append_kv(&record)?;
state.wal.sync_all()?;  // 1-10ms stall, blocks everything

// After: inside write lock  
state.wal.append_kv(&record)?;  // 1-10µs only
// sync_all() runs in background task, outside any lock

Trade-off: Up to 50ms of writes may be lost on power failure. This matches etcd’s raft commit model.

2b: Switch to parking_lot::RwLock

#

tokio::sync::RwLock is designed for locks held across .await points. Our lock is never held across .await, so we paid overhead for features we don’t use.

The switch was trivial — state.read().await → state.read(), state.write().await → state.write(). No spawn_blocking needed since critical sections are 1-11µs.

2c: Txn Comparison Uses Read Lock (Not Write Lock)

#

Before: txn() acquired the write lock just to evaluate comparisons — a read-only operation. This needlessly blocked all writers during comparison evaluation.

After: The comparison-evaluation phase uses state.read(), with the write lock only acquired for the execution phase. This reduces write lock contention during CAS-heavy workloads (common in k3s for lease renewal).

2d: Linearizable Transactions

#

Before: Txn evaluated comparisons under the read lock, dropped it, then re-acquired the write lock for execution. Between the drop and re-acquire, the state could change, making the comparison stale — a TOCTOU race.

After: Both comparison and execution happen under the write lock. For a single-node store, this provides linearizable read-modify-write semantics more simply than etcd’s quorum-based approach.

2e: Atomic Counters for Compact Rev and Status

#

compact_rev was previously guarded by the write lock even though it’s monotonically increasing and never read under contention. Moved to a global AtomicU64. Similarly, the 60s periodic status task acquired the read lock just to count keys/watchers/leases — replaced with AtomicU64 counters incremented at each mutation point.

Combined Result

#

But the most important improvement wasn’t captured by the benchmark: read availability under write load. Previously, any Range or Watch replay that arrived during a write’s fsync would stall for 1-10ms. Now that stall is 1-10µs — a ~1000× improvement for read-side latency under concurrent writes.

Phase 3: Eliminate Allocations

#

The perf profile identified __memmove_evex_unaligned_erms at 3.42% and malloc/cfree at 3.6% — allocation churn in the response path.

3a: Arc<[u8]> for Values

#

Before: Every KeyState stored value: Vec<u8>. Cloning (for prev_kv in responses, for watch events) memcpy’d the entire value.

After: value: Arc<[u8]>. Clone is an atomic increment. The deep copy only happens at the response boundary (when tonic needs Vec<u8>), and even that was later eliminated.

3b: prost::bytes::Bytes for Response Fields

#

Before: Arc<Vec<u8>> required .to_vec() at the response boundary to convert back to Vec<u8> for tonic.

After: All response fields — RangeResponse.kvs, PutResponse.prev_kv, Event.kv, Event.prev_kv — use prost::bytes::Bytes. Bytes::clone() is a refcount increment. The same Bytes goes directly into the protobuf response without any conversion or memcpy.

Why this works (wire encoding identity): In protobuf’s wire format, a message field and a bytes field are identical — both are wire type 2 (length-delimited), encoding as tag + varint(length) + raw_bytes. A protobuf decoder cannot distinguish them. The etcd proto defines RangeResponse.kvs as repeated mvccpb.KeyValue (message type), but prost happily accepts repeated bytes in its place — the encoder produces field tag 2 with the same length-delimited encoding either way. The bytes inside are just a serialized KeyValue message. Any compliant protobuf client (k3s’s etcd client, etcdctl, any gRPC library) decodes them identically. This is the foundational trick: store the pre-encoded message bytes, swap the Rust type from Message to Bytes, and the wire format never changes.

Impact on prefix scans:

3c: Inline CRC and Pre-Allocated Vecs

#

Inline CRC: KvWalRecord::new previously built a temporary crc_data Vec to compute the CRC. Changed to compute CRC incrementally over the 4 header fields + kv_bytes — no temporary allocation.

Pre-allocated kvs Vec: Range responses pre-allocate Vec::with_capacity(req.limit) instead of growing dynamically. Eliminates incremental reallocation during the main iteration loop.

3d: WatchEvent.key → Bytes

#

WatchEvent is cloned per matching watcher — for 140 watchers, each event is cloned 140 times. The key: Vec<u8> was deep-copied on every clone. Changing WatchEvent.key from Vec<u8> to Bytes turned those deep copies into atomic refcount increments. Same pattern as the response field optimization, same wire-encoding identity guarantee.

Combined Result

#

Phase 4: Eliminate Polling

#

The initial implementation had two polling loops: lease expiry (every 500ms) and the periodic status task (every 60s). Both acquired locks and spent CPU even when there was nothing to do.

4a: Event-Driven Lease Expiry

#

Before:

loop {
    sleep(500ms);
    write_lock();
    for each lease:
        if expired: revoke
    write_unlock();
}

This consumed ~2% CPU in the perf profile even with zero leases, and blocked all readers/writers during the check.

After:

// Compute sleep until earliest expiry
let sleep = earliest_expiry()
    .map(|e| sleep_until(e))
    .unwrap_or(future::pending());

// Notify on lease changes:
lease_grant()    → notify()
lease_renew()    → notify()
lease_revoke()   → notify()

When no leases exist, the task sleeps indefinitely — zero CPU, zero lock acquisitions. Lease expiry latency drops from up to 500ms to microseconds.

4b: Push-Based Watches (No Polling)

#

The initial watch implementation already used push via mpsc::UnboundedSender — a deliberate design choice from day one. But the two-phase WAL replay for watcher registration was polling-adjacent: each watcher opened its own file handle and read the entire WAL.

Before (per-watcher): 140 watchers → 140 Phase 1 scans + 140 lock acquisitions, with each Phase 2 blocking writes for ~500ms.

After (cross-stream batching, P6): All watchers from all streams share a single global batch queue. A 50ms timer collects creates, then processes them as one batch:

Stream A ──create──┐
Stream B ──create──┤
Stream C ──create──┤→ global_rx → [50ms timer] → flush_global_batch()
Stream D ──create──┘                          → 1× Phase 1 scan
                                               → 1× write lock acquisition
                                               → 140× register_watcher
                                               → 140× reply via oneshot

Result:

Phase 5: Reduce Serialization (Zero-Copy gRPC)

#

This was the biggest individual optimization — P7: protobuf-native WAL. The insight: every response (Range, Watch, Put) decodes a WalRecord from disk, constructs an mvccpb::KeyValue struct in memory, then re-encodes it with prost. This produces the same bytes that were already on disk.

The Data Flow Before

#

WAL read → WalRecord.parse() → construct KeyValue → prost::encode() → gRPC
  (raw bytes)    (struct)          (struct)          (Vec<u8>)     (send)

For a Range response returning 5000 keys: 5000× struct construction + 5000× prost encoding.

What Changed: Embed Protobuf in WAL Records

#

┌──────┬───────────┬────────────────┬────────────┬──────────────────────┬──────────┐
│ flags│ key_offset│ mod_rev_offset │ rec_len(4) │ kv_bytes (protobuf  │ crc32(4) │
│  (1) │    (2)    │     (2)        │            │ KeyValue message)    │          │
└──────┴───────────┴────────────────┴────────────┴──────────────────────┴──────────┘
          └────────── 9-byte header ──────────┘

The 9-byte header gives O(1) access to key, revision, and next-record offset without any protobuf parsing. The kv_bytes payload is a valid mvccpb::KeyValue protobuf — it can be sent directly over gRPC.

Why this works: The same wire-encoding identity applies here. When tonic sends a RangeResponse, the kvs field (wire type 2, length-delimited) carries a sequence of (tag + length + bytes) chunks. Normally, prost would encode each KeyValue struct by calling KeyValue::encode() to produce those bytes at response time. With the protobuf-native WAL, those bytes already exist on disk — we just hand them to tonic as Bytes. The protobuf decoder on the client side (k3s, etcdctl) sees exactly the same byte stream it would have received if prost had encoded the struct fresh. The KeyValue message on disk is the wire format — the WAL and the network protocol share the same byte representation.

Overlong Varints for Zero-Loop Decode

#

Protobuf varints normally require a byte-by-byte loop. Since the spec permits overlong varints, we encode key_length as a 4-byte overlong varint and mod_revision as an 8-byte overlong varint:

fn read_overlong_u32(buf: &[u8]) -> u32 {
    let raw = u32::from_le_bytes(buf[..4].try_into().unwrap());
    let m = raw & 0x7f7f7f7f;
    (m & 0x7f)
        | ((m >> 1) & 0x3f80)
        | ((m >> 2) & 0x1fc000)
        | ((m >> 3) & 0xfe00000)
}

No loop, no branch — pure bit manipulation. The protobuf stays valid because any compliant decoder must accept overlong varints.

The New Data Path

#

WAL read → KvWalRecord.parse() → send kv_bytes directly as gRPC response
  (header + protobuf bytes)        (zero serialization, zero deserialization)

Combined Result (P7 + Bytes)

#

Phase 6: Stop Doing O(n) Work

#

Several operations were doing full scans of the BTreeMap when they only needed bounded ranges.

6a: BTreeMap Range for Queries

#

Before: range() and delete_range() iterated the entire BTreeMap, skipping non-matching keys. For 1000 keys with a limit of 10, all 1000 were visited.

After: Use BTreeMap::range() for From/Prefix/Range/Point queries. Only the All case does a full scan.

fn range(start: Bound<&[u8]>, end: Bound<&[u8]>, limit: usize) -> Vec<KeyValue> {
    let mut kvs = Vec::with_capacity(limit);
    for (k, ks) in state.keys.range((start, end)) {
        if ks.deleted { continue; }
        kvs.push(ks.kv_bytes.clone());
        if kvs.len() >= limit { break; }
    }
    kvs
}

O(n) → O(log n + k) for bounded queries. Combined with pre-allocated kvs Vec, limited queries stop after limit matching keys.

6b: Combine BTreeMap Lookups in Put

#

put() did 3 separate BTreeMap::get() lookups (for ignore_value, ignore_lease, and prev state) before inserting. Combined into one.

6c: Batch WAL Writes in lease_revoke

#

For N keys on a lease: previously N WAL appends + N fsyncs, all under the write lock. Changed to collect all records into a Vec and call append_kv_batch once with a single sync_all().

6d: Cache RangeBound in Watch Registration

#

notify_watchers called resolve_range for every watcher on every event notification. Changed to pre-compute RangeBound at registration time and cache it in WatchRegistration.bound.

6e: Streaming WAL Reader (No Full-File Read)

#

Before: scan_kv called self.file.read_to_end(&mut buf) allocating a Vec<u8> the size of the entire WAL file. On a 270MB WAL, that’s 270MB allocated per startup or per watch batch.

After: A streaming reader using read_exact with rec_len — reads records one at a time with record-sized allocations. Since rec_len is available in the 9-byte header, the reader jumps directly to each record without loading the entire file.

This was primarily a memory optimization (eliminating a 270MB allocation), but it also improved startup latency by starting record processing before the file was fully read.

Combined Result

#

Phase 7: Production Learnings

#

Several optimizations came from real production issues that only appear at cluster scale.

7a: WAL Compaction (284 MB → 5.1 MB)

#

The WAL grew unboundedly — at k3s workload (~72 writes/sec), ~6 MB/hour. Projected 4.5 GB in 30 days.

The tail-copy approach:

Phase A [write lock]:    Snapshot all active keys + snapshot_wal_size
Phase B [no lock]:       Write snapshot records to wal.compact
Phase C [write lock]:    Copy old WAL tail bytes → wal.compact,
                         rename wal.compact → wal, reopen

Tail copy is the critical insight: any writes that commit during Phase B are captured as raw bytes and appended verbatim to the compacted WAL in Phase C. Zero data loss, atomic rename.

Production result:

7b: Historical Revision Queries (MVCC)

#

k3s runs periodic cache consistency checks by issuing LIST(resourceVersion=N). When N referred to a compacted revision, Kudurru was returning empty response instead of ErrCompacted — causing 14,000/day cache consistency errors.

The fix combined two insights:

1. Keep deleted keys in the BTreeMap with their delete_revision — this allows conclusive determination of the result set for any target revision without scanning the WAL.
2. WAL-based reconstruction for the delete→recreate cycle (the rebirth edge case) — scan only keys where the current value isn’t valid at the target revision.

7c: The Compaction Bug That Broke Everything

#

A CRC32C range mismatch between serialize() and deserialize() meant every WAL record since the project began failed CRC verification on read. Crash recovery was completely broken — Store::open() rebuilt an empty store, ignoring all persisted data. The bug was invisible because integration tests never restart the server.

7d: The k3s Integration Bugs (Three Fixes)

#

Deploying into a real k3s cluster revealed three issues that no unit test caught:

Fix 1 — Unlimited gRPC message size: Tonic’s default max_decoding_message_size is 4MB. Kudurru’s range responses could exceed this (~119KB/key × 40 keys = ~4.8MB), causing gRPC ResourceExhausted errors. These cascaded into “Cache consistency check failed” errors in k3s. Fix: set max_decoding_message_size(usize::MAX) on all 6 gRPC services.

Fix 2 — Stale-revision tolerance in watcher creation: The initial guard rejected watchers with start_revision > current_revision() with zero tolerance. But k3s always requests start_revision = current + 1 (a normal bookmark race), so all resource-type watchers were rejected on restart, preventing scheduling entirely. Fix: changed the threshold to start_revision > current_revision + 10_000.

Fix 3 — Version string not semver: maintenance.rs returned version: "kudurru-0.1.0". k8s’s etcd feature support checker failed to parse it as semver, defaulted to etcd v3.0.0 (minimum feature set), delaying etcd health check and scheduler leader election. Fix: changed the version string to "3.5.0".

7e: malloc_trim for Memory Release

#

After a stress test deleting 2.35M keys, RSS remained at 15.5GB — glibc was caching freed mmap’d pages for reuse. Running malloc_trim(0) released them back to the OS: RSS dropped to 235MB. Added as a periodic call in the 60s status task.

7f: Graceful Shutdown

#

Ctrl+C killed the process without WAL sync. Fixed by using serve_with_shutdown(addr, ctrl_c()) — tonic drains in-flight requests, then exits. Combined with deferred fsync, the data loss window is at most 50ms.

7g: Hardware CRC32C

#

The initial CRC32C implementation was a pure-software bit loop. Replaced with the crc32c crate which uses SSE4.2 crc32q on x86-64 — ~10× faster. Applied to both WAL writing and reading.

7h: Correctness Bugs Found in Production

#

Several subtle bugs survived unit tests and only manifested in production:

Compact handler deleting live keys: The compact() RPC was calling state.keys.retain(...), which deleted current key-values from the in-memory BTreeMap. On a production cluster, compact(8000) destroyed ~97% of keys (1561 of 1623). The WAL was not modified, so a restart restored everything. Fix: removed the retain call — etcd’s Compact is an MVCC operation that must never delete current values.

eval_compare ignoring tombstoned keys: Txn CAS comparisons did state.keys.get(key) without filtering out tombstoned (deleted) entries. k3s lease-renewal CAS transactions failed because they compared against the tombstone’s mod_revision instead of the alive key’s, causing an infinite failure loop with rapid revision growth. Fix: added .filter(|k| k.is_alive()) to all five CompareTarget arms.

WAL replay not restoring lease state: On restart, WAL replay restored key data (including lease ID references) but did not restore LeaseState metadata or bump NEXT_LEASE_ID past restored IDs. This caused lease-ID collisions and missing-LeaseState failures. Fix: added a restoration loop in init_from_wal() that scans alive keys for unique non-zero lease IDs, bumps NEXT_LEASE_ID, and creates conservative 1h-TTL LeaseState entries.

Watch replay events missing PrevKv: WAL replay always set prev_kv_bytes: Bytes::new(), so all replayed PUT/DELETE events had nil PrevKv. k3s treats nil PrevKv as a fatal error when prev_kv=true is requested, causing watch teardown and retry in a tight loop. Fix: replaced the helper with inline HashMap-based tracking in both Phase 1 and Phase 2 of WAL replay.

Final Results

#

Throughput Progression

#

Initial implementation:   8.7K ops/s  ┤ (single-worker put)
P7 (zero-copy WAL):      28.3K ops/s  ┤▸▸▸▸ 3.2×
+ Bytes refactor:        29.6K ops/s  ┤▸▸▸▸▸
+ parking_lot RwLock:    30.5K ops/s  ┤▸▸▸▸▸
+ BTreeMap range:        30.5K ops/s  ┤ (structural, not throughput)
+ Deferred fsync:        26.5K ops/s  ┤ (same — single-worker dominated)
  ─── concurrent peak ───
+ P7 zero-copy:         101K ops/s @8w┤▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸ (+38%)
+ Bytes + parking_lot:  103K ops/s @8w┤▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸▸ (+42%)

Final Benchmark (i9-11950H @ 2.6 GHz, localhost)

#

vs kine/SQLite (30-pod k3s cluster)

#

Production Validation

#

The production cluster — 214 days of uptime, 30+ workloads — was migrated from kine+SQLite to Kudurru via a migration tool that transferred 1,501 keys in 9 seconds using etcd v3 CAS transactions. Post-migration CPU dropped from ~61% of a core (kine + Go runtime + CGo) to 0.10% (Kudurru in Rust) — a ~610× improvement in CPU efficiency on the same hardware.

A stress test validated 2.35M operations from a remote worker over 1Gb LAN, sustaining 29,413 ops/s with zero errors. Kudurru’s own code consumed ~2.6% of CPU at saturation, confirming the storage layer is not the bottleneck — the gRPC/Tokio infrastructure is.

WAL compaction in production: 284 MB → 5.1 MB (98% reduction) in 52ms, running every ~10 hours with zero observable impact on write latency.

Storage Layer CPU Breakdown (perf profile)

#

The remaining ~96% is tokio runtime, kernel, and gRPC framing — the unavoidable cost of any async gRPC server. The storage layer is no longer the bottleneck.

Key Lessons

#

1. Profile before optimizing — then profile again

#

The initial perf profile showed the storage layer was only ~3.8% of CPU. The real bottlenecks were lock contention, allocation churn, and serialization — things that don’t show up in microbenchmarks. Without the profile, we would have wasted effort on BTreeMap tuning (4.09% of CPU in memcmp was already small).

2. The most impactful optimizations are architectural — not micro

#

The P7 protobuf-native WAL (zero-copy gRPC) delivered 2.4× latency improvement and 3.2× throughput gain. Deferred fsync eliminated ~1000× read-side blocking. BTreeMap range queries turned O(n) scans into O(log n + k). These architectural changes dwarfed any micro-optimization.

3. Eliminate polling everywhere

#

Polling was the root cause of 2% CPU, 500ms lease expiry latency, 1-second watch latency, and 140 unnecessary WAL reads at startup. Each was replaced with an event-driven equivalent: sleep_until(earliest_expiry), mpsc push channels, and batched registration. The pattern is universal: if you’re waking up to check if something happened, you’re wasting CPU.

4. Data movement is the real cost

#

The perf profile showed memmove at 3.42% and malloc/free at 3.6% — together more than the entire storage layer. Moving data from WAL → struct → protobuf (decode → transform → re-encode) was the dominant cost per operation. The zero-copy WAL eliminated this entirely: the bytes written to the WAL are the bytes sent over gRPC.

5. Production teaches what benchmarks can’t

#

The compact handler that deleted 97% of live keys. The CRC32C range mismatch that silently broke crash recovery since day one. The Txn comparison that didn’t filter tombstones, causing infinite CAS failure loops. The PrevKv=nil bug that sent watchers into reconnect storms. The 4MB gRPC message limit that truncated production LIST responses. None of these appeared in unit tests or benchmarks. Production is the only profiler that matters for correctness.

The corollary: build recovery tools early. When the compact bug deleted 97% of keys in production, three CLI tools (waldoctor, walrecover, walverify) were built to validate the WAL, reconstruct state, and produce a clean recovery WAL — 97,099 records with 0 CRC errors. Without them, the data loss would have required a full cluster rebuild.

6. Rust’s ownership model makes optimization safe

#

Changing Vec<u8> to Arc<[u8]> to Bytes — each step eliminated a memcpy without introducing use-after-free or data races. The compiler enforced correctness, letting us aggressively share buffers across the store → response path without fear.

7. Know when to stop

#

The final perf profile shows a server that is fundamentally Tokio-runtime-bound — ~50% of cycles go to the async infrastructure. The storage layer is ~3.8%. The theoretical peak throughput of the single RwLock (~100K ops/s) is 3 orders of magnitude above the target k3s workload (~72 ops/sec). Further optimization would yield zero observable benefit. The project’s concluding observation:

“13 of 16 identified optimization items were implemented. 3 were skipped as not providing meaningful benefit at the k3s workload scale. No further storage-layer optimizations are warranted for the target k3s workload (~72 writes/sec, 30 pods). The single RwLock remains the theoretical peak-throughput bottleneck (~100K ops/s), which is 3 orders of magnitude above the target.”

8. The final recommendation

#

From the project’s concluding PRD:

“The server is Tokio-runtime-bound, not CPU-bound or I/O-bound. ~50% of CPU cycles go to the async runtime infrastructure (polling futures, scheduling tasks, parking workers, processing timers). Kudurru’s own code accounts for <3%. The storage layer is already well-optimized at ~3.8% CPU. Stop optimizing.”

This optimization journey was assisted by DeepSeek V4 Flash on OpenCode — an interactive CLI agent that performed the iterative analysis, benchmark execution, and code transformations described above. The project evolved from initial concept to production deployment in approximately 11 days (June 13–24, 2026), with the core storage layer and gRPC protocol implementation completed in the first 24 hours and the remaining 10 days spent on production hardening, testing, and iterative optimization.