↓ Skip to main content

A Clock 4 Minutes Behind: Rolling NTP Time Sync Across the Home LAN

·1273 words·7 mins
Max Severance
Author
Max Severance
Architecting fabric convergence at hyperscale.

TL;DR
#

A headless Debian box was ~4 minutes behind real time, and nothing on it was syncing — no NTP, no chrony, no systemd-timesyncd. The obvious fix was just to turn one of them on, but that opened a bigger question: with a LAN full of Debian boxes, why not have one accurate box serve time to the rest? I installed chrony on the server (changwang, 10.222.1.22), configured it to serve the 10.222.1.0/24 LAN, then installed chrony as a client on two other machines (precision and odroid at 10.222.1.40), pointing them at changwang and disabling their old systemd-timesyncd. All three now sync from changwang, which itself syncs from the NTP pool. The clock is back to ~1 ms of real time instead of 4 minutes off.


The Setup
#

  • changwang (10.222.1.22) — a headless Debian (forky/sid) box, the one with the drifted clock
  • precision — another Debian box, running systemd-timesyncd
  • odroid (10.222.1.40) — the third box, also running systemd-timesyncd
  • All on the 10.222.1.0/24 LAN

The whole thing started as a casual observation: “is there time syncing set up in this box? i notice time is about 4 minutes behind.”

The Symptom
#

date -u on changwang showed the clock about 4 minutes behind UTC. Four minutes is a lot — that’s not jitter or NTP polling lag, that’s a clock that has been free-running for a long time with no correction at all. On a machine that terminates TLS, signs things, and writes file timestamps, four minutes off is enough to start causing real problems.

Diagnosing: Nothing Was Syncing
#

The first check was whether any time sync was configured or running:

$ systemctl list-units --type=service | grep -iE "ntp|time"
  containerd.service      loaded active running   container runtime
  user-runtime-dir@0      loaded active exited    User Runtime Directory
  ...

Nothing. No systemd-timesyncd, no NTP daemon, no chrony. There was no /etc/ntp.conf, no /etc/chrony/, and which found none of the usual clients installed:

$ which ntpdate ntp sync systemd-timesyncd
/usr/bin/sync      <- the only hit, and that's `fsync`, not time

So the box had literally nothing to keep time. The 4-minute drift wasn’t a bug — it was the natural result of a clock with no reference for however long it had been up.

Choosing: Chrony Over systemd-timesyncd
#

With no sync running, the cheapest fix is systemd’s built-in systemd-timesyncd (one service to start). But for a server, chrony wins on every axis that matters:

  1. Accuracy — chrony continuously models the clock drift and network latency; systemd-timesyncd does coarse periodic polls. On a busy LAN or NAT’d network, that is the difference between sub-millisecond and minutes-per-day.
  2. Gradual correction — chronyd slews the clock over time instead of stepping it in big jumps. A hard step confuses logs, TLS handshakes, and file timestamps; a slew is invisible.
  3. Better behavior around boot/offline — chrony can estimate drift and remember it (driftfile), so it’s reasonably close even before it can reach a server after boot.
  4. It can serve time too — this is the one that turned a one-box fix into a LAN project. chronyd is an NTP client and server. systemd-timesyncd is a client only.

Once you have a box that’s already the natural hub of a LAN of Debian boxes, the logic is simple: let it serve time, and point the others at it.

Installing Chrony on the Server
#

$ sudo apt install -y chrony

That pulled in chrony 4.8, created _chrony user/group, and symlinked the systemd unit. It was running and enabled immediately:

$ sudo systemctl status chrony
● chrony.service - chrony, an NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chrony.service; enabled)
     Active: active (running)

The stock /etc/chrony/chrony.conf already pointed at the Debian NTP pool (pool 2.debian.pool.ntp.org iburst), so within seconds it had a stratum-2 source locked and was pulling the clock in:

$ sudo chronyc sources
^* ntp1.eenet.ee   2   6   17   3   +56us[+479us] +/- 4285us

chronyc tracking showed the payoff immediately — the ~4-minute error collapsing toward an offset of 0.4 ms:

Last offset     : +0.000423718 seconds
RMS offset      : 0.000423718 seconds

Making It Serve the LAN
#

By default chrony only answers the servers it syncs from, not arbitrary clients. Serving the LAN needs an allow directive scoped to the network. I dropped it into the conf.d drop-in directory:

# /etc/chrony/conf.d/allow.conf
allow 10.222.1.0/24

Then a restart. Now changwang both receives time from the pool and serves it to anything on 10.222.1.0/24.

Clients: precision and odroid
#

The two clients both ran systemd-timesyncd. The plan for each:

  1. Install chrony (sudo apt install -y chrony)
  2. Stop and disable systemd-timesyncd (so they don’t fight)
  3. Write a chrony config pointing at the server: server 10.222.1.22
  4. Enable chrony
sudo systemctl stop systemd-timesyncd
sudo systemctl disable systemd-timesyncd
sudo systemctl enable --now chrony

The install step actually gave us a head start — apt enabled chrony on install, so it came up running. But there was a wrinkle: sudo systemctl disable systemd-timesyncd reported:

Failed to stop systemd-timesyncd.service: Unit systemd-timesyncd.service not loaded.

That’s because apt’s chrony install had already kicked systemd-timesyncd off the boot path (its enable symlink was removed as part of the deb) — so the unit wasn’t loaded to stop. Fine, that’s the outcome we wanted anyway; the disable just confirmed it.

Also worth noting: on the odroid box, a typo crept into the first config write — driftdate instead of driftfile, and maxuserskew instead of maxupdateskew. Chrony will complain or silently misbehave on those. Rewriting the config cleanly fixed it. Watch for copy/paste slop in generated configs; a restart plus chronyc sources confirms you’re healthy.

Verification
#

The proof is in chronyc sources on the clients — they should list changwang’s address (10.222.1.22) as their only source, at stratum 2 (a client of the pool):

On precision:

^? 10.222.1.22   2   6   3   1  +1276us[+1276us] +/- 6781us

On odroid (10.222.1.40):

^? 10.222.1.22   2   6   3   1   -916us[ -916us] +/- 6819us

Both show changwang as the stratum-2 source, with offsets around 1 ms — a far cry from 4 minutes behind. All three boxes now start chrony on boot. changwang holds the pool link, the clients hold changwang, and the whole LAN is within a couple of milliseconds of each other.

The Final Layout
#

NTP pool (debian.pool.ntp.org)
  └── changwang (10.222.1.22)   [server + client]
        ├── precision           [client]
        └── odroid (10.222.1.40) [client]

Learnings
#

  1. Check for sync before assuming the clock is “fine.” A 4-minute drift is a symptom that there’s no sync at all. On a headless box it’s easy to have zero time services configured and never notice until a timestamp comparison bites you.
  2. For a server, prefer chrony over systemd-timesyncd. Continuous slew over periodic steps, higher accuracy, and it can serve time to the rest of the LAN. The client/server dual role is what turns a one-box fix into a small infrastructure improvement.
  3. One well-synced box can anchor the whole LAN. Rather than pointing every box at the public pool, let one reliable hub do the internet-facing sync and the others sync from it. Fewer moving parts hitting the WAN, and the LAN stays internally consistent to the millisecond.
  4. A config typo can silently take you down. driftdate vs driftfile in chrony.conf won’t always scream at you. After any edit, sudo chronyc sources and sudo chronyc tracking are your confirmation — the numbers don’t lie.
  5. chrony installs enabled. The Debian chrony package wires up its systemd unit and starts it; sometimes the only thing you have to fight is the old client, and even that steps aside on its own.

This whole session — from noticing the drift to configuring three boxes — was assisted by DeepSeek V4 Flash on OpenCode, the interactive CLI agent that performed the diagnosis, config, and verification above.