Eric Wang
RSS

Writing

Time sync on AWS: why microseconds of clock offset matter

/10 min read/Reliability/#aws/#linux/#time-sync/#chrony/#ptp/#latency/#measurement

In How to measure latency without fooling yourself I said: if you take a timestamp on one host and another on a different host, the clock offset between them is in your number. This post is about keeping that offset small, and knowing how big it is.

I run infrastructure for a trading firm on AWS, where the tail matters more than the average. Clocks are part of the tail.

1. Why microseconds matter

Say host A stamps a message at send and host B stamps it at receive. B’s clock is 40 µs ahead of A’s. Every one-way latency you compute is now 40 µs too high. If the real latency is 30 µs, your dashboard shows 70 µs, and you go tuning the wrong thing. If B is 40 µs behind, you get negative latency, which at least is obviously wrong.

That is the measurement case. There are others:

  • Event ordering. Two hosts log events 10 µs apart. If the clocks disagree by more than 10 µs, you can’t say which came first from timestamps alone.
  • Log correlation. During an incident you merge logs from many hosts into one timeline. A bad clock puts the effect before the cause and sends you down the wrong path.
  • Audit and regulatory timestamps. Trading venues and firms in some jurisdictions must keep business clocks within a set tolerance of UTC and be able to show it. In the EU this is MiFID II RTS 25; the required accuracy depends on the type of activity. The point for operations: you need not just a good clock, but evidence that it was good, kept over time.
  • AWS itself. AWS CLI and SDK requests are signed with a time. A clock that is far off gets requests rejected.

Rule of thumb: if you compare timestamps from different hosts, the clock error has to be well below the gap you care about.

2. The Amazon Time Sync Service (NTP)

Check what you are syncing to:

chronyc sources -v

On EC2 you want to see the link-local Time Sync endpoint marked ^* (the selected source):

  • IPv4: 169.254.169.123
  • IPv6: fd00:ec2::123 (Nitro-based instances only)

These addresses are reachable from every instance without any VPC change, and traffic stays local. Amazon Linux 2023 and recent Amazon Linux 2 already use the IPv4 endpoint. If yours doesn’t, put this in /etc/chrony.conf before any other server or pool lines, as AWS documents:

server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

What the options mean:

  • prefer: pick this source over others when they agree.
  • iburst: send a quick burst at startup so chrony syncs in seconds, not minutes.
  • minpoll 4 maxpoll 4: poll every 2^4 = 16 seconds. Frequent polls track drift more closely.

Two things to know:

  • Link-local services share a limit of 1024 packets per second per interface, across DNS, instance metadata, NTP and others. Polling every 16 s is nowhere near it, but a misbehaving agent hammering IMDS can starve the rest.
  • There is also a public endpoint, time.aws.com, for backup or for machines outside EC2. Like the local one, it smears leap seconds (section 6).

3. Enhanced time and the PTP hardware clock

AWS has a more accurate option on newer instances. Launch into a placement group with the precision-time strategy:

aws ec2 create-placement-group \
  --group-name my-precision-time-pg \
  --strategy precision-time

Instances in that group run on hardware with direct access to the high-precision time source. You get two things:

  1. A better NTP source with no config change. The same 169.254.169.123 endpoint now shows up as stratum 1 in chronyc sources.
  2. A PTP hardware clock (PHC) on Linux. The Nitro card exposes a clock device through the ENA network driver. Reading a local device avoids the network jitter of NTP packets, so chrony can sync much tighter. AWS describes this as microsecond-accurate.

The supported families are Gen7 and later (for example M7i, C7i, R7g and the 8th-generation families); the list changes, so check the docs before you pick an instance type. Launches fail if AWS has no suitable hardware in that AZ, so plan for retries or another AZ. AWS says the placement group itself costs nothing extra.

To use the PHC you need ENA driver 2.10.0 or later, built and loaded with PHC enabled. Check:

cat /sys/module/ena/parameters/phc_enable      # 1 = enabled
for f in /sys/class/ptp/*; do echo -n "$f: "; cat "$f/clock_name"; done
ls -l /dev/ptp*

You want a clock named ena-ptp-<PCI slot>. The device number (/dev/ptp0, /dev/ptp1, …) depends on probe order, so don’t hard-code it. Recent Amazon Linux 2023 AMIs ship a udev rule that creates a stable /dev/ptp_ena symlink. On other distros, add one (the AWS docs give the rule).

Then point chrony at it:

refclock PHC /dev/ptp_ena poll 0 delay 0.000010 prefer

refclock PHC reads a hardware clock instead of asking a server. poll 0 reads it every second. delay tells chrony how much uncertainty to assume for each reading.

After a restart, chronyc sources should show PHC0 as the selected source (#*) at stratum 0, with the NTP endpoint still listed as a backup.

The PHC also gives you hardware packet timestamping: the NIC stamps each incoming packet in hardware, before the kernel touches it. Check with ethtool -T <interface>. That is the cleanest way to take receive timestamps for one-way latency, because software delays are not in the number.

4. chrony settings that matter

A minimal /etc/chrony.conf for a latency-sensitive host:

# PHC on supported instances
refclock PHC /dev/ptp_ena poll 0 delay 0.000010 prefer
# Link-local NTP
server 169.254.169.123 iburst minpoll 4 maxpoll 4

# Step the clock only at startup, then slew
makestep 1.0 3

driftfile /var/lib/chrony/drift
rtcsync

Without a PHC, drop the refclock line and put prefer on the server line.

makestep 1.0 3 means: if the clock is off by more than one second, jump it, but only during the first three updates. After that chrony only slews: it runs the clock slightly fast or slow until the error is gone. Steps are bad on a running system. Time jumps backwards, durations come out negative, timers fire early or late. So allow a step at boot and never later.

Keep the config in code. The Ansible post shows the pattern: template the file, restart the service, then assert the expected source is selected.

5. Checking offset: what the numbers mean

chronyc tracking
chronyc sources -v
chronyc sourcestats

In chronyc tracking, look at:

  • System time: how far the system clock is from chrony’s best estimate right now.
  • Last offset / RMS offset: the recent correction size and its long-term average.
  • Root delay / Root dispersion: the network delay and accumulated error back to the reference clock.
  • Leap status: should be Normal.

A low offset alone does not prove the clock is right. It says the clock agrees with its source. The honest question is: what is the maximum error versus true UTC?

That is what AWS ClockBound answers. It is an open-source daemon plus client library. For every timestamp it returns a window, earliest and latest, and true time is inside it. The width of that window is the clock error bound. For NTP sources it is roughly:

|local offset| + root dispersion + root delay / 2

Why this matters for ordering: say event X on host A has the window [100, 140] µs and event Y on host B has [150, 190] µs. The windows don’t overlap, so X happened first. If they overlap, the honest answer is “can’t tell from clocks”. ClockBound also reports whether the clock is synchronised or free-running, so a host that lost its source can mark its timestamps as unreliable instead of quietly drifting.

For latency measurement, the same idea: a one-way latency of 30 µs measured between two hosts with error bounds of 50 µs each tells you very little. Report the bound next to the number.

6. Leap seconds and smearing

A leap second is a one-second adjustment to UTC. Some clocks handle it by repeating or inserting a second. AWS’s NTP service handles it by smearing: spreading the extra second over a long window so the clock never jumps.

The catch, straight from the AWS docs:

  • The NTP endpoints (local and time.aws.com) smear.
  • The PHC does not smear. It follows UTC and adds the leap second as a step.
  • AWS recommends not mixing smeared and non-smeared sources during a leap second event.

Smeared and non-smeared time agree almost all the time. During a smear they differ by up to a second. A host mixing them, or two hosts on different source types, will disagree by far more than any microsecond budget.

Practical rules:

  • Pick one kind per fleet. Hosts that compare timestamps should use the same type of source.
  • Don’t add public non-smearing NTP pools next to the AWS sources.
  • If you use the PHC with NTP as backup, have a plan for a leap event (for example, take the NTP source out for that window).
  • In audit records, note which timescale you used.

Leap seconds are due to be dropped by or before 2035, and none has been added since 2016. That makes this rare, not impossible. AWS says it still plans to smear any that happen.

7. Monitor and alert on it

Time sync fails quietly. chrony keeps running, the clock keeps ticking, and nothing errors. So watch it like any other SLO.

What to collect per host:

  • System time offset and RMS offset from chronyc tracking
  • Selected source and its stratum from chronyc sources (alert if it changes from PHC to NTP, or to nothing)
  • Leap status
  • The ClockBound error bound and status, if you run it
  • Whether chronyd is running at all

A quick check for a script or exporter:

chronyc -c tracking | awk -F, '{print "offset_s=" $5, "rms_s=" $7, "leap=" $NF}'

-c gives comma-separated output that is easy to parse. Verify the field positions against your chrony version before you trust them.

What to alert on:

  • Offset or error bound above your budget. Set the budget from what you compare. If you measure one-way latency in tens of microseconds, the budget is a few microseconds, not “under a millisecond”.
  • Source lost or changed.
  • A step after boot.
  • Leap status not Normal.

And keep the history. For audit purposes, the time series of offset and error bound is your evidence that clocks were in tolerance. Store it with the same retention as the records it backs.

This ties back to SRE when latency matters as much as uptime: a host with a bad clock is producing wrong numbers. Treat that as a degraded host, not as a curiosity.

8. How this ties into measuring latency

  • Same host, same clock: use CLOCK_MONOTONIC for durations. It never jumps, and sync quality barely matters.
  • Across hosts: you need wall-clock time, so the clock error of both hosts is in your result. Add the two error bounds; that is your uncertainty.
  • Best case: hardware receive timestamps on the PHC, both hosts on the PHC, same placement-group setup.
  • Before/after comparisons: check that sync was healthy for both runs. A tuning “win” that is actually a clock change is the worst kind of result.
  • When in doubt: measure round trips on one host and halve them. It is less precise, but it doesn’t depend on two clocks agreeing.

Summary

  • Use the link-local Time Sync endpoint (169.254.169.123 or fd00:ec2::123) with chrony.
  • For tighter sync, use a precision-time placement group on a supported instance and add the ENA PHC as refclock PHC /dev/ptp_ena.
  • Allow makestep only at boot; slew after that.
  • Look at the error bound, not just the offset. ClockBound gives you one.
  • Don’t mix smeared (NTP) and non-smeared (PHC, public pools) sources during a leap second, and keep a fleet consistent.
  • Monitor offset, error bound, source and leap status. Alert on them, and keep the history.
  • Any cross-host latency number carries both clocks’ error. Report it.

References

  • How to measure latency without fooling yourself

    Averages hide the tail, load generators hide stalls, and clocks drift. A practical guide to measuring latency honestly: percentiles, histograms, coordinated omission, timestamps, warm-up, noise and fair before/after comparisons.

    12 min read/#linux/#performance/#latency

  • A low-latency Linux host tuning checklist

    A practical checklist for tuning Linux hosts for low, predictable latency: what each setting does, how to check it, how to set it, and what it costs. Measure first.

    13 min read/#linux/#performance/#latency

  • Making Linux host tuning repeatable with Ansible

    Hand-tuned hosts drift. How to put GRUB parameters, sysctls, tuned profiles, IRQ affinity and HugePages into an Ansible role, reboot safely, check the live state after boot, and catch drift before it shows up in your tail latency.

    13 min read/#linux/#performance/#latency