SRE when latency matters as much as uptime

In my first post I wrote that I now run infrastructure for a trading firm on AWS, where latency matters as much as uptime. This post is about what that sentence means in practice.

Most SRE material is written for web services. There, the main question is “is it up?” and latency is a secondary concern. In a trading-style environment both questions carry the same weight. That changes a lot of everyday decisions, and some standard advice turns out to be wrong.

Nothing below is specific to any one firm. The examples are hypothetical.

1. A slow system is a down system

On a typical web service, a request that takes 800 ms instead of 200 ms is annoying. The user waits. The request still succeeds.

In a latency-sensitive system, a late answer can be worth nothing. Say a system reacts to a price change. If it reacts after the price has moved again, the work was wasted, or worse. From the business side, that request failed, even though every health check was green.

So the first change is in the definition of “working”. An availability SLO alone is not enough. (An SLO, service level objective, is the target you agree to meet, such as “99.9% of requests succeed”.) You also need a latency SLO, and it has to be on the tail:

Not “average latency under 50 µs”.
Instead, something like “p99.9 under X, measured at this point, over this window”.

The tail is where the damage is. A good average can hide a system that stalls for milliseconds once a second. I covered why in How to measure latency without fooling yourself.

A practical way to frame it: count a request that misses the latency target as a failed request. Then one error budget covers both. A host that answers every request, but slowly, burns budget just like a host that drops them.

2. Redundancy costs latency

The standard reliability toolkit adds things: more replicas, more hops, more checks. Each one has a latency price.

Load balancers and proxies. Every extra hop adds time and a new place for queueing.
Synchronous replication. Writing to two places before you reply means waiting for the slower one, plus the network between them.
Cross-zone or cross-region placement. Spreading out protects against a zone failure. It also puts distance, and so time, between components that talk to each other.
Health checks and heartbeats. They use CPU, network and sometimes locks on the very hosts you care about.

None of this means “skip redundancy”. It means you choose it on purpose, path by path.

Say you have a hot path that must be fast and a control path that must be correct. You might keep the hot path short and close together, accept that a zone failure takes it out, and have a tested way to bring it up elsewhere. The control path can afford sync replication and more hops.

Also look at failover itself. A failover that takes 30 seconds, then serves from a farther location with cold caches, may meet the availability SLO and still break the latency one. Measure latency after failover, not just whether it completed.

3. Change management around the busy hours

In a web shop, deploying many times a day in business hours is a sign of maturity. In a latency-sensitive system the risk is shaped differently. A change that adds 20 µs to the tail is not an outage on any dashboard, but it matters.

What tends to change:

Change windows. Risky changes go outside the hours when the system is busiest. For trading, that usually means outside market hours. The exact windows depend on which markets you touch.
Promotion, not big bangs. The same commit goes dev, then stg, then prd. Each stage is measured before the next. That is the flow in Making Linux host tuning repeatable with Ansible.
Latency is part of the release check. “Tests pass” is not enough. Compare the tail before and after, on the same hardware, under the same load.
Rollback beats fix-forward. During busy hours you don’t debug in production. If a change looks wrong, go back to the last known-good state, then work out why. This only works if rollback is fast and has been tested.

The less obvious part: small changes count. A kernel update, a new agent, a library bump, a change in instance type. Any of them can move the tail. Treat them as changes, not housekeeping.

4. Headroom and noisy neighbours

Queueing theory has one lesson that matters here: as a resource gets close to full, waiting time grows fast, not linearly. A CPU at 50% and one at 85% may both look “fine” on a utilisation graph. The second one will have a much worse tail.

So capacity planning changes:

Plan for the tail, not for the average load. Size so that the busy minutes still leave headroom. Peaks in trading are often short and sharp.
Utilisation targets are lower. Running hot is a cost saving that you pay back in tail latency.
Noisy neighbours are a capacity problem. Anything sharing the host can steal time: other processes, the kernel, backup jobs, log shipping. In the cloud it can also be other tenants on shared hardware.

Ways to reduce the noise: isolate CPUs for the critical work, keep batch jobs off those hosts, and pick instance types where you are not sharing cores. The tuning checklist covers the host side.

Cost pressure works against all of this. Be explicit about it: idle headroom on a latency-critical host is not waste. It is what you are paying for.

5. Alert on the tail and on jitter

A typical alert set looks at errors, saturation and maybe median latency. For a latency-sensitive system that misses most of what goes wrong.

Things worth alerting on:

p99 and p99.9 latency against the SLO, with enough samples per window for the number to mean something.
Jitter. That is how much latency varies, not just how high it is. A system that is usually 10 µs but sometimes 500 µs is often worse to live with than one that is always 40 µs.
Change from baseline. A tail that crept up 30% since last week, with no alert threshold crossed, is still news.
Leading signals. Run-queue length, context switches on isolated cores, interrupt counts, packet drops, clock offset. These often move before the latency does.

Keep the usual rule: alert on what users feel, and make every alert actionable. Here, “users feel” includes slowness.

6. Fail fast or degrade?

In web services, graceful degradation is usually good. Serve stale data, drop a feature, keep the page up.

In a latency-sensitive system, the right answer depends on what a late or partial answer means. Say a component can’t get fresh data in time. Two options:

Degrade: carry on with what it has. The system stays “up”, but acts on stale input.
Fail fast: stop, reject, or step back, and say so clearly.

Often fail fast is the safer choice. A stopped system is obvious. A system quietly acting on stale data can do damage for a long time before anyone notices.

The same applies to retries and timeouts. A retry that succeeds after 200 ms may be worse than a quick failure. Set timeouts from the latency budget, not from a default. Be careful with retries, because they add load at the worst moment.

Decide this ahead of time, per component, with the people who own the business side. Write it down. It is not a call to make during an incident.

7. Observability has a cost

Every measurement takes time. On a web service that cost is lost in the noise. On a microsecond path it is not.

Logging in the hot path can block on I/O or allocate memory.
Tracing adds work to every request it touches.
Agents for metrics, security or logs take CPU, wake up on timers, and cause interrupts, sometimes on the cores you isolated.
High-resolution scraping adds load on exactly the hosts you are trying to keep quiet.

Some ways to keep it cheap:

Record into in-memory histograms on the hot path and export them from somewhere else.
Move log formatting and shipping off the critical cores and threads.
Sample traces instead of tracing everything.
Pin agents to housekeeping cores.
Measure the system with and without the agent. If you can’t tell the difference, good. If you can, you now know the price.

8. Incidents where “degraded” is the outage

Many incident processes have an unspoken rule: if it’s up, it’s not urgent. In a latency-sensitive system that rule is wrong.

What changes:

Severity includes latency. Define ahead of time what tail latency counts as a high-severity incident, the same way you would for an error rate.
Declare early. Latency problems are often fuzzy at the start. “It feels slower” is a fair reason to open an incident and look.
First move is usually to undo. What changed recently? Roll it back. Move load off the bad host. Remove it from service. Diagnose later.
Keep the evidence. Latency problems are hard to reproduce. Capture histograms, the timeline, and host state before you restart things.
Review it like an outage. A blameless review of a latency incident is just as useful as one for downtime. Ask why the alerts didn’t fire sooner and what the tail looked like before anyone noticed.

Summary

Count a late answer as a failed answer. Put latency SLOs on the tail next to availability.
Every bit of redundancy has a latency cost. Choose it per path, and measure latency after failover.
Change outside busy hours, promote dev to stg to prd, check the tail before and after, and roll back first.
Keep real headroom and treat noisy neighbours as a capacity problem.
Alert on p99/p99.9, jitter and drift from baseline, not only errors.
Decide fail fast vs degrade in advance, per component.
Measure what your observability costs and keep it off the hot path.
Treat “slow” as an incident, not a curiosity.

None of this replaces the basics from the first post. It just adds one more question to every decision: what does this do to the tail?

References

Google, Site Reliability Engineering, chapter 4 “Service Level Objectives” and chapter 6 “Monitoring Distributed Systems”: https://sre.google/sre-book/table-of-contents/
Google, The Site Reliability Workbook, “Implementing SLOs” and “Alerting on SLOs”: https://sre.google/workbook/table-of-contents/
Jeffrey Dean and Luiz André Barroso, “The Tail at Scale”, Communications of the ACM, 2013: https://research.google/pubs/the-tail-at-scale/
Gil Tene, “How NOT to Measure Latency” (talk): https://www.infoq.com/presentations/latency-response-time/
AWS Well-Architected Framework, Reliability and Performance Efficiency pillars: https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html