Knowing what happened

Resilience patterns decide how a service behaves under stress. This lesson is about being able to tell that it did.

Averages hide the outage

Average latency is the wrong headline metric because it is dominated by the many fast requests. Ten thousand requests at 20ms and a hundred at 8 seconds average out to under 100ms — a number that looks fine while a hundred people are watching a spinner.

Percentiles answer the question you actually have:

	What it tells you
p50	the typical experience
p95 / p99	the experience of the users who are having a bad time
max	usually a timeout, and worth knowing which

Watch p99. And be careful averaging percentiles across instances — the average of two p99s is not a p99 of anything. Histograms aggregate correctly; pre-computed percentiles do not.

Cardinality is a cost, and user ids are the classic mistake

A metric is a time series per unique label combination. Label a counter with the user id and you have created one time series per user — hundreds of thousands of them, each with its own retention, each consuming memory in the exporter and the backend.

The result is not a nice graph. It is a metrics system that falls over, and a bill to match. Same for request id, session id, full URL paths with ids in them, and error messages that contain values.

Labels are for things with a small, bounded set of values: status code, method, endpoint template (/orders/{id}, not /orders/8412), instance. When you need per-user detail, that is what logs and traces are for — they are keyed by time, not by cardinality.

Logs, traces and what each is good at

Structured logging means emitting a log line as key–value data rather than a sentence. The practical gain is that status=500 tenant=acme duration_ms=8123 can be queried, aggregated and alerted on. A sentence has to be parsed with a regular expression that breaks the first time somebody rewords the message.

A correlation or trace id in every line is what makes those queries useful. Without it you have a thousand lines from a hundred concurrent requests interleaved, and no way to say which belong together. With it, one field turns the pile into a story.

Traces give you what metrics and logs cannot: the shape of one request across services, with the timing of each hop. Metrics tell you the p99 got worse; the trace tells you which of the six calls in that request grew, and whether it grew because it was called more times or because each call got slower.

Trace context travels over HTTP as headers — traceparent and tracestate under W3C Trace Context. Every service must read them from the incoming request and pass them on outgoing ones. Miss that in one service and the trace breaks there: you get two disconnected traces and no link between them, which is worse than none because it looks complete.

Probes are two different questions

Probe	Question	On failure
liveness	is this process wedged?	the container is restarted
readiness	can it serve traffic right now?	it is removed from the load balancer

Conflating them is a common and painful mistake. If a liveness probe checks the database, then a database blip restarts every instance of your service — converting a dependency problem into an outage of your own. Liveness should check that the process itself is alive. Readiness is where dependencies belong.

Graceful shutdown

On SIGTERM, a service should: stop accepting new work, finish what is in flight, close resources, and only then exit. It must also be removed from routing before it stops accepting — otherwise the load balancer keeps sending requests to a socket that is closing, and clients see resets.

The usual sequence is fail readiness first, wait one probe interval, then drain. The ordering matters more than the timeout value.

SLI, SLO and the error budget

• SLI — the measurement: “proportion of requests served in under 300ms”.
• SLO — the target for that measurement: “99.9% over 30 days”.
• Error budget — what the target permits you to spend: 0.1%, which over 30 days is about 43 minutes.

The budget is the point. It converts reliability from an argument into an arithmetic: budget left means you can ship, deploy on a Friday, run the risky migration. Budget spent means the next work is reliability work, and that is a decision the number makes rather than the loudest person in the room.

A 100% target has no budget, which means no changes — which is why nobody credible sets one.

JDK Flight Recorder

JFR is designed to be on in production: it is built into the JVM, and at default settings costs a low single-digit percentage of throughput. That is what separates it from a profiler you attach during an incident — the events you need are already recorded when the incident starts, rather than beginning to be collected once somebody notices.

---

The instrumentation in code, and the incidents that shaped these choices: Surviving Production Java.