Observability for Debuggable Systems
How logs, metrics, traces, correlation IDs, and cardinality limits make production failures explainable.
Observability is the ability to answer new questions about a running system without shipping a special debug build. It is not the volume of telemetry. It is the quality of the evidence emitted at the boundaries where behavior changes.
Three signal types
| Signal | Best at |
|---|---|
| logs | discrete facts and failure context |
| metrics | aggregate rates, counts, latency, saturation |
| traces | request path, causality, and timing across components |
Use all three intentionally. A log line explains one event. A metric shows whether the event is widespread. A trace shows where time and errors accumulated in one request.
Correlation
Every request should carry a stable correlation identifier. In tracing systems, spans share a trace ID and each span represents a unit of work. Parent-child links show causality across service boundaries.
A useful span records:
- operation name,
- start and end time,
- status,
- stable identifiers,
- dependency name,
- bounded attributes,
- important events.
Do not put secrets or high-cardinality unbounded values into attributes.
Metrics and cardinality
Metric cardinality is the number of unique attribute combinations. High-cardinality labels such as user ID, raw path, email, request ID, or full error string can make metrics expensive or unusable.
For attributes $A_1, A_2, \dots, A_n$ with distinct counts $c_i$, the worst-case series count is:
\[\prod_{i=1}^{n} c_i\]Two moderate-cardinality labels can multiply into a large storage and memory cost. Prefer route templates over raw paths and error classes over full messages.
Debuggable boundaries
Instrument boundaries where state changes or uncertainty enters:
- request accepted,
- auth decision,
- queue admission or shed,
- dependency call,
- retry decision,
- database transaction,
- external side effect,
- background task start and finish,
- graceful shutdown phase.
The log should include what happened, where, stable IDs, and the reason. It should not require reading source to distinguish timeout, cancellation, validation, overload, and dependency rejection.
Practical checks
- Can one request be followed across services?
- Can an operator separate user error, dependency error, overload, and bug?
- Do latency histograms expose p95 and p99?
- Are queue depth, saturation, retry count, and shed count measured?
- Do metric labels have bounded cardinality?
- Are logs sampled only after preserving rare severe failures?
- Are secrets scrubbed before telemetry export?
Design guidance
Emit telemetry at decision points, not every line of code. Make attributes stable and bounded. Prefer a small set of high-signal events over noisy logs that hide the one fact needed during an incident.