Observability and incident response

Status: Complete. Last reviewed 2026-08-28.

Observability is the ability to answer consequential questions about a running system from produced evidence. It is not synonymous with collecting every signal or buying a telemetry product. Incident response is the decision system that uses that evidence under time pressure to limit harm, restore an acceptable service, communicate truthfully, and create durable learning.

Evidence, not a pile of signals

Start with decisions: whether users are harmed, which population is affected, where time or failures enter the path, what changed, and whether mitigation works. Instrument stable operation and outcome names, release identity, dependency, region, tenant class where safe, and correlation fields needed to join evidence. Telemetry describes an observation, not reality itself: dropped spans, sampling, aggregation, clock skew, retries, client abandonment, and broken instrumentation all introduce bias.

The common signals answer different questions:

Metrics aggregate counts, rates, distributions, and current values efficiently across a population. They expose trends and support alerting, but aggregation loses individual context.
Logs record discrete events with detailed context. Structured fields make them queryable; uncontrolled prose, sensitive payloads, and unbounded volume make them expensive or dangerous.
Traces connect causally related operations across boundaries and expose one request’s critical path. Head sampling can miss rare failures; tail sampling can retain interesting traces but needs buffering and a defined overload policy.
Profiles attribute CPU, allocation, lock, or wall-time samples to code. They explain resource consumption better than request logs but require representative conditions and safe overhead.
Events and audit records capture deployments, configuration changes, ownership actions, and business transitions. They often provide the change evidence that runtime signals lack.

Correlation is scoped. A trace ID connects one distributed execution; a request or job ID identifies another operational unit; a causation ID can connect asynchronous work after the original trace ends. None is automatically a business operation ID or authorization boundary. Propagate W3C Trace Context across trusted boundaries, validate incoming values, and avoid putting personal data or secrets in baggage, labels, or log context.

Telemetry has a budget and a failure mode

Instrument the boundary where an outcome becomes known: accepted, rejected, queued, completed, timed out, or reconciled. A request counter at ingress overstates success if clients disconnect or downstream work later fails. Conversely, application-only telemetry misses edge rejection and client experience. Combine server, dependency, queue, synthetic, and client evidence according to the service contract.

Metrics labels create a time series for each value combination. User IDs, request IDs, raw URLs, and exception messages cause cardinality growth, increasing memory, query cost, and alert instability. Keep dimensions bounded; place high-cardinality context in sampled logs or traces and link through exemplars where supported. Histograms preserve a bounded latency distribution suitable for aggregation, while client-side summaries/quantiles can be difficult to combine across instances.

Sampling is a cost and fidelity decision. Retain complete counters for service outcomes where feasible, sample high-volume detail, and state what analyses remain valid. Dynamic sampling can retain errors and slow traces, but then sampled data cannot estimate an unbiased population without weights and known policy. Redact before export, limit retention and access, and decide what collectors do when backends slow. Blocking application work to synchronously deliver telemetry can turn an observability outage into a product outage; bounded buffers and explicit drop accounting make that trade visible.

SLIs, SLOs, SLAs, and error budgets

An SLI is a measured indicator of service behavior, usually expressed as good events divided by valid events or as a distribution. An SLO is a target for an SLI over a window. An SLA is an external agreement with consequences and may use different scope or measurement. An error budget is the amount of unreliability allowed by the SLO; it is not a pool of incidents teams should intentionally consume.

Define the capability and viewpoint first. For a queue-backed import, valid events might be accepted imports and “good” might mean reaching a correct terminal state before a deadline. HTTP 200 rate alone would declare success while jobs stall. Exclude traffic only for a principled contract reason, and make exclusions observable so maintenance or instrumentation gaps cannot manufacture compliance. Use several SLOs only for distinct user promises, such as availability and completion latency, not one per internal component.

Choose targets from user need, dependency reality, and cost. A target of perfection removes room for maintenance and is often dishonest when dependencies cannot support it. A very loose target provides no product signal. Error-budget policy connects reliability evidence to actions: slow risky releases, prioritize repair, or require safer rollout when consumption is excessive. It should guide a decision with engineering and product context, not automatically punish a team for one number.

Actionable alerting and burn rates

Page a human for a condition that threatens a user-facing objective, requires timely judgment, and has a plausible action. Ticket slower capacity trends, noisy individual failures, and maintenance work. Every page needs an owner, severity, concise symptom, affected scope, relevant evidence, and a first diagnostic or mitigation path. If the receiver can only watch the graph, the alert is not actionable.

Alert on symptoms such as availability or deadline failure; use cause signals such as CPU, lock waits, replica lag, or one dependency’s latency for diagnosis and sometimes for preventive tickets. Cause-only paging produces noise and misses unfamiliar failures. Saturation can justify an urgent alert when exhaustion will soon violate the objective and there is a defined action such as load shedding or scaling.

Burn rate compares observed bad-event rate with the rate that would exactly consume the error budget over the SLO window. A high burn over a short window catches severe incidents quickly; a lower burn sustained over a longer window catches slow leaks. Requiring both a short and long window for a severity reduces single-spike noise while retaining urgency. Test alerts with historical incidents and controlled failure exercises, including missing telemetry, low traffic, delayed ingestion, and recovery. Alert resolution must also be stable so responders do not receive repeated open-close flapping.

Evidence-led diagnosis

Triage begins with impact, not a favored subsystem: what capability is failing, since when, for whom, and at what rate? Compare affected and healthy populations by release, region, route, tenant class, dependency, and input shape. Establish a timeline of deploys, configuration, traffic, data, provider, and infrastructure changes. Correlation is evidence, not proof; avoid reverting the nearest deploy without checking whether it matches the affected population and mechanism.

Trace the work and constrained resources. Rising end-to-end latency with growing application queue wait suggests admission or worker saturation. Low application CPU does not clear the service: database connection wait, lock wait, disk, downstream latency, DNS, quotas, or queue age may own the delay. A trace can locate time for one request; population metrics show breadth; a profile attributes local work; logs explain an explicit failure. Query the narrowest next piece of evidence that separates competing hypotheses.

Preserve uncertainty in the incident record: “checkout failures began after release X” is different from “release X caused failures.” Change one major variable at a time when impact permits and annotate mitigations, so resulting movement is interpretable. During severe harm, mitigation outranks perfect diagnosis: shed optional load, disable the failing path, stop a harmful producer, fail over, or restore a known-compatible release, then continue investigation from preserved evidence.

Incident command, communication, and mitigation

Declare an incident when coordinated response is cheaper than fragmented debugging. Assign an incident commander to set priorities and manage decision cadence; an operations lead to investigate and execute; and a communications lead for internal and external updates. Small incidents can combine roles, but one person should not silently own command, every terminal, and stakeholder communication.

Use a shared chronological record for observations, hypotheses, decisions, commands, owners, and results. Set explicit check-in times and escalation thresholds. The commander protects responders from duplicate requests and confirms who can authorize risky actions. Handovers state current impact, mitigations, evidence, hazards, and next decisions rather than forwarding an unstructured chat history.

Communications should say what users experience, affected scope when known, current mitigation, and the next update time. Do not speculate about cause or promise restoration times unsupported by evidence. Keep internal security-sensitive detail out of public updates while maintaining a truthful customer-impact record. Notify support, leadership, legal, privacy, or security according to pre-agreed triggers rather than improvising obligations during the event.

Mitigation reduces harm; remediation removes the defect; recovery restores normal service and verifies correctness. They are separate milestones. A graph returning to normal does not prove queued work drained, skipped notifications replayed, replicas converged, or financial effects reconciled. Define recovery checks from business invariants and monitor for recurrence before closing command.

Review, learning, and recovery exercises

A useful review reconstructs conditions and decision context without making individual carelessness the root cause. Ask why the action was reasonable with the information, tools, incentives, access, and safeguards present. Identify contributing technical and organizational conditions, detection and mitigation delays, what worked, and where luck prevented wider harm. Preserve concrete evidence while acknowledging telemetry gaps.

Actions should change a system: enforce a constraint, reduce a blast radius, add a release gate, improve a runbook, remove ambiguous ownership, or automate a recovery step. Give each an owner, priority, due/review point, and expected risk reduction. “Be more careful,” “add monitoring,” and a permanent manual checklist are weak unless they name the mechanism and how effectiveness will be verified. Not every incident warrants prevention at any cost; explicitly accept, transfer, mitigate, or eliminate each material residual risk.

Recovery capability requires exercises. Restore backups into an isolated environment, recover required keys and configuration, measure elapsed stages, verify data integrity and application invariants, and record the achieved recovery point rather than merely proving files can be read. Exercise dependency loss, region or control-plane failure, compromised credentials, unavailable operators, and corrupted data according to credible threats. Game days should have safety boundaries and learning goals; repeated practice validates runbooks, permissions, capacity, communication, and whether declared recovery objectives are achievable. Objective definitions and release-recovery choices are covered in deployment and release engineering.

Current and legacy context

Current: OpenTelemetry provides vendor-neutral signal APIs, semantic conventions, and collection components; managed backends and automated correlation are common. Instrumentation still needs product-specific outcomes, privacy controls, and validation.
Common: Mixed agents and schemas, sampled traces, structured and unstructured logs, dashboard-defined tribal knowledge, and several paging systems coexist during migrations. Standardize important operation names and correlation before pursuing total tool uniformity.
Legacy: Host-only monitoring, average latency, log searching without stable fields, threshold pages for every resource, and informal incident ownership can still operate critical systems. Ratchet toward user-centered SLIs, bounded labels, release evidence, declared command, and exercised recovery without discarding useful historical signals blindly.