System design

Status: Complete. Last reviewed 2026-08-28.

System design is the disciplined conversion of product and operating constraints into boundaries, state, interfaces, and failure behavior. In an interview, the goal is not to draw the largest architecture or guess the interviewer’s preferred stack. It is to expose assumptions, protect the strongest invariants, trace important paths, and explain when evidence would justify a more complex design.

Establish the decision frame

Begin by defining actors and capabilities: who submits work, who reads results, who administers it, and which external systems participate? Select one or two critical journeys. Clarify what is explicitly out of scope so the design can become concrete.

Turn vague quality goals into constraints. Ask for approximate active users and tenants, requests or events per second at normal and peak load, object/payload sizes, retention, geographic distribution, latency objectives, availability expectations, recovery objectives, privacy/residency rules, external quotas, cost limits, and team/launch timeline. If figures are unavailable, state reasonable symbols or assumptions and proceed; changing an assumption later should visibly change a decision.

Workload shape matters more than one impressive daily total. Capture read/write ratio, burstiness, fan-out, skew between tenants or keys, hot-object contention, batch size, growth, and background versus interactive work. Averages conceal peaks and heavy tenants. Estimate order-of-magnitude storage, transfer, throughput, concurrent work, and downstream calls only where they test feasibility or expose a bottleneck. Validate estimates with measurements before procurement.

Write the strongest invariants before naming technology. Examples include “one active reservation per resource and interval,” “a tenant cannot address another tenant’s data,” or “one logical provider event changes the ledger at most once.” Separate hard correctness from acceptable staleness, desired availability, and user experience. CAP slogans do not choose semantics; name the operation and what it may do during delay, partition, conflict, or dependency outage.

Choose ownership and the simplest boundary

Start with the simplest architecture capable of meeting those constraints, commonly a modular application, relational database, durable object storage where needed, and a queue for work that need not finish in the request. Define modules by policy and state ownership rather than framework folders. A network service is justified by independent lifecycle, scaling, security, data ownership, or organizational accountability—not because the diagram appears more senior.

For each stateful component, state what it owns, its source of truth, and who may write it. Local transactions make cross-table invariants straightforward. Splitting ownership introduces network failure, versioned contracts, asynchronous propagation, reconciliation, and duplicated operations. If extraction becomes justified, identify the seam, migrate callers compatibly, and avoid a period where two services independently own the same fact.

Draw trust boundaries and failure domains as well as boxes. A managed database can still be one regional dependency; several services sharing its credentials and schema are not operationally independent. Explain process/host/zone/region redundancy only against the stated availability and recovery objective.

Design APIs, data, and transactions together

Define a small external contract around capabilities. Include request identity, authorization scope, idempotency or conditional-update semantics, synchronous versus 202 asynchronous outcomes, stable machine-readable errors, pagination/order, and compatibility. Do not expose storage tables directly. See HTTP and API design for protocol, webhook, and evolution mechanisms.

Sketch entities, identities, cardinality, lifecycle states, ownership keys, and constraints. Show the unique/check/foreign-key constraints that enforce hard invariants and the indexes required by named access paths. Include tenant keys in uniqueness and routing where isolation requires them. State whether deletion means hard delete, lifecycle transition, retention, or tombstone.

Place a transaction around state that must commit atomically in one database. A preliminary availability read does not protect a booking from a concurrent writer; use a constraint, conditional write, lock, or serializable transaction suited to contention. Never imply that a database transaction rolls back an email or provider call. Bridge independent systems with stable operation IDs, idempotent consumers, an outbox, and reconciliation as described in distributed delivery and reliability.

Replication and partitioning follow access and consistency requirements. A read replica adds lag and routing behavior; it can violate read-your-write expectations. Partitioning adds routing, hot partitions, rebalance, cross-partition query/transaction, and recovery costs. State a partition key and demonstrate why its distribution and query locality fit before introducing it.

Trace behavior, not just structure

Trace one critical write and one critical read end to end. For each hop, say what is validated and authorized, what durable state exists, where the transaction commits, what response is returned, what may be cached or stale, and which identifiers connect logs and retries. Include at least one concurrent request or duplicate delivery.

Then insert failure between every pair of important steps. If the process dies after commit but before response, can the caller retry safely? If publish succeeds before a relay marks the outbox row, can consumers repeat safely? If a provider times out, is the outcome failed or unknown? If a replica or index is delayed, which path remains correct? A trace should end in a terminal state, bounded retry, compensation, or reconciliation owner rather than “the queue handles it.”

For reads, trace cache miss/hit, authorization, source of truth, consistency requirement, query/index, payload bounds, and degradation. Caches accelerate a contract but do not silently become authoritative. Explain invalidation or versioning, stampede control, and acceptable stale behavior by linking to the cache and Redis guarantees.

Asynchrony and failure policy

Move work asynchronous when completion can be decoupled from acknowledgement, when bursts need durable buffering, or when retries/outages should not occupy request capacity. Define the job/message schema, stable identity, ordering scope, attempt policy, timeout, idempotent effect, progress state, cancellation, dead-letter ownership, and deployment compatibility. Bound queue depth, worker concurrency, payload size, and per-tenant share. Queueing absorbs a burst; it does not create processing capacity.

Classify dependency failures as permanent, transient, throttled, conflict, ambiguous, or caller cancellation. Apply deadlines within the caller’s budget, exponential delay and jitter, retry limits, and circuit/admission controls where they protect capacity. Avoid retries at every layer. Define degraded behavior: reject safely, serve stated stale data, defer optional work, or make the dependency explicit to the user.

High availability requires eliminating or accepting each relevant single failure domain, but redundancy also creates failover, split-brain, and capacity questions. State recovery point and recovery time objectives, backup/restore verification, and whether failover capacity handles peak load. Disaster recovery is unproven until restoration and traffic switching are exercised.

Capacity, security, and operations

For likely bottlenecks, connect demand to capacity: request CPU/memory, database connections and write rate, lock contention, disk/transfer, queue service time, provider quota, and fan-out. Keep headroom for bursts, failover, deployment, and background work. Use load tests with realistic cardinality and skew to locate the saturation knee; then scale the constrained resource or reduce work. The detailed model belongs in performance, capacity, and scalability.

Threat-model assets, actors, entry points, and trust transitions. Authenticate identities, authorize each capability in tenant/resource scope, encrypt in transit and where the threat model requires at rest, minimize and rotate secrets, redact sensitive telemetry, validate untrusted files and URLs, and audit privileged actions. Rate, size, and concurrency bounds are security controls. Use the dedicated application and operational security guidance rather than treating “use HTTPS” as a security section.

Define service-level indicators around user outcomes: successful/valid completion latency, availability, freshness, backlog age, and correctness/reconciliation gaps. Add structured logs with operation and safe tenant context, metrics with controlled cardinality, and traces across synchronous and asynchronous boundaries. Alert on symptoms and exhausted error budgets with an owner and runbook, not on every internal fluctuation.

Deployment is part of the design. Explain rolling-version compatibility for APIs, messages, and schema; use expand/migrate/contract for storage; make backfills resumable and throttled; canary risky changes; and define rollback versus roll-forward. Feature flags need ownership and removal. Capacity and failure tests should cover cold caches, provider degradation, worker restart, duplicate messages, and restoration.

Evolution and decision triggers

Conclude with what the first version deliberately does not solve and the evidence that would trigger change. Examples: sustained connection saturation triggers query/pool work before a service split; measured replica-safe reads justify replicas; tenant size skew and isolation requirements may justify tenant placement or partitioning; provider reliability or cost may justify a second notification adapter. This demonstrates evolution without paying every complexity cost on day one.

Record consequential choices and assumptions in ADRs. Prefer migration seams that preserve ownership: stable API contracts, outbox events, versioned schemas, and bounded adapters. Do not create generic abstraction layers solely for hypothetical databases, brokers, or vendors. Revisit when an explicit trigger occurs.

Focused decision sketches

These are starting decisions, not duplicate mechanism guides:

Booking: define the allocation/time-overlap invariant and cancellation lifecycle; enforce it at the database boundary with a constraint or concurrency strategy from databases, indexes, and concurrency. Trace two contenders, timeout after commit, expiry, and payment compensation.
Webhook: verify the exact signed bytes, durably deduplicate a provider event, acknowledge quickly, then process with the API contract in HTTP and API design and retry/reconciliation model in distributed delivery.
Import: place the file in durable storage; stream/chunk with bounded memory; persist schema/version, checkpoint, progress, row errors, and cancellation; use bounded workers and backpressure from concurrency and asynchronous I/O. Define whether rows are atomic, independent, or staged before commit.
Multi-tenant SaaS: make tenant context explicit in identity, query scope, unique keys, cache keys, files, jobs, quotas, telemetry, and administrative paths. Choose shared rows, schemas, databases, or placement groups from isolation, scale, residency, restore, and operating-cost requirements; apply security’s tenant-isolation model.
Notification platform: separate notification intent, user preferences, channel rendering, provider attempt, and delivery evidence. Use stable IDs and per-recipient/channel deduplication, bounded fan-out, provider quotas, fallback policy, and honest accepted/sent/delivered semantics from distributed delivery.

The corresponding implementation exercises are in system-design and implementation scenarios. Search and other read-heavy designs use the same method: define freshness and ranking semantics, then justify an index, projection, or specialist engine from measured access paths rather than product category alone.

Current and legacy context

Current: Managed databases, queues, object stores, identity systems, and observability reduce implementation work but retain shared-responsibility, quota, outage, portability, and cost constraints.
Common: A modular monolith plus relational source of truth and selective asynchronous processing is often a stronger starting point than premature services; container or serverless deployment does not change data invariants.
Legacy: Shared mutable databases without ownership, unversioned messages, cron-only recovery, average-only capacity plans, and big-bang rewrites need characterization and staged migration rather than cosmetic redrawing.