Distributed delivery and reliability

Status: Complete. Last reviewed 2026-08-28.

Distributed workflows cross failure and transaction boundaries. Messages can be duplicated, delayed, reordered, or delivered after a caller times out; processes can fail between any two durable writes. Reliability comes from naming the invariant, recording durable progress, making effects repeat-safe, and reconciling ambiguous outcomes—not from assuming a broker provides one perfect delivery.

Delivery guarantees need a boundary

At-most-once delivery avoids broker redelivery by acknowledging before or without retry, so failures can lose work. At-least-once delivery retries unacknowledged work, so duplicates are normal. “Exactly once” is meaningful only for a stated effect and boundary. A broker may atomically consume and produce within its own log, while an email, payment provider, or database outside that transaction can still repeat.

A worker typically receives a message, performs work, commits state, then acknowledges. If it fails after commit/effect but before acknowledgement, the message returns. A visibility timeout or lease can also expire while a slow worker is active, allowing concurrent delivery. Set acknowledgement and visibility policy from maximum bounded processing time, but still design for duplicates.

Do not acknowledge before the effect is durably safe unless loss is acceptable. Do not acknowledge after unbounded retries forever; classify permanent errors, transient dependency errors, conflicts, cancellation, and poison payloads. Dead-lettering is a transfer of ownership, not resolution. It needs alerts, inspection, redrive policy, retention, and tooling that preserves idempotency.

Idempotent consumers and inbox state

An idempotent operation produces the intended state when repeated with the same logical identity. “Set status to shipped” may be idempotent, while “increment balance” is not unless tied to a unique ledger operation. Generate or propagate a stable operation/message ID and enforce it at the durable effect boundary.

For database effects, an inbox/deduplication record under a unique constraint can be committed with the business change. On duplicate, return the recorded result or no-op. A check in Redis followed by an independent database write has a crash gap and expiry window. Retention must cover plausible broker/provider replay; deleting dedupe state re-enables old effects.

External APIs need their own idempotency keys or a queryable operation ID. If a call times out, the result is unknown—not failed. Query status or reconcile before retrying an unsafe effect. Email providers may accept a stable message ID yet still have different guarantees; define whether occasional duplicate notification is tolerable and separate it from financial correctness.

The dual-write problem and transactional outbox

An application cannot usually atomically commit its database transaction and publish to an independent broker. Publishing before commit can expose data that rolls back. Publishing after commit avoids that but the process can die after commit and before publish. Framework “after commit” hooks close one ordering problem, not the crash gap.

The transactional outbox writes business state and an outgoing message record in one local transaction. A relay claims unpublished rows, publishes them, and records progress. The relay can crash after publish before marking sent, so publication remains at-least-once and consumers remain idempotent. Polling, change-data capture, or a database-log connector can drive the relay; each has ordering, lag, and operational trade-offs.

Outbox rows need a stable ID, aggregate/partition key, type/schema version, payload or reconstructable reference, creation time, attempt/claim state, and retention. Claiming must allow multiple relays without double work becoming unsafe. Monitor oldest-unpublished age, attempts, relay errors, broker acknowledgement, and table growth. Archival must not erase evidence needed for replay or audit prematurely.

Ordering is scoped

Global order is expensive and rarely the business requirement. Usually changes for one aggregate—order, account, tenant resource—need order, while independent aggregates can proceed concurrently. Partition messages by that key and carry an aggregate version/sequence. Broker partition order does not repair multiple producers assigning conflicting order; the authoritative writer must issue versions.

Consumers should define duplicates, gaps, and late messages. A version lower than applied can be ignored; a gap may pause and fetch/replay, or the consumer may refresh current state if events are notifications rather than a complete event-sourced history. Arrival timestamp is not reliable causal order, and retries can put old work behind new work depending on broker patterns.

Ordering can conflict with availability and throughput. One poisoned message can block an entity partition. Decide whether to quarantine that entity, stop the whole partition, skip with reconciliation, or compensate. Do not promise fairness merely because a queue is FIFO; multiple workers, priorities, redelivery, and visibility timeouts alter observation.

Commands, events, and schemas

A command requests that a named owner perform an action and can be rejected. An event states that a fact occurred and can have many consumers. A message is the transport envelope for either. Treating every synchronous dependency as an event obscures who owns failure and whether the caller needs an answer.

Message schemas are compatibility contracts. Prefer data, IDs, and explicit versions over serialized application objects. Add fields compatibly, tolerate unknown fields, and treat new enum variants and changed meaning as potential breaks. During rolling deployments, old and new producers/consumers coexist. Schema registries and contract tests help, but rollout and replay behavior remain application responsibilities.

Carry event time and production metadata for diagnosis, but use aggregate versions for business order. Correlation and causation IDs connect a workflow; message IDs identify deliveries/logical operations. Do not use a trace ID as the deduplication key for several distinct commands.

Sagas and compensations

A saga coordinates several local transactions without pretending there is distributed rollback. Orchestration has a coordinator issue commands and record workflow state. Choreography has participants react to events. Orchestration makes progress and recovery visible; choreography reduces central control but can hide cycles, ownership, and emergent coupling.

Each step needs a durable state, idempotent command identity, timeout, retry classification, and compensation where business-appropriate. Compensation is a new forward action, not time travel: refund can fail, cancellation may incur a fee, and an email cannot be unsent. Order compensations carefully and design them to repeat.

A state machine should make terminal, retrying, waiting, compensating, manual-review, and completed states explicit. Avoid holding database transactions across services. If a step’s outcome is ambiguous, reconciliation queries the participant or compares ledgers before deciding the next transition.

Reconciliation closes uncertainty

Retries repair known transient failures. Reconciliation finds divergence even when no process knows it failed. Compare local operations with provider reports, outbox with published/consumed state, reservations with orders, and workflow states with deadlines. Use stable IDs, amounts/versions, bounded windows, and checkpoints so reconciliation is resumable and safe.

Repair actions are idempotent and audited. Some mismatches can be fixed automatically; ambiguous money or entitlement differences may require manual review. Track age and count of unresolved items as a reliability objective. A green queue does not prove external and local state agree.

Failure timeline method

For any workflow, place crash points between every pair of steps:

receive message;
record attempt/idempotency;
change local state;
call external system;
record external outcome;
publish next message;
acknowledge.

Ask what is durable at each point, what a retry observes, whether the external result is known, and which reconciler can recover. This reveals gaps that architecture labels hide. Test by injecting failures at those boundaries and replaying duplicate, late, and out-of-order messages.

Current and legacy context

Current: Modern brokers offer different acknowledgement, partition, transaction, and consumer-group semantics; verify the selected product/version rather than generalizing from “a queue.”
Common: At-least-once delivery plus idempotent consumers, transactional outbox, aggregate-scoped ordering, and scheduled reconciliation form a practical reliability baseline.
Legacy: Serialized framework jobs, long retry chains without IDs, broker-as-source-of-truth assumptions, and choreography without workflow state should be migrated with replay compatibility and operational evidence.