Deployment and release engineering

Status: Complete. Last reviewed 2026-08-28.

A deployment is a distributed state transition, not a file copy. For some interval, old and new code, schema, messages, configuration, assets, and processes can coexist. Release engineering makes that transition repeatable and observable; a safe release preserves required compatibility at every intermediate state and has a tested route to a known-good state when evidence turns bad.

The compatibility system

A release artifact is only one participant. Callers may cache old assets, workers may hold old code, queued events may outlive several releases, replicas may lag, and instances may receive configuration at different times. “Backward compatible” therefore needs a named producer, consumer, representation, and coexistence window. A new writer that emits a value an old reader rejects is incompatible even if the database migration itself succeeds.

Map each change before choosing a rollout:

Schema and data: Which code versions can read and write each shape? Does DDL lock or rewrite data? Can a backfill resume, throttle, and coexist with writes?
Events and jobs: Can old consumers ignore additive fields and new consumers process retained old payloads? Are type names, enum values, ordering, and meaning stable?
Configuration and secrets: Which values must exist before process start? Can old and new values overlap during key or endpoint rotation? Does a changed default alter behavior fleet-wide?
Assets and clients: Can old HTML reference new assets and new HTML reference old assets? Content-addressed filenames and retaining prior bundles prevent a switch from producing 404s in cached pages.
External contracts: Will clients, webhooks, scheduled work, and third parties encounter both behaviors? Their deployment cannot usually be synchronized with yours.

Expand/migrate/contract is the general solution when independent versions coexist. First add a representation old software tolerates. Then deploy code capable of a transition, migrate or backfill data in bounded resumable units, observe convergence, and switch authority. Remove the old representation only after telemetry proves no live or replaying consumer needs it. Dual writes can bridge formats, but they introduce divergence: define which copy is authoritative, how failed writes reconcile, and when the bridge ends.

Database details remain engine-specific. An operation described as online can still take metadata locks, consume I/O, delay replicas, or block behind an existing transaction. Test the exact operation and data distribution, monitor lock and replication evidence, and keep large data movement outside the time-critical release step. The database reference covers those mechanisms; the release plan owns their ordering with the rest of the system.

Artifacts, containers, and process lifecycle

Build once, promote the same immutable artifact through environments, and attach source revision, dependency lock state, build provenance, and a unique release identity. Rebuilding for production allows repositories, clocks, or mutable base tags to produce something different from what passed verification. Configuration that varies by environment should be supplied and validated at deployment or startup; secrets must not be baked into image layers or emitted in provenance and logs.

An OCI image packages filesystem layers and metadata. It does not by itself guarantee reproducibility, vulnerability absence, correct configuration, or safe operation. Pin base images by immutable identity when reproducibility matters, minimize the runtime contents, generate an SBOM/provenance as policy requires, scan without treating a scanner as proof, and rebuild regularly for patched dependencies. Signatures and attestations establish statements about origin or process only to the extent that keys, builders, and admission policy are trusted.

Containers are replaceable processes, not small persistent hosts. Store durable business state in an appropriate external system or explicit durable volume, write disposable local state intentionally, and send operational output to a collected channel. Startup must validate required configuration and initialize bounded local state. Liveness asks whether restart can repair a stuck process; readiness asks whether it should receive work. A liveness probe that fails during a slow dependency incident can create a restart storm, while a readiness probe that synchronously stresses every dependency can amplify the outage.

Termination is part of correctness. Stop admitting traffic or claiming work, allow bounded in-flight operations to finish or checkpoint, then exit before the platform’s grace period ends. A hard kill remains possible, so handlers still need transactional boundaries, idempotency, and retry-safe effects. Long-running framework process commands and cache behavior are intentionally covered in Laravel deployment and long-running workers, not repeated here.

Choose a rollout from the failure model

Recreate stops the old population before starting the new one. It is simple and prevents version overlap, but causes unavailability unless another layer serves traffic. It suits bounded maintenance where compatibility work costs more than the accepted outage.

Rolling replacement gradually substitutes instances. It limits extra capacity but guarantees a mixed-version interval and may make rollback another rolling transition. Set surge/unavailable capacity so the remaining fleet can carry peak load, and account for startup warmup and connection pressure.

Blue-green prepares a complete new environment and switches routing. It offers a fast code-traffic reversal and clear comparison, at the cost of duplicate capacity and possible environmental drift. The database, queues, caches, and external effects are usually still shared, so it does not reverse incompatible state changes.

Canary or progressive delivery exposes a small, representative slice before expansion. It reduces blast radius only if assignment is stable where sessions or workflows require it, the sample includes risky tenants and paths, and gates can distinguish the candidate from the control. One percent of requests may still represent all traffic for a small region or one destructive background producer.

Feature flags separate code deployment from capability release. They support cohort control and fast disablement, but both branches, flag service failure, cached values, and asynchronous producers become operational states. Every temporary flag needs an owner, safe default, observability, and removal condition. A flag cannot make an incompatible schema or event emission safe by itself.

Progressive gates and evidence

A pipeline gate should answer a release decision, not merely report that a command ran. Before traffic, verify artifact identity and policy, schema/config preconditions, startup/readiness, and a small safe smoke path. During exposure, compare candidate and control for user-visible errors, latency distributions, saturation, queue age, dependency outcomes, and business invariants such as successful checkouts or balanced ledgers. Segment by release identity; fleet-wide averages can hide a failing canary.

Use explicit evaluation windows and minimum evidence rather than reacting to one noisy sample. Automated gates need missing-data behavior: “no telemetry” is not success. Absolute limits protect users; relative limits detect regressions when the whole system is otherwise healthy. Pause promotion when evidence is ambiguous, and make the authorized manual override audited with compensating checks. Deploy markers and release IDs in metrics, traces, logs, and audit events shorten the causal path from symptom to change.

A focused release timeline is:

establish compatible data and contracts;
publish an identified immutable artifact;
start a small population without broad traffic;
prove readiness and safe synthetic behavior;
expose a bounded cohort and evaluate technical and business gates;
expand in stages while retaining capacity and watching delayed work;
verify all process types and release identities;
contract old representations in a later release.

Rollback, roll-forward, and recovery objectives

Rollback restores a previous artifact or configuration; it does not undo facts already written or external effects already triggered. It is credible only while the previous version can consume the current schema, messages, assets, and configuration. Preserve the artifact, automate reversal, and test it under the actual rollout mechanism. Decide thresholds and authority before the incident so a team does not debate acceptable damage while it grows.

Roll-forward applies a new change that repairs the fault. Prefer it when data transformation is one-way, consumers have observed a new event, security exposure must be closed, or the fault is understood and a small fix is safer than restoring incompatible code. It takes longer when diagnosis or build time is uncertain. Often the fastest mitigation is first to disable a path, stop a producer, reduce traffic, or restore configuration, then choose rollback or roll-forward from evidence.

Recovery from release failure is distinct from disaster recovery, but the plans meet at data. RTO is the target time to restore a service after disruption. RPO is the acceptable data-loss interval measured backward from the incident. Neither is guaranteed by declaring a backup schedule. Replication can copy corruption, and a backup that has never been restored is only a claim. Define objectives per capability, inventory dependencies and encryption keys, use protected backups appropriate to the threat, and rehearse restoration plus integrity and application-level verification. The observability and incident-response reference covers exercise evidence and command during recovery.

Current and legacy context

Current: Immutable OCI artifacts, declarative orchestration, provenance, progressive exposure, and automated analysis are useful defaults where platform maturity supports them. They improve control but do not remove application compatibility work.
Common: Rolling fleets, managed databases, retained asynchronous messages, runtime configuration, feature flags, and several process types make version overlap normal even in a nominal monolith.
Legacy: Mutable shared hosts, in-place dependency installation, timestamped asset names without retention, manual schema edits, and backup-success emails may encode real constraints. Improve incrementally with release identity, scripted artifacts, compatibility stages, measured gates, and rehearsed reversal rather than requiring an immediate platform rewrite.