Performance, capacity, and scalability

Status: Complete. Last reviewed 2026-08-28.

Performance is whether a system meets a defined latency, throughput, and resource budget for a workload. Scalability is how those outcomes change as load and resources change. Optimization starts with a user-visible objective and measured constraint, not a preferred technology or an unattractive loop.

Define the workload and objective

Name the operation, population, input size, arrival pattern, concurrency, dependency behavior, and target environment. “The API must be fast” is not testable. “Checkout p95 under the agreed threshold at expected peak throughput, with an error-rate and resource-headroom budget” can guide measurement and trade-offs.

Latency is a distribution. p50 describes a typical observation; p95/p99 expose slower tails that often determine user experience and timeouts. Averages hide multimodal traffic and rare stalls. Segment by endpoint, tenant/input cardinality, cache outcome, dependency, region, and success/failure, while controlling label cardinality.

Throughput is completed work per time. Concurrency is in-flight work. Utilization is the fraction of a resource’s capacity in use. Saturation is queued or rejected demand beyond immediately available service. Errors include explicit failures and work that completes too late to be useful. Observe all four: a service can have low CPU while saturated on database connections or downstream quotas.

Establish a budget across the path: edge/queue wait, application CPU, database query/lock, cache, downstream calls, serialization, and transfer. A trace gives a critical path for one request; metrics show population behavior; profiles attribute CPU/allocation; logs explain discrete outcomes. No single tool is the truth.

Queueing changes tail latency

As utilization approaches a constrained resource’s capacity, small variability creates disproportionate queueing. More incoming work than service capacity grows backlog until callers time out, memory/broker retention fills, or admission control rejects. Adding application workers can make this worse by increasing contention at the database.

Apply Little’s Law carefully to a stable system: average in-flight work equals average throughput times average time in system. It can sanity-check whether observed concurrency and latency agree, but averages and steady-state assumptions do not predict burst tails. Measure queue age/depth, arrival/completion rates, pool wait, and utilization directly.

Concurrency limits should align with CPU cores for CPU-bound work and with connection/downstream capacity for I/O. Oversubscription can improve utilization during waits but increases memory, scheduling, and contention. Load test to find the knee where throughput stops increasing and latency/errors rise; retain headroom for bursts, failover, background work, and noisy neighbors.

Backpressure, deadlines, retry budgets, and load shedding are performance controls. An overloaded service should reject or degrade deliberately before it exhausts every shared pool. Retries add load precisely when dependencies are unhealthy; cap attempts, use delay/jitter, and avoid retries at every layer.

Diagnose before changing

Start with the symptom and compare healthy versus slow populations. Ask:

Is time spent queued or executing?
Which span/resource dominates the critical path?
Did work per request change: rows, queries, payload bytes, fan-out, allocations?
Is a pool, lock, CPU, memory, disk, network, or quota saturated?
Is the issue broad, tenant/input-specific, or tied to a dependency/cache miss?
What changed in code, data distribution, configuration, deployment, or traffic?

For databases, use the database and query-plan model: count queries and inspect exact execution plans, rows examined/returned, lock waits, connection wait, and replica lag. For application CPU, use a sampling profiler under representative load and inspect wall-time versus CPU profiles. For memory, distinguish live application allocations from process RSS, allocator fragmentation, retained long-lived state, buffers, and extension memory.

N+1 work can hide in ORM relations, accessors, authorization, serialization, template rendering, or callbacks. Eager loading can fix query count but over-fetch a huge graph; select only required columns/relations, batch or page, and measure peak memory. A single broad query is not inherently better than several targeted ones if it creates explosive joins or transfer.

Downstream fan-out affects the critical path. Sequential calls add latencies; bounded concurrency can overlap independent waits but cannot create quota/capacity. Remove unnecessary calls, batch, cache according to the dedicated cache contract, precompute, or make optional data asynchronous. Every remote call needs connection and total deadlines inside the caller budget.

Profile and benchmark credibly

Microbenchmarks isolate a mechanism and are useful only when that mechanism matters to the end-to-end objective. Prevent dead-code elimination where relevant, warm runtimes/caches according to the hypothesis, run enough samples, report distribution and environment, and compare results statistically rather than a single stopwatch. PHP callback/allocation overhead can dominate tiny operations, but optimizing it is irrelevant if I/O owns the path.

Load tests need realistic data cardinality/skew, request mix, think/arrival pattern, payloads, cache warmness, dependency latency/errors, and test topology. Closed-loop clients can hide overload by slowing arrivals as responses slow; open-model arrival tests better represent independent traffic when that is the production model. Protect production and third parties with explicit authorization and limits.

Test steady state, bursts, cold start, cache loss, dependency degradation, deploy/restart, and failover. Record client-side latency including errors/timeouts plus server queue/execution/resource metrics. A test generator can be the bottleneck; distribute it and validate sent versus completed rates.

Choose the smallest effective optimization

Remove work before making it faster. Then consider, in order driven by evidence:

fix the algorithm/data structure when CPU or growth is material;
eliminate N+1/repeated I/O and select less data;
add or reshape an index/query based on plans;
batch or overlap independent waits within bounds;
cache/precompute when staleness and invalidation are acceptable;
move nonessential work off the synchronous critical path;
tune pools/runtime only after understanding the constrained dependency;
scale resources when the work is necessary and architecture can use them.

State the correctness trade. Caching introduces staleness and failure modes. Read replicas add potentially stale reads. Denormalization adds synchronization/reconciliation. Approximation changes accuracy. Async completion changes the product contract. Compression trades CPU for network bytes. A performance win that violates tenant isolation or loses work is a defect.

Keep a before/after comparison under the same workload and observe regression dimensions: memory, write latency, downstream traffic, failure recovery, and operational complexity. Document the assumption and reversal trigger.

Capacity planning

Use measured service demand per request and resource capacity to estimate a starting point, then validate with load. Include traffic growth, peak-to-average, tenant skew, background jobs, redundancy/failover, maintenance, and safety margin. Capacity is multi-dimensional: CPU may fit while memory per worker, database connections, disk IOPS, network, or provider quotas do not.

PHP-FPM worker count is bounded by memory, CPU, and downstream connections. Measure per-worker RSS distribution under representative requests and ensure total workers plus system/sidecars fit memory without swapping. More children reduce FPM queueing only until another resource saturates. Queue-worker concurrency likewise aligns with database/provider capacity and job memory/duration.

Forecast from arrival/completion rates and utilization trends, but schedule load/failover tests before limits. Autoscaling has metric delay, provisioning/warmup time, minimum/maximum bounds, cooldown, and downstream consequences. Scaling on CPU misses I/O/pool saturation; scaling on queue depth without job age/cost can overreact to large cheap or small expensive jobs.

Scaling decisions

Vertical scaling is operationally simple and often the first rational step, but has host limits, failure blast radius, and nonlinear cost. Horizontal scaling increases request/worker capacity only when state is externalized or safely partitioned, traffic balances, and shared dependencies scale. Sessions/files/local caches and long-lived connections require explicit treatment.

Read replicas can offload eligible reads but introduce lag, routing, failover, and read-your-write issues. Partitioning/sharding can raise capacity or isolate tenants but adds routing, rebalancing, hot partitions, cross-shard queries/transactions, and operational recovery. Use it after evidence identifies the write/data boundary, not as a résumé pattern.

CDNs and edge caches reduce origin work and transfer latency for cacheable representations. Queues absorb bursts and decouple completion, but a growing queue is delayed work, not extra processing capacity. Backlog age and drain time determine whether the system meets its objective.

Laravel Octane or another long-running server can remove repeated bootstrap work and improve throughput when bootstrap is material. It changes service/request lifetime assumptions and memory behavior; it does not fix query plans or downstream waits. Benchmark the real application and audit state reset before adoption.

Production verification and regression control

Canary a change and compare latency distributions, throughput, errors, saturation, business outcomes, and cost against a control where possible. Warmup and traffic mix can bias results; use enough samples and segment. Feature flags enable quick disablement but add states requiring cleanup.

Create performance regression tests only for stable, important budgets. Microbenchmark thresholds can be noisy across CI hosts; prefer trend reporting or dedicated runners. Alert on symptoms and saturation with actionable context, not every metric fluctuation. An SLO/error budget connects performance degradation to user impact and delivery decisions.

Current and legacy context

Current: PHP 8.5/Laravel 13 are the repository baseline, but runtime, FPM, OPcache, Octane, database, and infrastructure behavior must be measured on the deployed versions.
Common: Container limits, autoscaling, managed databases/caches, CDN layers, and external quotas mean host CPU alone is an incomplete capacity signal.
Legacy: Premature rewrites, average-only dashboards, arbitrary worker counts, unlimited queues, and “scale horizontally” advice should be replaced with budgets, profiles, saturation evidence, and explicit trade-offs.