Debugging scenarios

Status: Complete. Last reviewed 2026-08-28.

These exercises test diagnosis from bounded evidence rather than recall of a named failure. Treat every packet as a synthetic incident snapshot, not a complete production record or benchmark.

How to use this page

1. Time-box the first pass to 10–15 minutes. State what is known, what is inferred, and what remains unknown.
2. Follow the diagnosis framework: establish impact, rank a small set of causal hypotheses, and choose the next observation or experiment that best separates them.
3. Propose an immediate mitigation that reduces harm without destroying evidence. Keep it separate from the root fix.
4. Define verification before opening the discussion guide. Include the user-visible symptom, the suspected mechanism, and regression or correctness signals.
5. Use the owning guidance and canonical questions afterward to deepen weak areas. The descriptive scenario anchors below are navigation aids, not canonical question IDs.

Stable median, rising tail latency

Incident brief

The GET /api/search endpoint still feels normal to most callers, but some requests now exceed the client deadline. The median and application CPU are stable. No broad error-rate alert fired, and traffic volume is within the range seen earlier in the week.

Evidence packet

Synthetic evidence: Illustrative five-minute production snapshots and two representative traces; values are intentionally compact and are not benchmark claims.

Signal	Earlier healthy window	Current window
Request p50	82 ms	85 ms
Request p99	430 ms	2.4 s
Application CPU	41%	43%
Responses past client deadline	rare	increasing
Database-pool wait p99	18 ms	1.7 s

trace=7ac1 tenant_size=small cache=hit  total=91ms  db.pool_wait=2ms   sql=24ms
trace=9be4 tenant_size=large cache=miss total=2.3s  db.pool_wait=1.6s sql=280ms

Candidate task

Explain the current and potential user impact, including why p50 and average CPU do not clear the service. Rank plausible hypotheses for the slow subset. Choose the next discriminator that would separate pool saturation from expensive SQL, cache-miss work, tenant skew, or a downstream retry. Give a safe immediate mitigation, a likely root-fix direction, and a verification plan covering latency distribution, timeouts, saturation, and correctness.

Discussion guide

The impact is concentrated in the tail: affected callers can time out even while the typical request remains fast, and abandoned server work can continue consuming capacity. The strongest current clue is database-pool queueing on large-tenant cache misses, but two traces do not establish why connections are unavailable.

A useful next step compares slow and normal populations by cache outcome and tenant/input size, then follows pool acquisition and connection-hold spans. Inspect in-use connections, acquisition wait, transaction duration, exact SQL and plans, and whether another workload shares the pool. This separates slow execution while holding connections from insufficient pool capacity or leaked/overlong transactions. Raising the pool or FPM count first could transfer saturation to the database.

Containment might admission-limit or degrade the expensive miss path, serve acceptable stale results, pause a competing background workload, or reduce deadlines/retries that hold resources after callers leave. The root fix should address the proven mechanism: for example bound the result, repair the query/index, shorten connection ownership, or prevent simultaneous regeneration. Verify with the affected workload distribution and compare p50/p95/p99, deadline outcomes, pool wait, SQL work, throughput, and database headroom; check that result semantics and cache freshness did not change unexpectedly.

Owning guidance: Performance, capacity, and scalability, especially “Diagnose before changing.”

Canonical practice: BACKEND-PERFORMANCE-01 — Diagnose rising p99

Exhausted PHP-FPM worker pool

Incident brief

During a traffic burst, the reverse proxy reports increasing upstream wait for ordinary web requests. PHP-FPM has not crashed, host CPU is moderate, and responders suggest doubling pm.max_children.

Evidence packet

Synthetic evidence: A representative FPM status sample, host snapshot, and trace comparison from the incident window.

Signal	Before burst	During burst
Active / total FPM processes	14 / 32	32 / 32
Listen queue	0	growing
Max-children reached counter	unchanged	increasing
Host CPU	48%	55%
Provider call p95	190 ms	1.1 s

proxy request_time=1.42s upstream_connect=0.00s upstream_header=1.39s
app   trace=4df2 fpm_queue≈0.86s php_execution=0.50s provider.catalog=0.37s

Candidate task

Describe impact and overload risk, then rank hypotheses for why all children remain busy despite moderate CPU. Name the discriminator you would gather before changing pool size. Propose a mitigation that protects callers and dependencies, a root fix, and verification that includes queueing, process memory, database/provider capacity, throughput, and tail latency.

Discussion guide

The listen queue and max-children events show FPM saturation; they do not prove that the configured limit is intrinsically too small. Moderate CPU and slower provider spans suggest workers are occupied waiting, while queued requests inflate end-to-end latency. Confirm this across traces and inspect request-duration distribution, provider outcomes, timeouts/retries, per-worker RSS, database connections, and whether one slow route monopolizes a shared pool.

Immediate options include shedding optional traffic, disabling or serving stale data for the provider-dependent feature, applying bounded provider deadlines and retry budgets, and isolating especially slow work. Raising children is defensible only if measured memory and downstream headroom support it; otherwise it creates more concurrent pressure.

The root fix follows the discriminator: repair or decouple the slow dependency path, bound fan-out, separate pools, or resize FPM from representative service time, RSS, and dependency capacity. Re-run burst and degraded-dependency tests while observing listen-queue age/depth, active/idle workers, completed throughput, p95/p99, errors, RSS/swap, database connections, and provider load. Recovery should drain the queue without a second overload wave.

Owning guidance: PHP runtime, memory, OPcache, and process models and Performance, capacity, and scalability.

Canonical practice: PHP-RUNTIME-04 — Size and diagnose an FPM pool, BACKEND-PERFORMANCE-03 — Size PHP workers

N+1 hidden during resource serialization

Incident brief

A projects endpoint passed controller review because its query uses eager loading. Production traces still show query count growing with the number of returned projects, and most repeated queries occur after the controller span ends.

Evidence packet

Synthetic evidence: Reduced application code and normalized query-log shapes from one request returning 40 projects.

// Controller
$projects = Project::query()->with('tasks')->limit(40)->get();
return ProjectResource::collection($projects);

// ProjectResource
return [
    'id' => $this->id,
    'tasks' => TaskResource::collection($this->tasks),
];

// TaskResource
return ['title' => $this->title, 'assignee' => $this->assignee->name];

1x  select ... from projects limit 40
1x  select ... from tasks where project_id in (...)
38x select ... from users where id = ?
query phase: controller=2 queries, resource transformation=38 queries

Candidate task

State the user and capacity impact, identify likely query triggers and alternative hypotheses, and choose a discriminator that locates the first lazy relationship access. Give a safe mitigation, the owning-query and resource root fix, and tests or telemetry that verify query growth, response shape, row volume, latency, and memory.

Discussion guide

The repeated user query shape and transformation timing point to $this->assignee in TaskResource, not the visible controller loop. Confirm with a database listener or trace linked to the first relationship access and test several parent/task cardinalities. Also inspect appended accessors, policies, nested resources, and reverse child-to-parent traversal before assuming this is the only hidden query.

A short-lived mitigation can cap page size or disable the costly optional field. The normal root fix is for the endpoint’s owning query to load the exact graph and columns the representation requires, such as constrained tasks.assignee, while the resource uses whenLoaded('assignee') so transformation cannot silently initiate I/O. If only an identifier or aggregate is needed, a projection may be smaller than hydrating another model graph.

Verification should use more than one record and assert a bounded query count independent of returned cardinality, expected JSON fields and authorization, rows transferred, and representative peak memory/latency. Preventing lazy loading in tests or development is a useful guard, but it does not choose the efficient eager-loading plan.

Owning guidance: Eloquent relationships and loading, especially “N+1 diagnosis” and “Serialization is a query boundary.”

Canonical practice: LARAVEL-RELATIONSHIPS-04 — Diagnose a hidden N+1, LARAVEL-RELATIONSHIPS-06 — Design a relationship-safe API resource

Deadlock retry plus a duplicate job effect

Incident brief

Two settlement jobs overlap for the same accounts. The database selects one transaction as a deadlock victim. The retried job completes, but the external ledger contains two credits for one logical settlement.

Evidence packet

Synthetic evidence: Correlated logs and simplified handler code; identifiers and amounts are fictional.

09:14:31 job=job-82 operation=settle-417 attempt=1 provider_ref=p-991 status=accepted
09:14:31 job=job-82 operation=settle-417 attempt=1 db_error=deadlock transaction_rolled_back=true
09:14:33 job=job-82 operation=settle-417 attempt=2 provider_ref=p-992 status=accepted
09:14:33 job=job-82 operation=settle-417 attempt=2 db_commit=true queue_ack=true

DB::transaction(function () use ($settlement) {
    foreach ($settlement->accountIds as $id) {
        Account::whereKey($id)->lockForUpdate()->firstOrFail();
    }

    $reference = $this->ledger->credit($settlement->amount);
    $settlement->markPosted($reference);
}, attempts: 3);

Candidate task

Assess financial and reconciliation impact. Separate hypotheses for the deadlock from the duplicate effect, and choose evidence that discriminates database retry, queue redelivery, and provider retry. Give an immediate containment and repair plan, then design a root fix for lock ordering, transaction boundaries, idempotency, and ambiguous outcomes. Define verification at each crash/retry point.

Discussion guide

The deadlock and duplicate are related but distinct. Inconsistent account-lock order can create cyclic waiting; the database correctly aborts one transaction. Retrying the whole closure repeats an external call that the database cannot roll back. The stable operation ID alongside two provider references is strong evidence, but confirm provider request metadata and queue reservation/attempt history before ruling out another producer.

Containment should stop or serialize affected settlement work, preserve logs, identify operations with more than one provider effect, and reconcile rather than issue blind compensating calls. A refund or reversal is itself a fallible audited operation.

The durable design acquires database locks in a stable order and keeps transactions short. Persist a settlement intent and stable idempotency key, call the provider outside a retryable database closure with that key, and conditionally record the resolved outcome. Unknown timeouts remain pending for provider lookup or reconciliation. Exercise failures before the call, after provider acceptance, before local commit, after commit, and before queue acknowledgement; assert one logical credit, convergent local state, safe redelivery, and observable unresolved outcomes.

Owning guidance: Databases, indexes, and concurrency and Laravel queues, events, listeners, notifications, and retries.

Canonical practice: LARAVEL-QUEUES-03 — Make a retried side effect idempotent, DATA-DELIVERY-01 — Make a payment consumer retry-safe

Growing queue-worker memory

Incident brief

Queue throughput degrades over a worker’s lifetime, then the container terminates the process for memory pressure. A replacement starts small and repeats the same pattern. Individual jobs remain below PHP’s configured memory limit.

Evidence packet

Synthetic evidence: Samples from one worker processing similar-sized jobs and a reduced listener registration found in the handler.

Completed jobs	PHP usage after job	Process RSS	Registered export listeners
1	46 MiB	78 MiB	1
100	91 MiB	130 MiB	100
300	176 MiB	224 MiB	300

public function handle(Exporter $exporter): void
{
    $rows = $exporter->loadRows($this->exportId);

    Event::listen(ExportFinished::class, function () use ($rows): void {
        Log::info('export-finished', ['rows' => count($rows)]);
    });

    $exporter->write($rows);
}

Candidate task

Describe availability and duplicate-processing risks around memory termination. Rank retained userland state, per-job peak, allocator fragmentation, and extension/native growth as hypotheses. Choose the next measurement that discriminates them. Propose containment, a root fix, and repeated-job verification covering PHP memory, RSS, retained object/listener counts, throughput, job identity, and cleanup after failure.

Discussion guide

The monotonic listener count and captured $rows provide a concrete retained-userland hypothesis: a process-lifetime dispatcher holds each closure and its job graph. The rising PHP usage supports it more directly than allocator-only fragmentation, though RSS/PHP differences should still be measured. Use heap/object inspection where available, count listeners and references after each job, and run the same jobs with registration removed. Also compare small and large jobs and inspect native allocations if PHP usage later stabilizes while RSS continues upward.

Set a conservative worker job/time/memory recycling limit and temporarily reduce job concurrency so the supervisor can replace workers without exhausting the host. This bounds damage but does not make termination safe; jobs still need idempotency because a hard kill can cause redelivery.

The root fix is to register a stateless listener once at boot or invoke logging directly, avoid process-lifetime capture of job data, release large graphs promptly, and stream/chunk if the per-job peak is also material. A same-process regression should process varied tenants and hundreds of representative jobs, assert bounded listener/context state, and show a stable post-warmup memory slope and throughput. Include a forced termination/redelivery test and verify no previous job’s tenant or payload appears later.

Owning guidance: PHP runtime, memory, OPcache, and process models and Deployment and long-running workers.

Canonical practice: PHP-RUNTIME-03 — Diagnose PHP memory versus RSS, PHP-CALLABLES-06 — Diagnose captured state in a worker

Cache stampede at expiry

Incident brief

A dashboard report is fast most of the hour, then database load and request tails spike around the same time across application instances. Redis latency remains normal and the overall cache hit ratio still looks healthy.

Evidence packet

Synthetic evidence: A key-level timeline and the cache-aside implementation for the hot report.

Relative time	Key state	Concurrent report computations	Source query load
before expiry	hit	0	baseline
just after expiry	miss	27	sharply elevated
refresh complete	hit	0	recovering

return Cache::remember(
    "dashboard:{$team->id}:v3",
    now()->addHour(),
    fn () => $reports->buildFor($team),
);

key=dashboard:team-18:v3 event=expired
next_2s requests=31 hits=4 misses=27 refresh_duration=1.8s redis_command_p95=normal

Candidate task

State caller and source-system impact, rank expiry stampede against Redis slowness, eviction, a hot shard, and a naturally expensive query, and name the next discriminator. Give a safe mitigation for the current wave, a root refresh design with explicit stale/failure behavior, and verification under cold cache, mass expiry, and refresher failure.

Discussion guide

The expiry correlation, concurrent computations for one key, and normal Redis command latency support a miss stampede. Confirm that computations overlap in traces and distinguish expiration from eviction or invalidation. Inspect refresh duration, source saturation, key-level outcomes, and whether related keys share deterministic TTLs. The global hit ratio is too broad to clear this path.

For immediate relief, restore a recently known-good value if its age is acceptable, rate-limit regeneration, or temporarily shed/degrade the report while the source recovers. Do not let every miss bypass the cache without admission control.

A root design can serve bounded stale data while one owner refreshes, use a short regeneration lease with owner-safe release and bounded wait, and add TTL jitter or early refresh for groups of keys. If stale output is forbidden, one computation proceeds while others wait briefly or fail according to a documented contract. Duplicate refresh remains possible after lease expiry, so computation and publication must be safe. Test cold start, simultaneous expiry, slow/crashed refresh, cache outage, and recovery while observing in-flight refreshes, stale age, lock contention, source load, tails, errors, and report correctness.

Owning guidance: Cache and Redis reasoning and Laravel cache, sessions, and Redis integration.

Canonical practice: DATA-CACHE-02 — Prevent a cache stampede, LARAVEL-CACHE-03 — Control a hot-key stampede

Cross-tenant data leaked through a cache key

Incident brief

A support ticket includes a project summary belonging to another tenant. Database queries and route-model binding are tenant-scoped. The affected endpoint recently added response caching.

Evidence packet

Synthetic evidence: Reduced endpoint code and request/cache logs. Tenant and project identifiers are fictional; the packet contains no real customer data.

$project = Project::query()
    ->where('tenant_id', $request->user()->tenant_id)
    ->findOrFail($projectId);

return Cache::remember(
    "project-summary:{$project->id}",
    now()->addMinutes(15),
    fn () => new ProjectSummaryResource($project),
);

request=r-11 tenant=tenant-a project_id=42 cache_key=project-summary:42 result=miss
request=r-29 tenant=tenant-b project_id=42 cache_key=project-summary:42 result=hit
response=r-29 serialized_tenant=tenant-a

Candidate task

Treat this as a security incident: define confirmed and possible impact, preserve useful evidence, and rank cache collision against unscoped lookup, stale worker context, and logging error. Choose the fastest safe discriminator. Give immediate containment and remediation, a root cache/authorization design, and verification across tenants, roles, invalidation, cache failure, and long-running workers.

Discussion guide

The same key for tenant-local project IDs and the cached serialized tenant make cross-tenant key collision the leading hypothesis. Verify raw cache values and correlated response data carefully, then identify all callers, key versions, affected time range, and whether role-dependent fields create additional variants. Preserve access/cache logs under incident controls; do not copy sensitive payloads into ordinary debugging channels.

Containment should disable the affected cache path or endpoint, purge the unsafe namespace, and prevent further exposure while incident owners assess notification and audit obligations. Purging alone is not a fix because the next pair of colliding requests recreates the leak.

Prefer caching authorization-neutral data under an application/environment/schema/tenant/resource key, then applying current authorization and field filtering after retrieval. If the representation truly varies by permission, include a reliable permission/version dimension or do not shared-cache it. Invalidation occurs after commit and cache failure must not bypass policy. Tests should use equal local IDs in two tenants and several roles, reverse request order, run hit/miss paths, and assert no foreign fields or timing-dependent access. Also test two sequential operations in the same long-lived worker and audit every similar cache-key constructor.

Owning guidance: Laravel cache, sessions, and Redis integration and Validation, authorization, policies, and tenant boundaries.

Canonical practice: LARAVEL-CACHE-01 — Design a safe cache key and invalidation plan, LARAVEL-BOUNDARIES-04 — Design tenant isolation

Mixed release and schema assumptions

Incident brief

Web traffic is healthy after a release, but queued invoice jobs begin failing. The deployment switched web instances to the new artifact and applied a column rename before all old queue workers had exited.

Evidence packet

Synthetic evidence: A reduced release timeline, fleet inventory, and worker error. Release names and schema are illustrative.

Time	Event
10:02	Migration renames invoices.total_cents to amount_cents
10:04	Web fleet reports release r52
10:05	queue:restart signal published
10:09	Two workers still report release r51; one is completing a long job

worker=w-7 release=r51 job=SendInvoice invoice=884 attempt=2
SQLSTATE[42703]: Undefined column: invoices.total_cents
queue_oldest_age=increasing web_health=ok

// r51 worker code still resident in the process
$total = $invoice->total_cents;

Candidate task

Describe customer, queue, and rollback impact. Rank mixed worker code/schema incompatibility against a failed migration, replica lag, stale configuration, and malformed payloads, then choose the discriminator that proves fleet and schema state. Give a safe incident mitigation, decide whether roll-forward or rollback is credible, design the compatible root migration/deployment sequence, and define mixed-version verification.

Discussion guide

The release-labelled worker and missing old column strongly support a destructive schema change while old code remained active. Confirm the effective schema on the worker’s database target, all process release IDs, restart-signal visibility, supervisor replacement, active long jobs, and queued payload versions. A green web health check says nothing about long-running consumers.

Containment may pause the affected queue and stop old workers gracefully, then deploy code/schema compatibility before retrying failed jobs. Whether to restore the old column, deploy a tolerant r52 consumer, or roll back depends on writes already made and whether old code can understand current data. Blind retries only consume attempts; switching the web symlink back may deepen incompatibility.

The root sequence is expand, migrate, contract: add amount_cents without removing total_cents; deploy code that can coexist and dual-write or otherwise preserve both representations; backfill in bounded resumable batches; switch reads after validation; restart and verify every long-lived runtime; stop old writes; remove the old column only in a later release after the rollback window closes. Test old/new producer-consumer combinations against the expanded schema, queued payload compatibility, worker draining and hard termination, backfill concurrency, retry safety, release-aware smoke checks, and rollback before contraction.

Owning guidance: Laravel deployment and long-running workers and Deployment and release engineering.

Canonical practice: LARAVEL-DEPLOY-03 — Design a compatible schema change, LARAVEL-DEPLOY-02 — Explain why queue workers run stale code