Client Disconnect & Request Cancellation Observability
Architecture decision and telemetry design for measuring Cloudflare Workers request cancellation behavior, clientRequestId correlation, and stale pending recovery.
Documented in decisions/request-cancellation-observability.md (2026-11-11), this page outlines how Slloy treats client disconnects during image and video generation on Cloudflare Workers and why our recovery safeguards (imageSweeper.ts and keepAlive) are designed never to gamble on undefined platform cancellation timing.
1. Problem Statement & Empirical Findings
Two historical comments in the codebase assumed that "Cloudflare Workers immediately cancel in-flight execution the moment a client disconnects". However, controlled local probes against wrangler dev (workerd) with curl -m 1 disconnecting after 1 second against 3s and 45s upstream delays showed that workerd itself completed both upstream fetch calls and logged WORKER_UPSTREAM_DONE 200.
Because client disconnect propagation happens at Cloudflare's edge ingress layer—and how long an isolate continues executing after a client closes the TCP/HTTP3 stream is undefined platform behavior—Slloy follows six strict architectural rules:
- Never Branch Business Logic on Undefined Edge Cancellation Timing:
- Primary image execution runs inside the Cloudflare Queue consumer (
IMAGE_QUEUEinserver/src/imageJobs.ts), which is completely decoupled from the browser HTTP request lifecycle. - Video task creation wraps the post-charge
provider.createVideoTask()+repo.finishGeneration(taskId)sequence inkeepAlive(c, submit)(ctx.waitUntil) sotask_idis persisted even if the client disconnects mid-request.
- Primary image execution runs inside the Cloudflare Queue consumer (
- Always Keep the 10-Minute Database Sweeper (
imageSweeper.ts):- The
10-minute stale pending image sweeper (SWEEP_CRON = "*/5 * * * *") is mandatory regardless of client disconnect behavior because DLQ exhaustion, consumer crashes, Worker version rollouts, or transient DB errors can also leave rows instatus = 'pending'.
- The
- Verify Edge Behavior via Controlled Staging Probes (
Q1–Q3):- Test
3s,45s, and150supstream delays onstaging.slloy.com(real Cloudflare edge) with a1sclient disconnect to measure:- Q1: Does
request.signalfire anabortevent? - Q2: Are in-flight upstream
fetchcalls and subsequent DB writes terminated? - Q3: How long does execution survive after disconnect?
- Q1: Does
- Test
2. Correlation & Telemetry Architecture (clientRequestId)
When observing real user disconnects in production logs:
- Why
cf-rayCannot Be the Primary Correlation Key:- On inline fallback paths or aborted requests, if the user closes the tab before the HTTP response headers arrive, the browser never receives the
cf-rayheader. - Instead, the browser generates a UUID (
clientRequestId = crypto.randomUUID()) before calling/api/generateand passes it in the request header (cf-rayis logged alongside as a secondary field).
- On inline fallback paths or aborted requests, if the user closes the tab before the HTTP response headers arrive, the browser never receives the
- Ground-Truth "User Left" Signal (
pagehideBeacon):- When a tab refreshes, closes, or navigates away during generation, the frontend fires
navigator.sendBeacon("/api/telemetry/leave", JSON.stringify({ clientRequestId }))(with mobile backgrounding viavisibilitychange === "hidden"tagged separately askind=hidden). - The server validates the UUID shape, rate-limits the endpoint, and logs a single structured line
[gen] client_left client=<uuid> ray=<ray> kind=pagehide|hiddenwithout writing to any database table.
- When a tab refreshes, closes, or navigates away during generation, the frontend fires
Structured Log Events (log.emit("gen", ...))
| Event Tag | Trigger Location | Key Fields | Question Answered |
|---|---|---|---|
[gen] accepted | Before acceptJob() returns | path=queue|inline, batch, gen, count, refs, chargeMs | Denominator for queue vs. inline path split |
[gen] phase | After charge, queue.send, upstream return, R2 persist, and DB commit | phase, elapsedMs, gen | Phase-by-phase latency distribution |
[gen] request_aborted | c.req.raw.signal abort listener on /generate | phase, elapsedMs, gen | Answers Q1 and Q3 on live edge traffic |
[gen] client_left | POST /api/telemetry/leave | client, ray, kind=pagehide|hidden | Ground truth that the user left the page |
[gen] inline_done | After inline runImageJob() finishes | canceled, durationMs, ok, refunded, failCode | Answers Q2 and Q5 |
[gen] persist_orphan | R2 persistence succeeded but finishGeneration lost the race | gen, key | Tracks orphaned R2 objects when sweeper settles first |
[sweep] image_round | End of imageSweeper.ts round (when scanned > 0) | scanned, refunded, waiting, minAgeMin | Real-time count of swept stale image jobs (Q4) |
3. Direct SQL Quantification for Stale Pending Jobs (Q4)
Always prefer direct PostgreSQL queries over log aggregation when measuring long-term stale/swept image generation rates:
SELECT count(*) FILTER (WHERE status = 'pending') AS currently_stuck,
count(*) FILTER (WHERE status = 'failed' AND fail_code = 'TIMEOUT') AS swept_or_timed_out
FROM generations
WHERE kind = 'image'
AND created_at > now() - interval '7 days'
AND created_at < now() - interval '10 minutes';Reference Image Preprocessing & Trust Boundary
Browser-side EXIF/sRGB/white-background normalization (image-prep.ts) paired with low-cost Worker header sniffing (imageSniff.ts).
Testing Strategy, Postgres Template Cloning & Worker E2E
Multi-tier test suite (server unit, design-tokens/i18n tools, millisecond Postgres template DB cloning, Playwright browser tests, and workerd E2E).