Skip to Content
Slloy Docs
Architecture, Design & Testing

Client Disconnect & Request Cancellation Observability

Architecture decision and telemetry design for measuring Cloudflare Workers request cancellation behavior, clientRequestId correlation, and stale pending recovery.

Documented in decisions/request-cancellation-observability.md (2026-11-11), this page outlines how Slloy treats client disconnects during image and video generation on Cloudflare Workers and why our recovery safeguards (imageSweeper.ts and keepAlive) are designed never to gamble on undefined platform cancellation timing.

1. Problem Statement & Empirical Findings

Two historical comments in the codebase assumed that "Cloudflare Workers immediately cancel in-flight execution the moment a client disconnects". However, controlled local probes against wrangler dev (workerd) with curl -m 1 disconnecting after 1 second against 3s and 45s upstream delays showed that workerd itself completed both upstream fetch calls and logged WORKER_UPSTREAM_DONE 200.

Because client disconnect propagation happens at Cloudflare's edge ingress layer—and how long an isolate continues executing after a client closes the TCP/HTTP3 stream is undefined platform behavior—Slloy follows six strict architectural rules:

  1. Never Branch Business Logic on Undefined Edge Cancellation Timing:
    • Primary image execution runs inside the Cloudflare Queue consumer (IMAGE_QUEUE in server/src/imageJobs.ts), which is completely decoupled from the browser HTTP request lifecycle.
    • Video task creation wraps the post-charge provider.createVideoTask() + repo.finishGeneration(taskId) sequence in keepAlive(c, submit) (ctx.waitUntil) so task_id is persisted even if the client disconnects mid-request.
  2. Always Keep the 10-Minute Database Sweeper (imageSweeper.ts):
    • The 10-minute stale pending image sweeper (SWEEP_CRON = "*/5 * * * *") is mandatory regardless of client disconnect behavior because DLQ exhaustion, consumer crashes, Worker version rollouts, or transient DB errors can also leave rows in status = 'pending'.
  3. Verify Edge Behavior via Controlled Staging Probes (Q1–Q3):
    • Test 3s, 45s, and 150s upstream delays on staging.slloy.com (real Cloudflare edge) with a 1s client disconnect to measure:
      • Q1: Does request.signal fire an abort event?
      • Q2: Are in-flight upstream fetch calls and subsequent DB writes terminated?
      • Q3: How long does execution survive after disconnect?

2. Correlation & Telemetry Architecture (clientRequestId)

When observing real user disconnects in production logs:

  • Why cf-ray Cannot Be the Primary Correlation Key:
    • On inline fallback paths or aborted requests, if the user closes the tab before the HTTP response headers arrive, the browser never receives the cf-ray header.
    • Instead, the browser generates a UUID (clientRequestId = crypto.randomUUID()) before calling /api/generate and passes it in the request header (cf-ray is logged alongside as a secondary field).
  • Ground-Truth "User Left" Signal (pagehide Beacon):
    • When a tab refreshes, closes, or navigates away during generation, the frontend fires navigator.sendBeacon("/api/telemetry/leave", JSON.stringify({ clientRequestId })) (with mobile backgrounding via visibilitychange === "hidden" tagged separately as kind=hidden).
    • The server validates the UUID shape, rate-limits the endpoint, and logs a single structured line [gen] client_left client=<uuid> ray=<ray> kind=pagehide|hidden without writing to any database table.

Structured Log Events (log.emit("gen", ...))

Event TagTrigger LocationKey FieldsQuestion Answered
[gen] acceptedBefore acceptJob() returnspath=queue|inline, batch, gen, count, refs, chargeMsDenominator for queue vs. inline path split
[gen] phaseAfter charge, queue.send, upstream return, R2 persist, and DB commitphase, elapsedMs, genPhase-by-phase latency distribution
[gen] request_abortedc.req.raw.signal abort listener on /generatephase, elapsedMs, genAnswers Q1 and Q3 on live edge traffic
[gen] client_leftPOST /api/telemetry/leaveclient, ray, kind=pagehide|hiddenGround truth that the user left the page
[gen] inline_doneAfter inline runImageJob() finishescanceled, durationMs, ok, refunded, failCodeAnswers Q2 and Q5
[gen] persist_orphanR2 persistence succeeded but finishGeneration lost the racegen, keyTracks orphaned R2 objects when sweeper settles first
[sweep] image_roundEnd of imageSweeper.ts round (when scanned > 0)scanned, refunded, waiting, minAgeMinReal-time count of swept stale image jobs (Q4)

3. Direct SQL Quantification for Stale Pending Jobs (Q4)

Always prefer direct PostgreSQL queries over log aggregation when measuring long-term stale/swept image generation rates:

SELECT count(*) FILTER (WHERE status = 'pending') AS currently_stuck,
       count(*) FILTER (WHERE status = 'failed' AND fail_code = 'TIMEOUT') AS swept_or_timed_out
  FROM generations
 WHERE kind = 'image'
   AND created_at > now() - interval '7 days'
   AND created_at < now() - interval '10 minutes';

On this page