Backup, Rollback & Disaster Recovery Runbook
Recovery runbooks for Cloudflare Worker code rollbacks, append-only R2 media backups (rclone copy), quarterly Neon 7-day PITR drills, and DLQ inspection.
Established after the 2026-11-16 six-dimension production audit, this runbook covers all three production asset classes. Remember the fundamental rule: wrangler rollback only rolls back Worker code—it never rolls back PostgreSQL data or R2 media files.
1. Three Asset Classes & Recovery Posture
| Asset Class | Backup Mechanism | Retention Window | Target RTO | Verification Status |
|---|---|---|---|---|
| Worker Code | wrangler rollback --env production | Cloudflare version history | 1–3 minutes | Verified (Deploy Production Cloudflare rollback job) |
| PostgreSQL Data | Neon Point-In-Time Recovery (PITR) | 7 days (history_retention_seconds = 604800) | 30–60 minutes | Verified 2026-11-16 (~18 min drill on drill-pitr-2026-11-16) |
R2 Media (/uploads/*) | Daily rclone copy (02:10 CST) to slloy-production-uploads-backup | Append-only (indefinite) | Scales with volume | Verified 2026-11-16 (235 objects / 380.3 MiB) |
2. R2 Media Backup & Restore (slloy-production-uploads-backup)
Daily Incremental Backup Architecture
- Every object key generated by
server/src/storage.ts(buildKey) includes a random 12-hex suffix (<YYYY>/<MM>/<genId>_<12hex>.<ext>), making object keys immutable. - The GitHub Actions
R2 Backupworkflow runs daily at02:10Beijing Time underenvironment: productionusingrclone copy(append-only;rclone syncis strictly forbidden becausesyncdeletes destination files if the source bucket is accidentally emptied). - Credentials Trap Guard: R2
rclonerequires S3-compatible credentials (32-charR2_ACCESS_KEY_ID+64-charR2_SECRET_ACCESS_KEYwithObject Read & Writeon bothslloy-production-uploadsandslloy-production-uploads-backup). The workflow explicitly rejectsCFAT_Cloudflare Account API tokens at theCheck credentialsstep.
Restoring Missing R2 Objects
# 1. Dry-run first to inspect what will be copied back (zero writes)
rclone copy r2:slloy-production-uploads-backup r2:slloy-production-uploads \
--checksum --ignore-existing --dry-run --stats-one-line -v
# 2. Execute restore (copies only missing objects, never overwrites or deletes)
rclone copy r2:slloy-production-uploads-backup r2:slloy-production-uploads \
--checksum --ignore-existing --stats-one-line -v
# 3. Single-object or single-month targeted restore
rclone copy r2:slloy-production-uploads-backup/<yyyy/mm>/ r2:slloy-production-uploads/<yyyy/mm>/ \
--include "<genId>_*" -v
# 4. Smoke-check restored media via Worker route
curl -sI https://slloy.com/uploads/<yyyy/mm/xxx.png> | head -53. Neon PostgreSQL 7-Day PITR Drill & Emergency Recovery
Quarterly PITR Drill Procedure (Zero Impact on Production Branch)
Perform once per quarter in the Neon Console (slloy-production / nameless-lab-96517142):
- Choose Target Timestamp
T: Pick a timestampTearlier than the most recentgenerationsrow so you can prove rows afterTare absent in the restored branch. - Create Branch from History: In Neon Console -> Branches -> Create branch -> From history, select timestamp
T. - Verify in Read-Only Transaction (
BEGIN READ ONLY) usingslloy_com_owner(note:slloy_appdoes not haveUSAGEon schemadrizzleunless explicitly granted):-- Primary Pass Criterion: Must return 0 rows after T SELECT count(*) FROM generations WHERE created_at > timestamptz '<T>'; -- Secondary Checks: Total row count, latest row before T, and migration count SELECT count(*), max(created_at) FROM generations; SELECT count(*) FROM drizzle.__drizzle_migrations; - Pass Criteria:
count(*) WHERE created_at > Tis0(do not judge by how closemax(created_at)is toT, because during low-traffic hours the last generation beforeTmay be several hours earlier).- Known rows created after
Tin production do not exist in the restored branch. - Total elapsed time from click to verification
< 60minutes (2026-11-16drill completed in18minutes).
- Delete Drill Branch Immediately & Verify Deletion:
- Delete the drill branch in Neon Console so it does not accrue compute charges or risk accidental connection.
- Re-probe the deleted branch endpoint: wait until PostgreSQL returns
28000 The requested endpoint could not be found(note: immediately after deletion, Neon's proxy may briefly return28P01 password authentication failed—only28000confirms the endpoint is gone).
Emergency Production Database Recovery Sequence
If production data is corrupted and must be rolled back to timestamp T:
- Enter Maintenance Mode (Stop New Writes):
Set
MAINTENANCE_MODE = "1"inwrangler.jsonc(env.production.vars) and deploy (or temporarily detachslloy.com/*routes). - Create PITR Branch at
Tin Neon Console. - Audit Restored Branch: Verify
userscount> 0,created_at > Tcount= 0, anddrizzle.__drizzle_migrationsmatches the code SHA. - Repoint Hyperdrive &
NEON_DATABASE_URL: Updateslloy-production-neonHyperdrive connection string and GitHubproductionsecretNEON_DATABASE_URLto the new branch's direct hostname, then runcheck-pending-migrations. - Deploy & Smoke Test: Confirm
https://slloy.com/api/healthreturnsok: trueandmode: "postgres". - Exit Maintenance Mode: Set
MAINTENANCE_MODE = "0"and redeploy.
4. Dead-Letter Queue (slloy-production-images-dlq) Handling
Messages in slloy-production-images-dlq indicate an image batch failed queue consumption 3 times.
- User Impact Is Automatically Mitigated:
server/src/imageSweeper.tsautomatically marks anypendingimage generation older than10minutes asfailed(failCode = "TIMEOUT") and refunds the user's credits. - Operator Inspection:
npx wrangler queues info slloy-production-images npx wrangler queues info slloy-production-images-dlq - Automated Shadow Alert: Because the CI deploy token lacks Queues read permissions,
server/src/opsAlert.tsmonitorstimeoutRefunds24h > 10every5minutes and emailsOPS_ALERT_EMAIL_TOwhenever image timeout refunds spike.
Feature Environments (Per-Branch Preview Workers)
Ephemeral per-branch Cloudflare Workers (<name>.zsg8482.workers.dev), PR /feature-env slash commands, shared staging DB/R2 architecture, and automated TTL cleanup.
Public Site Design System & Tokens
Complete token specification, dark/monochrome themes, typography scale, Base UI constraints, Motion transitions, and design-tokens test enforcement.