Skip to Content
Slloy Docs
Operations, Deployment & DR

Backup, Rollback & Disaster Recovery Runbook

Recovery runbooks for Cloudflare Worker code rollbacks, append-only R2 media backups (rclone copy), quarterly Neon 7-day PITR drills, and DLQ inspection.

Established after the 2026-11-16 six-dimension production audit, this runbook covers all three production asset classes. Remember the fundamental rule: wrangler rollback only rolls back Worker code—it never rolls back PostgreSQL data or R2 media files.

1. Three Asset Classes & Recovery Posture

Asset ClassBackup MechanismRetention WindowTarget RTOVerification Status
Worker Codewrangler rollback --env productionCloudflare version history1–3 minutesVerified (Deploy Production Cloudflare rollback job)
PostgreSQL DataNeon Point-In-Time Recovery (PITR)7 days (history_retention_seconds = 604800)30–60 minutesVerified 2026-11-16 (~18 min drill on drill-pitr-2026-11-16)
R2 Media (/uploads/*)Daily rclone copy (02:10 CST) to slloy-production-uploads-backupAppend-only (indefinite)Scales with volumeVerified 2026-11-16 (235 objects / 380.3 MiB)

2. R2 Media Backup & Restore (slloy-production-uploads-backup)

Daily Incremental Backup Architecture

  • Every object key generated by server/src/storage.ts (buildKey) includes a random 12-hex suffix (<YYYY>/<MM>/<genId>_<12hex>.<ext>), making object keys immutable.
  • The GitHub Actions R2 Backup workflow runs daily at 02:10 Beijing Time under environment: production using rclone copy (append-only; rclone sync is strictly forbidden because sync deletes destination files if the source bucket is accidentally emptied).
  • Credentials Trap Guard: R2 rclone requires S3-compatible credentials (32-char R2_ACCESS_KEY_ID + 64-char R2_SECRET_ACCESS_KEY with Object Read & Write on both slloy-production-uploads and slloy-production-uploads-backup). The workflow explicitly rejects CFAT_ Cloudflare Account API tokens at the Check credentials step.

Restoring Missing R2 Objects

# 1. Dry-run first to inspect what will be copied back (zero writes)
rclone copy r2:slloy-production-uploads-backup r2:slloy-production-uploads \
  --checksum --ignore-existing --dry-run --stats-one-line -v

# 2. Execute restore (copies only missing objects, never overwrites or deletes)
rclone copy r2:slloy-production-uploads-backup r2:slloy-production-uploads \
  --checksum --ignore-existing --stats-one-line -v

# 3. Single-object or single-month targeted restore
rclone copy r2:slloy-production-uploads-backup/<yyyy/mm>/ r2:slloy-production-uploads/<yyyy/mm>/ \
  --include "<genId>_*" -v

# 4. Smoke-check restored media via Worker route
curl -sI https://slloy.com/uploads/<yyyy/mm/xxx.png> | head -5

3. Neon PostgreSQL 7-Day PITR Drill & Emergency Recovery

Quarterly PITR Drill Procedure (Zero Impact on Production Branch)

Perform once per quarter in the Neon Console (slloy-production / nameless-lab-96517142):

  1. Choose Target Timestamp T: Pick a timestamp T earlier than the most recent generations row so you can prove rows after T are absent in the restored branch.
  2. Create Branch from History: In Neon Console -> Branches -> Create branch -> From history, select timestamp T.
  3. Verify in Read-Only Transaction (BEGIN READ ONLY) using slloy_com_owner (note: slloy_app does not have USAGE on schema drizzle unless explicitly granted):
    -- Primary Pass Criterion: Must return 0 rows after T
    SELECT count(*) FROM generations WHERE created_at > timestamptz '<T>';
    
    -- Secondary Checks: Total row count, latest row before T, and migration count
    SELECT count(*), max(created_at) FROM generations;
    SELECT count(*) FROM drizzle.__drizzle_migrations;
  4. Pass Criteria:
    • count(*) WHERE created_at > T is 0 (do not judge by how close max(created_at) is to T, because during low-traffic hours the last generation before T may be several hours earlier).
    • Known rows created after T in production do not exist in the restored branch.
    • Total elapsed time from click to verification < 60 minutes (2026-11-16 drill completed in 18 minutes).
  5. Delete Drill Branch Immediately & Verify Deletion:
    • Delete the drill branch in Neon Console so it does not accrue compute charges or risk accidental connection.
    • Re-probe the deleted branch endpoint: wait until PostgreSQL returns 28000 The requested endpoint could not be found (note: immediately after deletion, Neon's proxy may briefly return 28P01 password authentication failed—only 28000 confirms the endpoint is gone).

Emergency Production Database Recovery Sequence

If production data is corrupted and must be rolled back to timestamp T:

  1. Enter Maintenance Mode (Stop New Writes): Set MAINTENANCE_MODE = "1" in wrangler.jsonc (env.production.vars) and deploy (or temporarily detach slloy.com/* routes).
  2. Create PITR Branch at T in Neon Console.
  3. Audit Restored Branch: Verify users count > 0, created_at > T count = 0, and drizzle.__drizzle_migrations matches the code SHA.
  4. Repoint Hyperdrive & NEON_DATABASE_URL: Update slloy-production-neon Hyperdrive connection string and GitHub production secret NEON_DATABASE_URL to the new branch's direct hostname, then run check-pending-migrations.
  5. Deploy & Smoke Test: Confirm https://slloy.com/api/health returns ok: true and mode: "postgres".
  6. Exit Maintenance Mode: Set MAINTENANCE_MODE = "0" and redeploy.

4. Dead-Letter Queue (slloy-production-images-dlq) Handling

Messages in slloy-production-images-dlq indicate an image batch failed queue consumption 3 times.

  • User Impact Is Automatically Mitigated: server/src/imageSweeper.ts automatically marks any pending image generation older than 10 minutes as failed (failCode = "TIMEOUT") and refunds the user's credits.
  • Operator Inspection:
    npx wrangler queues info slloy-production-images
    npx wrangler queues info slloy-production-images-dlq
  • Automated Shadow Alert: Because the CI deploy token lacks Queues read permissions, server/src/opsAlert.ts monitors timeoutRefunds24h > 10 every 5 minutes and emails OPS_ALERT_EMAIL_TO whenever image timeout refunds spike.

On this page