D1 recovery
Fence every D1 write, restore databases by Time Travel bookmark, reconcile durable effects, verify, and reopen — as a reviewed, resumable operation. Plus protected exports.
Restoring a D1 database on its own is dangerous. It can rewind idempotency records, resurrect queue messages that were already delivered, and leave old writers committing into the restored data. Fenced recovery makes restore an operation the deployment takes part in. Traffic stays out until the schema, derived state and external-effect reconciliation are confirmed.
Each database is an independent restore point. Cloudflare has no atomic
restore across AUTH_DB, DB and other stores. A restore of several bindings
is an ordered saga with per-binding results. It never guarantees consistency
between those databases.
Enable
database: defineDatabase("cloudflare-d1", {
recovery: { enabled: true }, // operatorKey defaults to QB_RECOVERY_OPERATOR_KEY
// recovery: { enabled: true, operatorKey: { env: "OPS_RECOVERY_KEY" } },
}),Then set the operator key as a Worker secret (32 or more characters) for each target:
openssl rand -hex 32 | npx wrangler secret put QB_RECOVERY_OPERATOR_KEY --env productionThe compiler accepts the option only for cloudflare-d1 on the Cloudflare
runtime. Enabling it adds:
- the per-deployment
QuickbackControlDurable Object (QB_CONTROL). This is the same object as durable effects. There is never a second control object; - write admission on every D1 binding the Worker receives (auth, features, files, webhooks, audit);
- an admission check on HTTP, queue, cron and inbound-email entry points;
- the signed operator bridge at
POST /__quickback/operator.
Without the secret, the bridge answers 503 RECOVERY_OPERATOR_DISABLED. It
never falls back to another credential.
How writes are admitted
Each isolate keeps one observation of { mode, generation } per D1 binding
from the control object. The observation is refreshed at most every 5 seconds
and stamped with the Worker clock taken before the call.
Every statement that is not provably read-only (SELECT, VALUES,
EXPLAIN, a read PRAGMA) goes to D1 as a native batch. That includes
statements behind Drizzle, Better Auth and the generated runtime. The batch
starts with one assertion:
SELECT CASE WHEN ?1 >= now_ms - 30000 THEN 1 ELSE <raise> END -- ?1 = observation time- Fenced binding: refused before submission with
DEPLOYMENT_FENCED(503,committed: false). - Stale observation: D1 itself rejects the assertion, so the whole batch rolls back with
RECOVERY_ADMISSION_EXPIRED(503,committed: false). This holds however long the request ran, and whether the invocation crashed or the Worker clock lagged. - Control object unreachable: an observation younger than 25 seconds is still used. After that, writes fail closed with
RECOVERY_CONTROL_UNAVAILABLE(503).
fence records its time. drain reports quiescent only once 35 seconds
have passed since the fence (the 30-second window plus 5 seconds of clock-skew
margin) and no plan-037 receipt is still unsettled. Every observation that said
"open" was taken before the fence, so no write admitted under it can commit
after that point. No permit is held. A crashed Worker leaves nothing to release
and nothing to wait for: its admission simply expires inside D1.
Cost. A write gets no extra round trip: the assertion reads no table and travels in the same D1 batch. A single statement becomes a two-statement batch. Each isolate makes one control-object call per 5 seconds while it serves traffic. Reads are unchanged. Latency and availability on a deployed account have not been measured yet. The control object is a hard dependency for traffic: it is a single Durable Object per deployment.
exec() cannot carry the assertion, so it is refused on guarded bindings. Use
prepare() or batch(). Writes through .raw() come back from the batched
result in object-key order. Duplicate column names in a write's RETURNING
clause are not preserved.
Entry points while fenced
| Entry | While any binding is fenced |
|---|---|
| HTTP | 503 DEPLOYMENT_FENCED with Retry-After: 30, for reads and writes, before auth or database middleware |
| Queue consumer | the batch is retried after 30 seconds; no handler runs |
| Cron | the run is skipped and logged |
| Inbound email | rejected with a maintenance reason |
A response that took longer than 30 seconds re-checks before release. If the
deployment was fenced, or its generation changed meanwhile, the response is
withheld with 503 RECOVERY_GENERATION_CHANGED. Every response carries
x-quickback-generation, the sum of the per-binding generations. With
read replication, a client that sends
x-quickback-generation along with its bookmark has the bookmark dropped when
the generation differs. A bookmark from before a restore can then never point
past it. A bookmark sent without the header behaves as before: the wait is
bounded by bookmarkWaitMs.
What is not covered
The guarantee covers generated entry points only. These are unmanaged writers, which you must stop and reconcile yourself:
- Durable Object classes, which receive the raw
env. The restore plan lists every class exceptQuickbackControlandBroadcaster, and refuses to plan until each is declared with--writers-stopped; - Workflows, also listed by the plan;
- custom Workers, scripts or external clients that use the database through the API, and
wrangler d1 execute; - consumers outside this Worker, which receive effect envelopes without admission.
Streaming response bodies continue after release and are not re-checked.
Restore
# 1. Read-only plan. Resolves the timestamp to an immutable bookmark per binding.
quickback db restore-plan --env production --url https://api.example.com \
--binding DB --timestamp 2026-11-18T10:00:00Z --out restore.json
# …or pin the exact bookmarks a checkpoint recorded:
# --checkpoint checkpoint.json (instead of --timestamp)
# 2. Review restore.json, then apply with the exact digest it printed.
QB_RECOVERY_OPERATOR_KEY=… quickback db restore-apply restore.json --confirm <plan-digest>
# 3. Reopen with the resume digest that apply printed after verification.
quickback db restore-resume --operation r-… --plan restore.json --confirm <resume-digest>The plan records:
- the target: account, logical environment, Wrangler environment, Worker and URL;
- each binding's database ID, current bookmark, target bookmark and timestamp;
- the local migration manifest hash;
- the control contract version and current generations;
- the enabled recovery hooks (versioned);
- the unmanaged writers you declared stopped.
A timestamp is not an exact point. Time Travel restores to "any minute":
a timestamp resolves to the latest bookmark the platform recorded at or before
it, which can be minutes older than the timestamp. Writes committed shortly
before the timestamp can be lost. In a live drill, every timestamp from 9 s to
35 s after a burst of writes resolved to the bookmark from before the burst. To
restore an exact point, take a checkpoint when you have one and
plan with --checkpoint. The plan refuses a checkpoint that was edited, or that
has no bookmark for a selected binding's database.
The plan contains no rows and no credentials. Its digest is the SHA-256 of the
canonical document. Repeat --binding to restore several databases as an
ordered saga.
restore-apply rereads the target identity, the Worker's recovery contract and
the local migrations, and refuses if anything changed since review. The
operation ID comes from the plan digest. Each stage is a compare-and-set in the
control object, outside every restored database, and appended to its history:
| Stage | Work |
|---|---|
planned | operation recorded |
fenced | fence each binding |
drained | poll drain until quiescent (--drain-timeout, default 120 s). Unsettled effect receipts stop here: resolve them with quickback effects resolve |
prior-bookmarks-recorded | current bookmarks captured: the undo point |
restoring | each binding is marked unknown, then restored by bookmark (wrangler d1 time-travel restore --json). The returned undo bookmark is recorded at once |
restored | every binding restored |
reconciling | rotateGeneration per binding (idempotent across crashes), then reconcileRestoredEffects on the effects binding |
verified | the checks below passed; a resume digest is recorded |
resumed | resume each binding (only via restore-resume) |
Running restore-apply again with the same plan continues from the recorded
stage. It never starts a second blind restore.
- Lost acknowledgement. If a restore call fails or returns no usable acknowledgement, the binding stays
unknown. Completion is never inferred from a changed bookmark. Rerun with--retry-unknown: restoring to the same immutable bookmark under the fence is safe. - Partial success. With several bindings, finished ones stay
restored, and the run stops with the fence held. - Failure. Every failure leaves the fence held.
restore-status --operation <id> --url …shows the stage, history, per-binding restore results, recorded bookmarks, rotation, effect counts and verification.
Verification
restore-verify restore.json runs the same checks under maintenance through the
bridge. They are bounded and read-only:
PRAGMA quick_checkreportsok;- the restored
d1_migrationslist equals the deployed Worker's migrations exactly. If not, the restored schema is incompatible: apply a separately reviewed migration or deployment, then verify again. Migrations are never rolled back automatically; - the generation was rotated;
- no durable effect from an older generation is left unreconciled;
- no virtual tables exist unless the
searchrecovery hook is registered. That hook belongs to plan 038, full-text search, and is not registered yet, so a database with FTS tables does not reopen.
Two failed verifications stop the operation until you pass
--acknowledge-failures. restore-resume reruns verification and recomputes
the digest. Any change since verification refuses the reopen. That includes a
moved generation, a schema change or a reconciled effect.
Recovery hooks
| Hook | Covers |
|---|---|
admission | the fence and drain above |
generation-rotation | the non-rewound generation. Old effect envelopes become stale. The read cache does not depend on it: its epoch token is restore-safe on its own |
effects-reconcile | an intent with an accepted receipt is marked accepted and not resent. An uncertain one is quarantined. A never-dispatched one is quarantined, or reissued in the new generation with --reissue-pending at plan time |
search | not registered (plan 038). Virtual tables block verification |
Undo
quickback db restore-undo-plan --operation r-… --plan restore.json --out undo.jsonThis builds a new reviewed plan whose targets are the bookmarks captured at
prior-bookmarks-recorded. Apply it like any plan. It is another fenced saga.
It does not reverse external effects that ran in between.
Checkpoints
quickback db checkpoint --env production --binding DB [--binding AUTH_DB] --out checkpoint.json
records each binding's current bookmark and migration manifest. It is an
inventory of exact points to plan against (restore-plan --checkpoint), not a portable backup. Time Travel
retention depends on your plan and backend. The CLI does not assume a horizon.
A bookmark the platform no longer has fails with RECOVERY_BOOKMARK_UNAVAILABLE.
Exports
quickback db export-plan --env production --binding DB --format d1 \
--directory /secure/exports --out export.json
quickback db export-apply export.json --confirm <digest>
quickback db export-verify /secure/exports/<file>.sql.receipt.json
quickback db export-status export.json| Format | Contains | Restorable into D1 |
|---|---|---|
d1 | ordinary tables (whole database or --table) | yes |
sqlite3_cli_only | the whole database, including virtual tables (Wrangler --compatibility sqlite3_cli_only) | no, an analysis artifact only |
export-plan inventories tables, including virtual tables and FTS shadow
tables, without exporting data. A d1 export of a database with virtual tables
is refused unless you select ordinary tables with --table. Tables are never
omitted silently. After importing such an export, rebuild the search state.
export-apply:
- Rechecks the digest, the target and the directory.
- Records each binding's bookmark.
- Runs
wrangler d1 exportinto an exclusively created.partialfile. - Verifies the size, SHA-256, format and the planned tables.
- Renames the file atomically and writes a receipt.
The directory must be an absolute real path, with no symlink, outside every
Git worktree. It is created with mode 0700. Files and receipts get 0600.
Receipts hold hashes, sizes, bookmarks and table names, never rows, SQL or
signed URLs. Wrangler output is never echoed.
An interrupted or failed export stays .partial and cannot verify. Rerun with
--restart to discard it. Repeated bindings are exported independently, each
at its own bookmark. Exports offer no cross-database snapshot, upload, Git
attachment or retention deletion.
A remote D1 export can make the database briefly unavailable to queries. Export does not fence. Run it in a quiet period, or fence first.
Operator bridge
The CLI signs every request with the operator key from its own process environment. Plan files never contain the key. The signed string covers the method, the exact origin and path, a millisecond timestamp, a single-use nonce and the SHA-256 of the body:
QBOP1\nPOST\nhttps://api.example.com/__quickback/operator\n<timestamp>\n<nonce>\n<body-sha256>The Worker checks it before any auth or database middleware.
- Timestamps more than 5 minutes off are refused.
- Nonces are claimed in the control object, so a replay is refused.
- A request signed for another origin fails. Use a different key per target.
- Session cookies, API keys and other application credentials cannot reach any operation.
The bridge also carries the plan-037 operator path:
quickback effects state --url https://api.example.com [--binding DB]
quickback effects list --url https://api.example.com --state quarantined|dead|pending
quickback effects replay <id> --url https://api.example.com --confirm-hash <payload_hash> [--reconciled not-delivered]
quickback effects resolve <id> --url https://api.example.com --binding DB --dispatch accepted|failed
quickback effects reconcile --url https://api.example.com [--reissue-pending]list returns IDs, states, attempts, error classes and payload hashes. It
never returns payloads.
Limits
- The admission window is fixed: 30 seconds plus 5 seconds of skew. A single D1 batch that starts executing later than that after its observation fails and must be retried.
- A deployed-account drill has not been run yet. Before relying on this in production, rehearse export, verify, restore and resume against a disposable staging database, and keep the receipts.
- Tenant Durable Object recovery is separate. See Tenancy.
- Restoring the features database rewinds
org_data_keys. Organization data keys live there, and tenant objects are not rewound with it. A bookmark from before an organization's key rotation (or before its key was created) brings back the older key table: values in tenant objects written under the removed version, and encrypted tenant dumps sealed to it, no longer open (ENCRYPTION_UNREADABLE,TENANT_DUMP_KEY_VERSION_UNKNOWN). Prefer a bookmark after the last rotation. After reopening, each tenant object follows the edge back to the key version that exists, so take a fresh backup of each tenant then. Recovery reads and writes keys through the same guarded bindings as every other generated write, so key creation is fenced with the rest.
Durable effects
Record queue effects in the same D1 transaction as the writes, deliver them at least once with leases and receipts, and fence them across restores.
Neon
Use Neon serverless PostgreSQL with Quickback. Connection modes, Worker-scoped RLS, the set_config claim model, and the deploy workflow.