Database per Tenant
tenancy.isolation 'durable-object' gives every organization its own SQLite database in a Durable Object, served by a generated tenant Worker, while the edge Worker keeps auth, routing and platform tables.
Set tenancy.isolation: 'durable-object' to give every organization its own
SQLite database. Organization-scoped tables move out of the shared features D1
into one Durable Object per organization. Each tenant's reads and writes land
on its own object, not on one central database.
import { defineConfig, defineRuntime, defineDatabase, defineAuth } from "@quickback/compiler";
export default defineConfig({
name: "events-app",
template: "hono",
features: { organizations: true },
providers: {
runtime: defineRuntime("cloudflare"),
database: defineDatabase("cloudflare-d1"),
auth: defineAuth("better-auth", { plugins: { organization: true } }),
},
tenancy: {
isolation: "durable-object",
key: "organization",
jurisdiction: "eu", // optional: pin every tenant object to a jurisdiction
backup: { bucket: "TENANT_BACKUPS", every: "6h", retainDays: 90 },
lookups: { inviteCode: { table: "invites", column: "code", hash: true } },
},
});| Option | Default | Effect |
|---|---|---|
isolation | 'shared' | 'shared': one features D1 with organization_id on every row. 'durable-object': one SQLite Durable Object per organization. |
key | 'organization' | The tenant key. Only the active organization is supported. |
jurisdiction | none | 'eu' or 'fedramp'. Every tenant object is created in that jurisdiction. Fixed once deployed: changing it fails the build (TENANCY_JURISDICTION_CHANGED), because the new value addresses a different, empty namespace. |
backup | none | Alarm-driven dumps of every tenant object to R2: bucket (the tenant Worker's R2 binding, default TENANT_BACKUPS), bucketName (default <name>-tenant-backups), every (30m / 6h / 1d, at least 15 minutes, default 6h; or 'off': no scheduled dumps — point-in-time recovery is the routine backup, and the bucket only serves operator backups, dump restores and moves), retainDays (default 30). See Backups and restore. |
lookups | none | Platform lookup indexes: name → { table, column, hash? } on a tenant table. hash: true stores a keyed hash, never the value (credentials). See Lookup indexes. |
outbox | { receipts: { retainDays: 7, maxRows: 50000 } } | Delivery receipts: one row per outbox entry the edge delivered or failed, kept retainDays and at most maxRows per object. receipts: false keeps none. See Outbox delivery. |
allowNamespaceDeletion | false | Permit deleting the tenant class. See Deleting the namespace. |
vars | none | Non-secret Worker vars tenant code may read as env.<NAME>. See Vars in tenant code. |
operator.http | unset (every route, with a warning) | false: no generated operator routes at all; { allow: [...] }: only the named routes. See Operator routes. |
operator.purge | deprecated — use operator.http | true, or false when your code imports purgeTenant: mount the raw POST /tenants/:id/purge. Refused beside operator.http. |
Nothing changes for projects that leave isolation unset.
Two Workers
The build emits two Workers from one project.
| Worker | Config | Holds |
|---|---|---|
| Edge | wrangler.toml, src/index.ts | Better Auth, routing, the features D1 (platform tables + tenant_registry), the CMS and Account SPAs |
| Tenant | wrangler.tenant.jsonc, src/tenant/index.ts | QuickbackTenant, a SQLite-backed Durable Object running the tenant routes. No Better Auth, no D1, no auth secrets. |
Both Workers use the project's providers.runtime.config.compatibilityDate (default 2025-01-01); below 2025-11-17 the tenant Worker also gets the enable_ctx_exports flag. Set the newest date your installed wrangler's workerd supports.
The edge authenticates the request, resolves the tenant (the active
organization, or a lookup for scope tokens), and calls
TENANT.getByName(organizationId).handle(request, principal, version) over
RPC. The object runs the generated CRUD, batch and action handlers on a
db built over its own ctx.storage (drizzle-orm/durable-sqlite). The firewall
still applies inside the object, and the object refuses a principal for another
organization (403 TENANT_MISMATCH).
A tenant table with no routes left (routes: false on read, create, update and
delete) gets no routes module, no mount on either Worker and no OpenAPI path
items: its paths answer 404 to every caller, signed in or not. Server code
still reads and writes the table.
The tenant Worker declares its class with exports (no [[migrations]]) and
carries no binding to itself. The edge binds it with script_name = "<name>-tenant". Deploy the tenant Worker
first: npm run deploy runs npm run deploy:tenant before the edge deploy.
When providers.runtime.config.accountId is set, the tenant Worker carries the
same account_id as the edge, so it cannot deploy into a different account.
The build refuses to change which tenant Worker a deployed edge binds
(TENANCY_TENANT_WORKER_CHANGED). It reads every QuickbackTenant binding in
the existing wrangler.toml (top level and [env.*], in any key order or
quoting) and compares it with the new config. If a binding cannot be read —
an inline table, a binding without script_name, an edge table with no
name, or a table declared twice — the build fails with
TENANCY_TENANT_WORKER_UNKNOWN instead of skipping the check. Two cases are refused:
- a deployed edge script would bind a different tenant script;
- a tenant script bound today is no longer bound by any target. This covers renaming a target's edge and tenant Worker together, and dropping a target.
A different tenant script name addresses a new, empty namespace, so every
tenant would look wiped. To move on purpose, migrate the tenants first, then
remove the old TENANT binding from wrangler.toml and rebuild. A build
without wrangler.toml cannot run this check, and warns.
Named environments
With named environments, every
target deploys its own pair: its edge Worker (environments.<key>.name) and
its own tenant Worker (environments.<key>.tenantWorker.name). The tenant
Worker's script name owns the QuickbackTenant namespace, which holds every
organization's database. Like name, it is required on every target and never
derived.
environments: {
dev: {
name: 'events-app-dev',
tenantWorker: { name: 'events-app-tenant-dev', backupBucketName: 'events-app-tenant-backups-dev' },
/* worker, bindings, domains … */
},
production: {
name: 'events-app',
tenantWorker: { name: 'events-app-tenant', backupBucketName: 'events-app-tenant-backups' },
/* … */
},
}All targets run the same bundle. The build emits:
wrangler.toml: one[[env.<key>.durable_objects.bindings]]per target, withscript_nameset to that target's tenant Worker. Durable Object bindings do not inherit, and the top level emits no tenant binding, so no target can reach another target's tenant databases.wrangler.tenant.jsonc: no top-levelmainorexports, so a barewrangler deploy -c wrangler.tenant.jsoncfails with Missing entry-point. The top-levelnameis an inert placeholder,<name>-tenant-unselected. A positional entry (wrangler deploy -c wrangler.tenant.jsonc src/tenant/index.ts) deploys an unbound Worker under that name. Without the placeholder, Wrangler would take thepackage.jsonname (the production edge) and deploy tenant code over it. No target may use the placeholder. Oneenv.<key>block per target carriesmain,exports, that target'saccount_id(itsaccountId, orproviders.runtime.config.accountId), itsTENANT_BACKUPSbucket (backupBucketName), and the rate-limit namespaces from its own edge stanza.package.json:deploy:<key>:tenant. Eachdeploy:<key>runs it before the edge deploy.dev:tenantruns thedevtarget. There is no baredeploy:tenant.
The tenant registry, lookups and outbox live in each target's own features D1
(worker.bindings.d1.features). Jobs ride each target's own queues
(worker.bindings.queues).
The build refuses:
- a target without
tenantWorker.name; - two targets with the same tenant Worker;
- a backup bucket that is another target's bucket, or one of the same target's
worker.bindings.r2Buckets(the same per-target R2 uniqueness rule as every other bucket); - a tenant Worker named like an edge or files Worker, or like the top-level
project
name; backupBucketNamemissing whiletenancy.backupis set, or set while it is not.
Only the target that deploys as the project name may reuse the single-target
<name>-tenant script and <name>-tenant-backups bucket. Set them on that
target when you adopt environments in a deployed project, so it keeps its
tenant databases. The continuity check above refuses any other name for it.
A tenantWorker is also refused when tenancy.isolation is not
'durable-object'.
The sharing refusals apply even when two targets deploy into different
Cloudflare accounts (accountId), where the same names would not collide.
This is deliberately conservative. Per-client accounts (one target per client)
may relax it later.
The edge↔tenant contract is versioned (TENANT_RPC_VERSION in
src/tenant/contract.ts, the same constant in both Workers). A tenant Worker
answers versions from TENANT_RPC_MIN_VERSION up to its own and returns
503 TENANT_RPC_VERSION_MISMATCH otherwise. The current protocol is version 13
(version 3 put dump restore under the shared restore fence and added the
authorization epoch; version 4 adds tenant commands and epoch-carrying restore
permits; version 5 adds callJob() for result jobs; version 6
adds queue outbox entries for platform queues; version 7
the bounded backup read; version 8 the callJob epoch fence; version 9 the
backup-fenced purge; version 10 room commit permits; version 11
leased outbox delivery; version 12 revoke outbox entries
for scope revocation; version 13 armSchedules() for
tenant schedules). An edge below version 11 gets no
inline claims and keeps using takeOutbox / ackOutbox. An edge below
version 12 is never handed a revoke entry, by a claim or by takeOutbox:
those entries wait for the new edge, so no revocation is settled unapplied. It still
accepts version-2 to version-5 edges: no method those edges
call changed meaning, so deploying the tenant Worker first opens no 503 window.
An edge older than version 6 is handed no outbox entries from the first
queue entry on, so it never acknowledges a message it would not send; those
entries wait, in order, for the new edge,
and a version-2 edge's dump restore fails closed until the edge is deployed.
A percentage (gradual) deploy of the tenant Worker is safe from version 3
to 4. While it runs, some tenant objects and their control objects use the old
code and some the new, and both directions work. A control object answers an
older holder's permit request with the plain token string that holder expects,
and the holder releases it as before. A version-4 holder whose control object
still runs version 3 gets a version-mismatch refusal before anything is
written, then retries at TENANT_RPC_MIN_VERSION with the old protocol. Its
permit then carries no authorization epoch, so command() answers
unavailable / TENANT_RPC_VERSION_MISMATCH until that control object is
upgraded. handle(), jobs and alarms keep serving, and the restore fence is
the same permit table either way. command() itself accepts only version 4
(TENANT_PERMIT_VERSION). Upgrading from version 1 requires both Workers to
be deployed. Old edge
calls fail closed while the tenant Worker is upgraded first.
Tenant commands
Besides handle(), the object has one more entry point for trusted edge code:
command(version, request). It runs a compiler-owned program inside the
object. Callers pick a command by name and pass its fields. They never send
SQL, callbacks, method names or credentials, and app code cannot add commands.
Every name is reserved under qb.*.
import { tenantCommand } from './lib/tenants';
const res = await tenantCommand(env, organizationId, { name: 'qb.epoch' });
if (res.status === 'ok') res.reply.authEpoch; // { tenantId, authEpoch, lifecycle: { state, schemaVersion } }| Command | Mode | Reply |
|---|---|---|
qb.epoch | read | { tenantId, authEpoch, lifecycle: { state: 'active', schemaVersion } }: the authorization epoch |
qb.admitScope | read | Registered only when a scope kind is tenant-backed. { kind, instanceId, tenantId, authEpoch, roles, subjectId, provingTable, subKeys }, or denied with SCOPE_EPOCH, SCOPE_NOT_FOUND, SCOPE_NO_ROLE, SCOPE_WRONG_SUBJECT or SCOPE_AMBIGUOUS |
qb.scopeGrantRead | read | Registered only when a tenant-backed kind declares oauth.serverGrant. { kind, instanceId, subjectId, tenantId, authEpoch, linkedUserId, roles, subject, instance } (identity and instance projections as JSON), or denied with SCOPE_NOT_FOUND or SCOPE_GRANT_SUBJECT_DENIED |
qb.scopeGrantLink | write | Same registration. { kind, instanceId, subjectId, tenantId, authEpoch, outcome: 'linked' | 'unchanged' }, or denied with SCOPE_EPOCH, SCOPE_NOT_FOUND, SCOPE_GRANT_SUBJECT_DENIED or SCOPE_GRANT_LINK_CONFLICT |
qb.roomAccess | read | Registered only when a resource-backed realtime room is on a tenant table (Tenant rooms). { room } when the room's row passes the table firewall for a member of this organization, else denied with ROOM_ACCESS_DENIED |
tenantCommand first reads the organization's tenant_registry row, and wakes
the object only when the registry vouches for it:
| Registry row | Result |
|---|---|
| None: the platform has never served the tenant, or its first report is still in flight | unavailable / TENANT_NOT_PROVISIONED (retryable). The object is not woken, so no empty tenant database is created. |
purged | unavailable / TENANT_PURGED, without waking the object |
Written under another tenancy.jurisdiction | unavailable / TENANT_JURISDICTION_MISMATCH |
| Registry read fails | unavailable / TENANT_UNAVAILABLE |
| Any other state | The object decides (archived, migration and restore are checked there) |
A tenant gets its row when it first serves a request or a job, or when an
operator provisions it (POST /tenants/:id/provision, provisionTenantTarget
in src/lib/tenants.ts: verified against Better Auth's organization table
first). Until one of those, a new organization's commands answer
TENANT_NOT_PROVISIONED. denied is kept for definitive refusals made by a command.
Pass only a server-verified organization id, never one taken from the request.
The HTTP forwarder (tenantForward) does not read the registry: it routes the
authenticated principal's organization, so it adds no D1 read per request.
The object then checks each request in this order:
- The RPC version.
command()accepts only version 4 or later (TENANT_PERMIT_VERSION). ThenTENANT_COMMAND_FINGERPRINT, a hash of the command registry that both Workers carry insrc/tenant/contract.ts. It is checked before the name, so a split deploy reports itself as one. Then the command name and the exact request shape: unknown fields are refused, and string fields are capped at 512 characters. Then the target: the request'stenantIdmust be this object's name. Partition objects and control objects serve no commands. Each failure throws (TENANT_RPC_VERSION_MISMATCH,TENANT_COMMAND_FINGERPRINT_MISMATCH,TENANT_COMMAND_INVALID) before the command reads any tenant data. As with every RPC, waking the object still runs its own migrate on wake first. - The restore permit. The permit also carries the
authorization epoch, and no restore can begin while it is held. A pending
restore answers
unavailable/TENANT_RESTORE_PENDING. - Archived, purged and unmigrated objects answer
unavailable(TENANT_ARCHIVED,TENANT_PURGED,TENANT_MIGRATION_FAILED). - The body runs in one
ctx.storage.transactionSync, with noawait. A write command marks the object dirty inside that transaction, so the next backup includes it, and the backup is scheduled after the transaction commits.
tenantCommand(env, tenantId, request) returns
{ status: 'ok', reply } | { status: 'denied', code } | { status: 'unavailable', code }.
It checks the reply against the command's schema; a reply of the wrong shape
becomes unavailable / TENANT_COMMAND_REPLY_INVALID. Read commands are
retried once after an object reset or a retryable RPC error. Write commands are
sent once, and overloaded objects are never retried. Failures carry codes only:
TENANT_UNAVAILABLE, TENANT_OVERLOADED, TENANT_DEPTH_EXCEEDED,
TENANT_RESTORE_PENDING, the registry codes above, or one of the refusal codes above. A version or
fingerprint mismatch is unavailable, not denied: it is deploy skew, not a
policy decision. For a write command, unavailable means the outcome is
unknown: the write may have committed before the failure. Logs record the
command name and code, never the request or the error text. A malformed (or
null) request, an unknown command, or an organization id that is reserved, contains
: or is longer than 512 characters throws.
Commands do not carry the wake report or the outbox signal that handle()
responses do. A command write that queues outbox entries (lookup, audit,
realtime, jobs) is applied on the object's next drain: the next proxied request
that reports a non-empty outbox, the object's next alarm (which asks the edge to
drain it), syncTenants(env), or drainTenantOutbox.
Placement
The firewall decides where each table lives. A table with an organization arm
anywhere in its firewall is a tenant table: at the top level
(q.scope("organization")), inside all:, inside any:, or optional. So
"org staff OR the attendee of one event" is a tenant table too. An exception
firewall never is. Every other table is a platform table, served by the
edge from the features D1. Every compile prints the resolved table:
[quickback] tenancy: durable-object — table placement
table placement why
attendees tenant organization arm (any:)
notes platform no organization firewall
projects tenant organization firewall
tasks tenant organization firewallTenant ownership
Inside the object, a tenant table's effective firewall is the authored one behind a tenant pin:
organizationId = ctx.tenantId AND (<the firewall you wrote>)ctx.tenantId is the organization the object serves. The object sets it from
its own name on every request and job, overwriting whatever the caller sent;
the edge strips it before forwarding. Three things follow:
- Reads. Every read is pinned to the object's organization, including
skipTenantScopesrelationship probes and the sysadmin escape. A row naming another organization — a bad copy or restore — is invisible to everyone. - Writes. Every insert stamps
organizationIdfromctx.tenantId: CRUD create, changesets, upserts, batch and the actiondb. An attendee with no active organization creates rows in the right tenant. - Who sees what is still the authored firewall.
any: [{ field: 'organizationId', equals: 'ctx.activeOrgId' }, { field: 'eventId', equals: 'ctx.scope.event' }]lowers to tenant AND (org member OR attendee of this event). The org arm means org membership. A relationship or scope caller never getsctx.activeOrgIdfrom its relationship row, so it reads its own event's rows, never the organization's. An org-only table is for members only: a non-member's list or create gets403 ORG_REQUIRED, and a read by id finds nothing. The pin itself never satisfies that check — creates on a members-only table still need the caller's ownctx.activeOrgId, while an org-OR-scope table's creates are bounded by the firewall-on-write check.
ctx.tenantId is compiler-owned: the build adds the pin for you, and a
firewall that names ctx.tenantId itself is a compile error, in either
isolation mode.
A tenant object holds one organization, pinned through one column. A firewall
whose org arms bind two different organization columns (a buyer and a seller
org on one row) fails with TENANCY_ORG_COLUMNS. An any: [org, owner] table
that moves into tenant objects prints a warning: each request reaches one
object, so an owner without an active organization no longer reaches their
rows across organizations.
The pin does not change the edge↔tenant contract, so TENANT_RPC_VERSION
stays the same. The pin is compiled into the tenant Worker's own code, the
tenant object sets ctx.tenantId itself, and the edge strips any forwarded
value. A mixed rollout is therefore safe in both directions: an older edge
cannot supply a tenant, and an older tenant Worker never reads one.
Placement changes on deployed tables
Once a project runs with isolation: 'durable-object', its features D1 holds
only platform tables. So a table that is in the deployed features D1 (the
quickback/drizzle/features snapshot) and that the firewall places in tenant
objects is moving — a shared project switching to 'durable-object', or an
any: / optional organization arm that the old rule placed on the platform.
Without a declared placement it fails the build with
TENANCY_PLACEMENT_MOVED: deploying would leave
its rows in D1 while each tenant object serves an empty table. Declare
placement: 'tenant' once its rows are copied into each organization's object,
or placement: { platform: true, crossTenant: true } to keep it where it is.
quickback build sends the deployed table list as
migration.priorState.existingFeaturesTables (an older CLI does not). A build
with no deployed-schema facts at all — no existingFeaturesTables and no
features history, or a latest snapshot that does not parse — cannot tell, so every table whose placement derives
tenant only from an any: / optional organization arm fails with
TENANCY_PLACEMENT_UNCONFIRMED until it declares its placement (or the build
runs with a current CLI). A project with no migration history at all is
greenfield: the CLI sends an empty list, and a raw-files validate compile
treats missing history the same way.
Override it per table with placement. An organization-scoped table kept on
the platform must say so, because every tenant's rows then share one database:
placement: { platform: true, crossTenant: true },All features sharing a partition kind must use the same key column. All tables in one feature share a placement, because they share one schema file and one route file. To shard a high-volume feature further, see Partitions.
Every compile warns when a tenant table's organizationId or a foreign-key
column has no index: each request filters, and each FK check reads, on it.
Admission tables stay on the platform
Only a tenant-backed scope kind, its
tenant namespaces and its tickets admit inside tenant
objects. Every other admission and request-time authorization path runs on the
edge Worker and reads its tables from the platform D1. If a tenant object held
one of those tables, the D1 copy would be missing. Or, during a migration, it
would be a stale copy whose rows keep admitting subjects the tenant has
already revoked. So the build fails with TENANCY_ADMISSION_DEPENDENCY when
any of them reads a tenant or partition table. A scope kind that reads a
tenant table but can't be tenant-backed fails with the planner's own codes
instead (see Which kinds can be tenant-backed),
and a namespace with TENANCY_NAMESPACE_*:
| Surface | Tables checked | Paths named in the error |
|---|---|---|
authz.scopes.<kind> | each role's relationship tables (from and any hops), each role's self.table, and the instance table | POST /scope/v1/enter (when a role is subject: 'user' | 'both'), ctx.mintScope (when a role is 'supplied' | 'both'), instance-bound OAuth (authorize, code exchange, refresh, issueScopeGrant, bound namespace and ws-ticket checks), the ctx.self recheck on every action route, and the scope refresh / mintScope of each namespace using the kind's relationships |
authz.namespaces.<name> | relationship lanes' tables and the loads table | namespace gate and hydration |
realtime.wsTicket | scopeTable, the tables behind role / access.authzRole, and the namespace lanes | ws-ticket mint |
| A platform feature | every relationship its CRUD, read, view, masking and action access, and its firewall arms, resolve: authz.roles relationship roles, authz.rules, authz.permissions (including arrows from / to) and direct via arms, with hops and loads | request-time authorization |
A platform action with anchor | the relationship's tables and its reach tables | anchored transaction |
A platform action's refs and operator | each ref's table, each ownedVia.junction, and operator.target.table | loaded and firewalled before execute |
| A platform table's derived JSON keys | each derivedFrom.table | CRUD reads and action requires: |
A live view (defineView) | root and every project-table include | the view materializer (GET /views/..., ws-ticket view lane) |
The error names the surface, the paths that depend on it, and each tenant
table with its placement. A scope kind whose instance table is ambiguous also
fails: set authz.scopes.<kind>.instance. A kind with no resolvable instance
table loads no instance, so only its relationship tables are checked, unless
the kind is also a tenancy.lookups key. Then its tokens are routed by an
instance id that no table binds to a tenant, so the build fails until you set
instance.
The table above is what the compiler names in friendly messages. Three more checks stand behind it:
- Platform action source. A platform action or feature
lib/file that names a tenant table fails the build. So does a schedule (services/schedules), a queue handler that is not a tenant job, and anyquickback/libhelper, auth hook (quickback/hooks), middleware (quickback/middlewares) or Better Auth plugin (quickback/plugins) the edge Worker loads (scanned per module). That covers its export name as an identifier, its SQL name as a string (an inlinesqliteTable("projects", …)), andFROM/JOIN/INTO/UPDATE/TABLE <name>in any string or template (in actions, only withallowRawSql). The scan is conservative: a local variable that only shares a tenant table's name is refused too, so rename it. Instead of a cron sweep, declare a tenant schedule (each object runs its own work on its own alarm) or have the edge arm a tenant job (scheduleTenantJob); to read a tenant's data from the edge, call a result job. - The emitted edge bundle. After all files are generated, the compiler
walks the edge Worker's runtime imports from
src/index.ts. It follows relative imports,tsconfig.jsonpathsaliases such as~/,export … from, and literalimport()/require(). Type-only imports are skipped. The build fails if an edge module imports a tenant feature's module (its schema, table, resource or routes), or callsimport()/require()with a computed path the walk can't follow. - At runtime.
src/lib/firewall-registry.tsis the one edge module allowed to import tenant modules. It maps every table to its firewall for the scopeddb. Only its tenant imports are exempt; its other imports are still walked. Its tenant entries throwTENANT_TABLE_ON_EDGEoutside the tenant Worker, so an edge query that slips past the compile-time checks fails instead of reading D1. A raw SQL string that never goes through the scopeddbis caught only by the compile-time scan.
To fix it, keep those tables on the platform. An organization-scoped one needs
placement: { platform: true, crossTenant: true }. A platform table can't
hold a foreign key into a tenant table (TENANCY_FK_PLATFORM_TO_TENANT), so
move the tables it references with it, or drop the reference. Or remove the
dependency.
Tenant-backed scope kinds
An instance-bound OAuth kind (authz.scopes.<kind>.oauth) whose whole
admission program reads one organization object is tenant-backed: its
admission, its namespaces and its tickets run in that object, and its tables
never exist in the platform D1. Every compile decides this from the
requirements in Which kinds can be tenant-backed;
there is no switch. A tenant-backed kind gets this identity contract:
- Pinned grant references. An instance-bound OAuth grant stores
qb2:<kind>:<tenantId>:<instanceId>:<mode>:<epoch>[:<ceiling>]instead of<kind>:<id>.modeisdiscover(the authorize flow: any of the kind's roles) orexact(issueScopeGrant: only the URI-encoded{ roles, subjectId, provingTable }ceiling). A reference parses only in its one canonical spelling. An exact reference withoutprovingTablefails refresh withinvalid_grant(reauthorize); the table is never guessed. An older unpinned grant for the kind fails refresh withinvalid_grant, so the user reauthorizes. The compiler never guesses a tenant for it. - Where the tenant comes from. At authorize it comes from the kind's
lookup (
tenancy.lookups.<kind>), never from the caller's active organization or the request. If the lookup matches several organizations, the request fails withinvalid_request. If it has no row yet, the request fails withtemporarily_unavailable(retry).issueScopeGrantalso accepts atenantHint, the organization that trusted server code read from the credential row. The hint is used only while the lookup has no row. A lookup that disagrees with it refuses. Once issued, a grant always routes to its pinned tenant, even if the lookup changes later. - Authorization epoch. The grant pins the tenant's
authorization epoch, read with the
qb.epochtenant command. Every code exchange and refresh compares it with the current epoch. After a restore they differ, and the grant fails withinvalid_grant, so the user signs in again. - v2 claims. The
qb_scopeclaim gainsv: 2,tenantIdandepoch(andprovingTablefrom admission). The bound lane refuses a token for a tenant-backed kind without them (401).ctx.oauth.boundTobecomes{ kind, id, tenantId, epoch }, and the edge routes the request toboundTo.tenantId. It never uses a lookup oractiveOrgIdfor that token. - Revocation. For a tenant-backed kind,
revokeScopeGrantrequires thetenantIdthe grant pins. It deletes the user's pinned grants for that instance in both modes, at every epoch and under any tenant. Instance ids are collision-resistant, so this also catches a grant pinned through a wrong hint. User-wide revocation (ban, sign-out everywhere) deletes the kind's pinned and unpinned grants. - Kind names.
qb2is reserved as a scope kind name in every project. A tenant-backed kind's name must be an identifier, because it is a segment of its grant references. - Realtime. Rooms on tenant tables become
t1:<tenant>:<key>, and the kind's identity tags becomet1:<tenant>:scope:<kind>:<provingTable>:<subjectId>. See Durable Objects → Tenant rooms. - Id contract. The kind's instance table and each role's proving table
need server-generated, collision-resistant ids across every tenant object.
The build fails with
TENANCY_ADMISSION_ID_STRATEGYwhen the table hasgenerateId: false | 'short' | 'serial',allowClientIds: true, or a numeric primary key.
Which kinds can be tenant-backed
The compiler plans each scope kind's admission program: the instance read plus one probe per user-provable role, in the kind's declared precedence. An OAuth kind can run that program, and every other path it has, inside its tenant object only when all of these hold. The first five are permanent: those paths never run in a tenant object. A kind reading a tenant table that misses any of them fails the build with each missing requirement under its own code:
| Code | Requirement |
|---|---|
TENANCY_LEGACY_SCOPE_ENTER | The kind declares oauth. POST /scope/v1/enter runs on the edge against D1 and pins no tenant, so a kind admitted only through it can't read tenant tables. |
TENANCY_LEGACY_SCOPE_MINT | Every role is subject: 'user'. A 'supplied' / 'both' role can be minted by ctx.mintScope on the edge, with no tenant. |
TENANCY_ADMISSION_SELF | No role declares self: { table }. The ctx.self recheck runs on every edge action route. |
TENANCY_ADMISSION_HOPS | No role proves through a relationship with hops. Tenant admission proves the relationship's own table. |
TENANCY_NAMESPACE_MINT | No namespace on the kind's :param declares mintScope or acceptScope. A legacy scope token carries no tenant. |
TENANCY_ADMISSION_MIXED_PLACEMENT | Every table the kind reads (instance, relationship tables and hops, self.table) lives in the tenant object. None on the platform. |
TENANCY_ADMISSION_PARTITIONED | None of them lives in a partition. |
TENANCY_ADMISSION_UNRESOLVED_INSTANCE | The instance table resolves to exactly one table. Set authz.scopes.<kind>.instance. |
TENANCY_ADMISSION_TENANT_COLUMN | The instance table and every proving table have an organization column. |
TENANCY_ADMISSION_LOOKUP | tenancy.lookups.<kind> indexes the instance table's primary key: { table: '<instance>', column: '<pk>' }. |
TENANCY_ADMISSION_SQL_HELPER | No role proves through a SQL helper (Postgres). |
Every error lists all the violations at once. A kind that meets them all is tenant-backed in every compile.
Tenant admission (qb.admitScope)
For a tenant-backed kind, the edge never reads the kind's tables. Every
admission (authorize, code exchange, refresh, issueScopeGrant, the bound
ws-ticket mint) goes through the generated dispatcher src/lib/scope-admission.ts.
Platform kinds keep their D1 evaluator there. A tenant-backed kind sends one
qb.admitScope command to the pinned tenant. The object
runs admitLocal (src/tenant/scope-admission.ts) in one synchronous SQLite
read, under the restore permit:
- The request's epoch must equal the permit's epoch (
SCOPE_EPOCH). - The instance row, read through its own firewall with tenant scopes stripped
(soft deletes still apply), must exist once and belong to this tenant
(
SCOPE_NOT_FOUND). - Each user-provable role is probed in precedence order, with the same
predicates the D1 evaluator uses. An exact ceiling probes only its own roles
on its own proving table, with
id = subjectId. A row whose organization is not the tenant never matches, even if it was copied in. - The first role with any match chooses the subject
(provingTable, id). Two rows on that role deny the admission (SCOPE_AMBIGUOUS). Every later role probes only the chosen row, so a later role's duplicate rows never deny. The reply carries only that subject's roles and sub-keys, plusprovingTable. It never unions roles across subjects.
The edge checks every reply field against the request: kind, instance,
tenant, epoch, known roles on one proving table, the ceiling's subject, roles
and table, and declared sub-keys only. A reply that disagrees is treated as
unavailable, never as an admission. Errors map like this:
| Outcome | At authorize | At code exchange / refresh | ws-ticket (bound or session) |
|---|---|---|---|
| denied | access_denied | invalid_grant | 403 |
| unavailable (archived, purged, restoring, migrating, unreachable) | temporarily_unavailable (503) | temporarily_unavailable (503) | 503 TENANT_ADMISSION_UNAVAILABLE |
The admission policy (kinds, precedence, predicates, sub-keys) is part of the
command fingerprint, so two Workers built from different policies refuse each
other. Adding qb.admitScope does not change the RPC version: the fingerprint
covers the whole command registry. So while the two Workers deploy, an edge
on the previous build gets unavailable (TENANT_COMMAND_FINGERPRINT_MISMATCH)
from every command, qb.epoch included, until it is deployed too. Bound
tokens of tenant-backed kinds answer temporarily_unavailable in that window.
Nothing is admitted from a mismatched registry. The tenant bundle imports no Better Auth, no drizzle-orm/d1 and no
scope-oauth. The legacy POST /scope/v1/enter never serves a tenant-backed
kind; when no other kind remains, the route isn't emitted at all. None of
these paths reads the platform D1.
Server-issued grants
A tenant-backed kind that declares
oauth.serverGrant
gets two more commands. They are reached only through the generated
readScopeGrantSubject and linkScopeGrantSubject (in src/lib/scope-oauth.ts),
which the edge action calls with the subject its own credential row names;
no edge code reads or writes the guest table itself.
qb.scopeGrantReadselects the instance row and the subject row inside one synchronous transaction: object tenant = instance tenant column = subject tenant column, subject resource column = the instance id, neither row soft-deleted, the subject admittable (the OR of the ceiling roles'where). It writes nothing.qb.scopeGrantLinkre-runs those checks, returnsunchangedwhen the subject is already linked to the user, and otherwise runs one conditionalUPDATE … SET <user column> = :userId WHERE <all of the above> AND (<user column> IS NULL OR <user column> = :replaces) RETURNING id. Anything but exactly one row isSCOPE_GRANT_LINK_CONFLICT(still live, linked to someone else) orSCOPE_GRANT_SUBJECT_DENIED, with nothing written. On a link it puts the audit row (event_kind: 'scope_grant_link', acredential-mediatedactor envelope) in the object's outbox in the same transaction; the edge drains it toaudit_events. The request's epoch must equal the permit's: a restore between the read and the link denies.
Routing follows the kind's lookup; the action's tenantHint (the
organization on its credential row) is used only while the lookup has no
row, and the object's own equality checks make a wrong hint admit nobody.
While the object is held, archived, restoring, moving or migrating, both
commands answer unavailable, so the action answers 503 and its credential
is untouched: nothing spends it before the action's own consume, after
issueScopeGrant. Like qb.admitScope, adding these commands does not
change the RPC version: the command fingerprint covers them (and the
server-grant program), so a split deploy refuses before any write.
Tenant namespaces
A namespace (authz.namespaces.<name>) whose relationship lanes, loads
tables and claimed routes all live in the organization object runs in the
tenant Worker. Claimed routes are the standalone actions whose path is under
its prefix and the generated table routers mounted there. The edge mounts a
forwarder at the prefix. The tenant Worker mounts the namespace's gate file
(src/routes/__namespace-<name>.routes.ts) and, behind it, every route the
namespace claims. A tenant feature's action under the prefix may sit outside
its feature's own path (TENANCY_ROOT_ACTION doesn't apply to it).
The edge routes a namespace request by the prefix's first :param, through
the tenant-backed kind whose requestField it is:
| Caller | Tenant |
|---|---|
| Instance-bound token of the kind | The tenant it pins (ctx.oauth.boundTo.tenantId). No lookup. A token for another instance: 403. |
| Anyone else | tenancy.lookups.<kind> for the id. Several organizations: 409 TENANT_AMBIGUOUS. No row: the same 403 ACCESS_ROLE_REQUIRED the gate gives a caller who proves no lane, so an unknown id and an id you can't reach look alike. A new instance is indexed after its write, so it can answer that 403 briefly; retry. A failed lookup read: 503 TENANT_UNAVAILABLE. |
The edge matches the prefix on the percent-decoded path Hono routes on, and
the bound-token confinement uses the same path, so an encoded variant
(/api/v1/%65vent/…) is routed and confined exactly like the plain one. A
request that reaches the namespace mount but matches no namespace is 404; it
never falls through to active-organization routing. An encoded / (%2F)
stays encoded and is never an instance id. The caller's active organization
never routes a namespace request. A caller
whose active organization is a different one loses its organization seat and
its organization and team roles before the request is forwarded. Inside the
object, an organization firewall arm then matches only a member of this
organization; everyone else gets the rows their relationship proves.
In the object, the gate re-proves an instance-bound token with admitLocal,
the same program qb.admitScope runs. It uses the raw tenant SQLite, the
request's restore permit epoch, and the token's exact subject and proving
table. No RPC and no D1 read. The token's tenant must be the object's, and
its epoch the permit's: a token issued before a restore gets 403, so the
user reauthorizes. A session caller passes a lane only as the subject
admission would choose: after the lane matches, the gate runs admitLocal in
discover mode and requires the same row. So the kind's precedence decides,
and two rows on the choosing role deny (403), exactly as at authorize. Then the relationship lanes, loads hydration (through the
loaded table's firewall) and the per-lane scope refresh run against tenant
SQLite exactly as on a single Worker. A namespace request costs one RPC (the
forward) plus, for a session caller, one lookup read in D1.
A claimed table's legacy forward
(legacy) is not a namespace route. The edge routes its old URL
(/api/v1/locations) by active organization, exactly as it routed the
table before the claim: a caller with no organization gets 400 TENANT_REQUIRED
at the edge, and the organization object is always the caller's own, so the
§2a strip never applies. The forward runs in the tenant Worker
(src/routes/__legacy-<feature>.routes.ts, mounted beside the namespace),
requires ctx.activeOrgId === ctx.tenantId, reads the instance and any row from
tenant SQLite, and dispatches in-process to the namespace route: no second RPC.
A namespace that can't run in one place fails the build:
| Code | Cause |
|---|---|
TENANCY_NAMESPACE_SPLIT | Its lanes, loads tables or claimed routes mix tenant and platform. |
TENANCY_NAMESPACE_PARTITIONED | One of them is in a partition object. |
TENANCY_NAMESPACE_UNROUTED | It runs in tenant objects, but its :param is no scope kind's requestField, or a platform kind's, so a request can't name its organization. |
TENANCY_NAMESPACE_MINT | It declares mintScope or acceptScope for a tenant-backed kind. |
Edge namespace actions
A tenant namespace's routes run in the tenant Worker, which has no env
bindings, secrets or Better Auth. An action under the prefix that needs them
(presigning an R2 upload, creating a Better Auth user, sending email) declares
edge: true. It keeps its URL and runs on the edge Worker:
import { z } from "zod";
import { defineAction, ActionError } from "../.quickback/define-action";
import { callTenantJob } from "../../../lib/tenants";
export default defineAction({
description: "A presigned upload URL for one event.",
path: "/event/:eventId/upload-url",
method: "POST",
edge: true,
input: z.object({ filename: z.string().min(1).max(200) }),
access: { roles: ["scope:event:staff", "scope:event:attendee"] },
async execute({ ctx, env, input }) {
const eventId = ctx.admittedInstance?.kind === "event" ? ctx.admittedInstance.id : undefined;
if (!eventId || !ctx.tenantId) throw new ActionError("No admitted event", "BAD_REQUEST", 400);
// Tenant data only through a result job, in the tenant the gate admitted.
const res = await callTenantJob<{ name: string }>(env, ctx.tenantId, { type: "event.name", eventId }, "event-name:" + crypto.randomUUID());
if (res.status !== "ok") throw new ActionError("Try again shortly", "EVENT_UNAVAILABLE", 503);
return { url: await presign(env.UPLOAD_SIGNING_KEY, `${ctx.tenantId}/${eventId}/${input.filename}`), event: res.result.name };
},
});The edge mounts these actions at the namespace prefix ahead of the forwarder,
each behind a gate on its own route. Every other route under the prefix is
forwarded as before. The gate reads no table. One qb.admitScope in the
instance's tenant decides:
| Caller | Admission |
|---|---|
| Instance-bound token of the kind | Exact admission of its subject in the tenant and epoch it pins. A revoked subject, or a token issued before a restore, gets 403. A token for another instance gets 403. |
| Session | The tenant from tenancy.lookups.<kind>, never the active organization, then discover admission under that tenant's current epoch: the namespace's relationship lanes, chosen by the kind's precedence. |
Session, { roles } lane | A member of that organization holding one of the lane's roles, when the tenant confirms the instance exists there and the user proves no row (SCOPE_NO_ROLE). Members of other organizations never qualify. |
Every denial, including an unknown or not-yet-indexed id, is the namespace's
403 ACCESS_ROLE_REQUIRED. A tenant that can't answer is 503 with
Retry-After, never a denial. On success ctx.tenantId is the admitted
tenant and ctx.admittedInstance is { kind, id } for the admitted instance.
This routing metadata grants no scoped roles. Relationship admission also sets
ctx.scope.<kind> to the admitted entry (id, roles, subjectId,
provingTable); the organization-role lane clears that scope entry. A caller who isn't a member of that organization loses its
organization seat and roles, as on forwarded requests. Then the action's own
access runs against that ctx.
An edge action is a platform action:
- It lives in a platform feature. A feature with no tables is simplest. The
file location doesn't change its URL. In a tenant feature the build fails
with
TENANCY_EDGE_ACTION_PLACEMENT. - Its
dbis the platform D1. It reads tenant data through result jobs called withctx.tenantId. Naming a tenant table, or importing a tenant feature's module, fails the build (TENANCY_ADMISSION_DEPENDENCY). - The namespace's
loadshydration never runs on the edge. Readingctx.<exposeAs>fails the build (TENANCY_EDGE_ACTION_CTX). Usectx.scope.<kind>, or read the row with a result job. - Its path must be under a tenant namespace's prefix
(
TENANCY_EDGE_ACTION_NAMESPACE). Withoutedge: true, a platform action there splits the namespace (TENANCY_NAMESPACE_SPLIT).
Security notes:
- The gate proves the caller holds a role on the instance when the request
starts. A result job runs later, with system rights, in a separate
transaction. A revocation can land between the two. When a write must be
authorized atomically with the authorization, recheck the subject inside the
job (its message carries
ctx.scope.<kind>.subjectId), or keep the write in a tenant action behind the tenant gate. - The job's tenant must be
ctx.tenantId. Never use an id from the request (TENANCY_JOB_CALL_TENANT_SOURCE). Scope any job input by the admitted instance (ctx.scope.<kind>.id). - Each request costs one
qb.admitScopeRPC, plus a lookup read and aqb.epochRPC for a session caller.
In shared mode (no isolation: 'durable-object'), edge has no effect. The
namespace already runs on the edge.
Requests
| Request | Result |
|---|---|
| Tenant route, active organization set | Forwarded to that organization's object |
| Tenant route, no active organization | Resolved through a lookup named after the token's scope kind; otherwise 400 TENANT_REQUIRED. Several matching organizations: 409 TENANT_AMBIGUOUS. |
| Tenant namespace route | A bound token: the tenant it pins. Anyone else: the kind's lookup for the :param (409 TENANT_AMBIGUOUS; an unknown id gets the gate's own 403). Never the active organization. |
| Partitioned route | Forwarded to the partition's object; see Partitions for PARTITION_KEY_REQUIRED / PARTITION_KEY_MISMATCH |
| Tenant route, unauthenticated | 401 AUTH_MISSING |
| Platform route | Served by the edge from D1 |
| Tenant object whose migration failed | 503 TENANT_MIGRATION_FAILED with the migration tag and SQLite error in details.error (never the SQL) |
| Archived tenant | 423 TENANT_ARCHIVED (checked inside the object, no D1 read) |
| Purged tenant | 410 TENANT_PURGED |
Body larger than 1 MiB (Content-Length) | 413 PAYLOAD_TOO_LARGE at the edge |
| Tenant object reset, or a retryable RPC error | GET / HEAD: retried once on a fresh stub. Anything else is sent once. Then 503 TENANT_UNAVAILABLE with Retry-After. The registry is not touched. |
| Tenant object overloaded | 503 TENANT_OVERLOADED with Retry-After. Never retried. |
| Subrequest depth limit exceeded | 508 TENANT_DEPTH_EXCEEDED |
| Any other RPC error | 503 TENANT_UNAVAILABLE; rows without an ordered health report become migrating with the error. An existing ordered health report is preserved. |
Same-tenant unsafe actions
A tenant feature may declare unsafe: { reason, crossTenant: false }. The
action runs in the tenant object like any other, with the same access checks
and security wrapper as on the edge:
unsafeDbis the object's own SQLite, without the firewall. It can't reach the D1, the auth database or another organization: those are not in the object.unsafeDb.scopeTo({ organizationId })accepts only the object's own organization. Any other throwsTENANT_SCOPE_MISMATCH(403).- Cross-tenant
unsafe(unsafe: true, or an object withoutcrossTenant: false) andunsafe.authWritesare refused (TENANCY_UNSAFE_ACTION). Thectxandenvchecks still apply (TENANCY_CTX_UNAVAILABLE,TENANCY_ENV_UNAVAILABLE).
Transactions
Inside a tenant object, db.transaction(cb) is
ctx.storage.transactionSync(). Only statements that run before cb returns
are in the transaction. Every action helper of a tenant feature types db and
unsafeDb as that synchronous handle: define-action.ts, the per-table
define-action.<table>.ts and the namespace define-action.<scope>.ts.
Do all I/O first, then pass a synchronous callback:
const rows = await db.select().from(projects);
await db.transaction((tx) => {
for (const row of rows) {
tx.update(projects).set({ name: "x-" + row.name }).where(eq(projects.id, row.id)).run();
}
});The callback's return type is checked: returning a PromiseLike is a type
error — an async callback, a Promise, a union containing one
(flag ? work(tx) : undefined), or a query builder not yet executed (builders
are thenable; end every statement in .run(), .all(), .get() or
.values()). Returning any is refused too, because it can hide a Promise:
annotate the helper's return type. An awaited body runs after the transaction
has committed, so nothing after the first await would roll back.
The build backs the type for code that casts it away ((db as any)): on a
transaction(...) call on a db or tx handle, an async callback, or one that
returns or starts a Promise (await, Promise.*, .then), fails with
TENANCY_ASYNC_TRANSACTION. Put // quickback-allow-transaction on the call's
line, or the line above, to opt one call out.
Batch handlers on tenant tables are transactional. PATCH /batch and
PUT /batch with failFast: true run in ctx.storage.transaction() and roll
back every write on failure (meta.transactional: true). See
Batch operations. That loop
awaits only tenant SQLite, so a tenant table's create / update / upsert
access and its guards must be declarative: a function there fails the build
(TENANCY_BATCH_ASYNC), because it could await other I/O and let another
request's writes interleave into the transaction.
Hard deletes and the audit log
The tenant Worker has no AUDIT_DB. A hard DELETE (single or batch) on a
tenant table writes its audit_events row to the object's outbox — in the
object's own SQLite, in one transaction with the delete, so a failed delete
leaves no audit row — and the response tells the edge to drain it. The edge
sets each row's target_org_id and actor_org_id to the object's organization
(tenant code cannot attribute a row to another one; a mismatch is logged and
kept in metadata.tenantClaimed), writes the rows to the audit database
(idempotent by id), then acknowledges them. Unacknowledged rows are retried
after the object's next response. A malformed audit payload moves to the
object's __qb_outbox_dead table instead of blocking the queue, and
tenant_registry.outbox_dead_letters counts them. Database failures leave rows
queued for retry; error messages never classify them as permanently invalid.
Purge first drains pending rows and copies dead letters into the audit database.
If either step fails, or pending rows remain, purge refuses to erase the object.
Purge is exclusive. It takes the control object's maintenance fence, which is
refused (TENANT_BUSY) while any request, job or alarm holds a permit,
including one waiting on I/O, so nothing writes into the object after it is
erased. deleteAll() erases any marker the object keeps itself, so purge then
turns that fence into a permanent purged marker in the object's control
object (__qb_control__:<name>, outside the tenant's storage). Only then does
it call deleteAlarm() and deleteAll(), and mark its own storage purged.
A reset anywhere in between leaves a tenant that is still purged, never a live
empty one:
- Every request, command, job, alarm and schedule takes a permit from the
control object, and the permit carries the marker. The object answers
410 TENANT_PURGED, and commands answerunavailable/TENANT_PURGED, even while its own storage still saysarchived. - On wake, an archived object asks its control object, and so does any object
that has not yet had one successful check. If the marker is there, the object
does not migrate: it marks itself
purgedand reportstenant purged. If the control object cannot be read, the object migrates as usual and asks again on its next wake. It still serves nothing, because its permits carry the marker. - Archive, unarchive, restore, cancel-restore, retry-migration, backups, scheduling and lookup rebuilds all refuse a purged tenant.
tenant_registry still says archived after an interrupted purge. To recover,
retry POST /tenants/:id/purge. Purge is resumable: once the marker exists it
needs no fence, erases whatever is left, and marks the registry purged. The
retry keeps last_purge_bookmark from the first attempt and never records a
bookmark taken after the erase.
The marker is a control-object operation, not just a flag. Workers built before it already refuse every permit and restore while an operation is set, so a mixed deploy fails closed on it. Limits:
- During a gradual deploy, a control object still on the previous Worker cannot record the marker. Purge then releases its fence and fails before it erases anything; retry it once the deploy finishes.
- Do not purge while a tenant Worker rollout is in progress. An edge Worker on the previous build reaching a tenant object on the previous build runs the old, unmarked purge.
- Rolling the tenant Worker back to a build from before the purge marker
un-protects interrupted purges. The marker still refuses permits there, but
that build's
POST /tenants/:id/cancel-restoretreats it as an uncertain restore and can clear it.
Migrations
Tenant tables have their own migrations in quickback/drizzle/tenant/. The
build bundles them into quickback/drizzle/tenant/migrations.ts, and each
object applies pending migrations when it wakes:
- Inside
blockConcurrencyWhile, so no request runs against an old schema. - All pending migrations in one transaction, recorded in
__qb_tenant_migrations. - A point-in-time-recovery bookmark is taken first and reported to
tenant_registry.last_pre_migration_bookmark. - A failure never throws from the constructor. The object records it and
answers
503 TENANT_MIGRATION_FAILEDwith the real SQLite error (0001_x: UNIQUE constraint failed: …). The failing statement goes to the logs andtenant_registry.last_error, never to the client. Other tenants keep serving. - Before migrating, the object records which migrations it is attempting
and flushes that record (
storage.sync()). A wake that finds the same attempt unfinished was preceded by a reset mid-migration — the 30 sblockConcurrencyWhilelimit, but also eviction, a deploy or a failover — so it retries. After 3 resets of the same pending set the object marks itself failed instead of retrying on every request. A different migration set starts a fresh count;POST <api>/tenants/:id/retry-migrationclears the count and migrates again.
SQL triggers on tenant tables — your sql: triggers
and the compiler's own (the FK match checks) — install in the same wake, after
the migrations, whenever their content hash changes or a migration ran (a table
recreate drops its triggers). They never reach the features D1. A dump restore
drops every table with its triggers, loads the rows with no trigger firing, then
runs this same wake, so the triggers come back; a point-in-time restore rewinds
the triggers together with the data.
Keep wake work well under 30 s. The object never sets an alarm while recovering, so a failing wake cannot reset the platform's alarm retry budget.
Restoring the bookmark discards writes made after it. It recovers a failed migration; it is not a rollback.
The build refuses drizzle-kit table-recreate migrations on tenant tables
(TENANCY_UNSAFE_RECREATE) when the table is referenced by another table, or
when the copy targets a NOT NULL column that was nullable or has no DEFAULT.
Inside the migration transaction PRAGMA foreign_keys=OFF is a no-op, so the
recreate's DROP TABLE cascade-deletes child rows in every tenant. No
migration is written. Make the change additive (a new nullable column or a new
table, backfilled by a data migration) and rebuild.
Keep tenant schemas additive across one release: during a rollout, objects still on the previous Worker version read the same tables.
Backups and restore
With tenancy.backup set, every tenant object backs itself up to R2 on its own
alarm. A write marks the object dirty and schedules the next backup every
later; an object nobody writes to runs no alarm and takes no backup. With
every: 'off' no dump is ever scheduled (a running one finishes and pending
ones are dropped on the next wake): point-in-time recovery (30 days) is then
the routine backup, and the bucket serves only operator backups, dump restores
and moving an organization between objects. A purge deletes the purged object's
dumps (the receipt-fenced one it was admitted with included): no copy outlives
the object except point-in-time recovery.
A backup is one NDJSON object at <objectId>/<startedAt>-<uuid>.ndjson: a header (the
object, its applied migrations, its tables), one line per row
({"t": table, "r": row}, BLOBs as {"$b64": …}), and a footer with the row
count and the rows per table ({"qb": "end", "rows": n, "counts": { table: n }};
every table in the header appears, empty ones as 0). A dump restore checks
them before it touches anything: per-table counts that disagree with the rows
(either way) or do not add up to rows refuse the dump. Dumps written
before this footer gained counts have only rows. It streams in steps: each alarm reads tables in rowid order, uploads
R2 multipart parts of exactly 5 MiB (only the last part is smaller), and keeps
the leftover bytes and the (table, rowid) cursor in object-local storage, so
the next step resumes where the last stopped. When it finishes, the edge
records tenant_registry.last_backup_at / last_backup_key from the object's
outbox, and dumps older than retainDays are deleted.
An active tenant’s backup is not a point-in-time snapshot: rows written while it runs may or may not be in it. For an exact copy, restore a point-in-time bookmark.
| Operator route | Effect |
|---|---|
POST /tenants/:id/backup | Take a full backup now; returns { key, rows, counts, bytes, parts, startedAt, finishedAt, generation, archived } |
GET /tenants/:id/backups | List the object's dumps |
GET /tenants/:id/backups/counts?key=<key> | One dump's footer, { key, rows, counts }, from a single bounded read of the dump's last 256 KiB (the dump is never loaded). counts is null for a dump written before footers carried it. A key of another object is refused (400 TENANT_BACKUP_READ_INVALID), a missing dump is 404 TENANT_BACKUP_NOT_FOUND. A tenant Worker older than this route answers TENANT_RPC_VERSION_MISMATCH: deploy the tenant Worker first. Also tenantBackupCounts(env, id, key) in src/lib/tenants.ts. |
GET /tenants/:id/bookmark | The object's exact current point-in-time bookmark, { bookmark, observedAt }. Record it before a risky change; restoring { "bookmark": "…" } returns to exactly this point (never restore { at: observedAt }, which is approximate). Refused for a purged object, or one whose purge is under way (its last bookmark is tenant_registry.last_purge_bookmark). |
GET /tenants/:id/outbox-receipts | Delivery receipts: ?since=&limit=&outcome= |
POST /tenants/:id/schedules/arm { "types"?: [...] } | Re-arm the organization's tenant schedules (all, or the given job types); each runs after its debounce unless due sooner. Registered organization objects only. Returns { armed }. POST /tenants/schedules/arm-all does every registered organization (bounded concurrency) — a manual one-off after a deploy, never a cron. |
POST /tenants/lookups/migrate { "limit"?: n } | With a hashed lookup: convert one page of pre-hash rows across objects. Returns { rekeyed, remaining }; repeat until remaining is false. |
POST /tenants/:id/rebuild-lookups | Delete the object's platform lookup rows and re-publish every current key from its rows, under the primary TENANT_LOOKUP_KEY (after a rotation). Its keys may not resolve until the drain finishes. Returns { applied }. |
POST /tenants/:id/restore { "backupKey": "…" } | Archived objects only. Restore a dump: drop every tenant table, re-create the schema at the dump's migrations, load its rows (one transaction), then apply the migrations added since. An old dump comes forward to the current schema. Refuses a dump from another object, a truncated dump, or one taken at migrations this Worker does not know. Returns { kind: "dump", backupKey, rows, pending: true }; the object restarts and stays archived. Pending scheduled jobs are dropped (see restore quarantine). |
POST /tenants/:id/restore { "bookmark": "…" } or { "at": <ms or ISO> } | Archived objects only. Point-in-time recovery (30 days): onNextSessionRestoreBookmark, then the object restarts. Writes after the bookmark are discarded. Returns { kind: "pitr", bookmark, undoBookmark, pending: true }. A bookmark is exact. at is resolved with getBookmarkForTime, which Cloudflare documents as approximate: on a real deployment, writes made a few seconds before at were missing after the restore. For a precise target use GET /tenants/:id/bookmark beforehand, or the undoBookmark of an earlier restore. |
An operator backup on an archived organization acquires exclusive maintenance
admission before scanning. It refuses with TENANT_BUSY while older requests,
jobs or alarms still hold permits, abandons an unfinished active scan, and
starts a fresh one while blocking unarchive, restore and purge. Its completed
receipt has archived: true and the current generation. Active and scheduled
backups have archived: false and retain the incremental behavior above.
Reading a completed backup in operator application code
readTenantBackup from src/lib/tenants.ts transports the latest completed,
quiescent backup through the tenant Worker. The backup bucket remains bound
only to that Worker. No download HTTP route, arbitrary SQL, archived tenant-job
admission or edge bucket access is added. Call this helper only after your
application verifies the operator and target organization; it does not perform
HTTP authentication itself, like the other operator helpers.
After archiveTenant and backupTenant, pass only the receipt's key,
generation and bytes to
readTenantBackup(env, organizationId, { key, generation, bytes }).
The result is { key, generation, bytes, etag, data }, with data an
ArrayBuffer of the original encoded NDJSON. Use new TextDecoder().decode(data)
only after the helper succeeds, then validate and project the application data
you intend to retain. Keep raw dumps private; a dump contains all tenant tables.
The hard whole-buffer limit is 8 MiB of encoded bytes. Optional maxBytes
may lower that limit, never raise it. A larger dump fails with
TENANT_BACKUP_READ_TOO_LARGE; it is not truncated. Do not purge a tenant when
retention fails. Larger exports need a future bounded streaming protocol.
The helper requires the exact latest completed receipt, an archived organization, a matching current generation, and no writes since that backup. Old receipts, active backups, foreign keys, partition/control objects, changed byte lengths and malformed replies refuse. Legacy receipts without the new fields require a fresh archived backup. The tenant checks R2 metadata and the actual stream length, then returns binary bytes rather than base64 or a result-job JSON payload. Exclusive maintenance remains held through the body read: pending restore or purge refuses the read, and restore, purge or unarchive cannot begin during it. The edge checks the registry before and after the RPC; application workflows must still recheck their maintenance fingerprint before a later destructive operation.
This helper needs tenant RPC version 7. Deploy the tenant Worker before the edge Worker; an older tenant Worker cannot supply the new operation. A failed or ambiguous maintenance release keeps the existing recovery fence, requiring operator reconciliation rather than silently reopening admission.
The same operations are backupTenant, listTenantBackups and restoreTenant
in src/lib/tenants.ts. Dump restore has an 8 MiB encoded NDJSON limit
(413 TENANT_RESTORE_TOO_LARGE), checked before reading the body or changing
any table. Larger dumps need PITR until streaming restore is implemented.
Restore cancels any in-progress multipart upload, clears its cursor and carry,
and schedules a fresh backup. Both restore kinds and migration retry acquire exclusive
admission before reading R2 or changing data. They refuse with TENANT_BUSY
while previously admitted HTTP handlers, deferred waitUntil work (including
nonblocking after-commit hooks and effects.enqueue callbacks), jobs, alarms or
backup work remain in flight. HTTP responses return before nonblocking effects
settle; their admission permits remain held through nested deferred work and
release after success or rejection. Retry after that work finishes. New work is quarantined while the
operation runs; a known success or failure releases its matching gate. An
ambiguous operation or failed release retains quarantine for explicit operator
recovery. Backup and restore operations are serialized.
Use effects.enqueue, nonblocking afterCommit hooks or
c.executionCtx.waitUntil(...) for deferred tenant work. Authored access to the
raw waitUntil export of cloudflare:workers is refused with
TENANCY_UNTRACKED_BACKGROUND in tenant modules and their statically named
relative authored dependencies, including aliases and helper re-exports that
bypass the request's tracker. This includes a dynamic import's ordinary
fulfillment callback reading the namespace through implicit arguments[0],
even when its declared parameter is unused or the access is in a nested arrow.
Uninspectable argument or namespace escapes fail closed. Unrelated Cloudflare
exports remain available when their access is explicit. External packages, path
aliases and computed module names are outside this source-level check. Keep
unrelated Cloudflare imports; use the
tracked context instead of importing waitUntil. Await database work that must
finish before the response. Arbitrary promises or timers launched without
await or a tracked scheduler are outside this lifetime contract; do not retain
a tenant database handle in them.
A failed upload keeps its checkpoint and retries; writes during a backup
schedule another backup after it completes.
Restore quarantine
Every restore, dump or point-in-time, follows the same steps:
POST /tenants/:id/archive. A restore of an object that is not archived is refused (tenancy: archive before restore), as is a restore while an earlier one is pending (TENANT_RESTORE_PENDING).POST /tenants/:id/restore. It returnspending: true; the object restarts.POST /tenants/:id/syncreconciles the restore under a new outbox generation. The object is still archived.- Inspect it, then
POST /tenants/:id/unarchive. Nothing serves a restored object before this explicit step. Unarchive refuses (TENANT_RESTORE_PENDING) until step 3 has reconciled the restore, or while an unconfirmed restore or stranded maintenance fence is open — clear those withPOST /tenants/:id/cancel-restore.
A dump carries no schedule: dump restore drops the object's pending scheduled
jobs (scheduleTenantJob), its job-run dedupe records and its queued non-audit
outbox entries in the same transaction as the load, so nothing scheduled
against the replaced rows fires against the restored ones. Reschedule
recurring work after unarchive. PITR instead rewinds the schedule to the
bookmark with the rest of the database.
Restore first acquires an external admission
gate: a small control object in the existing tenant namespace, under a reserved
__qb_control__: name. The gate is outside the database being rewound. It
refuses restore while requests or outbound jobs are in flight, then quarantines
handlers, jobs and alarms until reconciliation. Control objects use the class's
implicit ctx.exports namespace; no extra namespace or binding is provisioned.
A tenant object does not ask its control object per request: it keeps the
permit it was last granted as a lease and admits its own requests, jobs and
alarms under it with no control RPC. The lease is an ordinary permit, so the
gate still refuses restore while it is held. Restore, purge, migration retry
and archived backups first revoke it: refused (TENANT_BUSY) while any work
admitted under it (deferred callbacks included) is still running; otherwise
the lease is released, and none is granted again until the fence is released.
A lease admits work for at most 30 seconds; the next admission after that
re-validates with the control object. An idle lease stays registered until it
is revoked or replaced, so a control object reports TENANT_BUSY to any
caller that skips the object's own revocation. A restart or eviction leaves
its permit behind; the object's next admission, or the next restore's prune,
removes it. Cross-object work (an outbox job from another
object) still takes and releases the source object's permit per delivery.
Restore fences the old outbox generation and removes platform lookup rows.
After a confirmed restore restarts the object, call POST /tenants/:id/sync to
rebuild lookups under the new generation. Reconciliation discards the queued
outbox entries other than audit rows and scope revocations, because they
describe writes the restore replaced (a dump restore already dropped them with
the load). A queued revocation is kept: rolling data back never un-revokes a
claim. A dump restore is confirmed once its load commits; a dump that is
refused (too large, truncated, another object's) changes nothing and releases
quarantine like a rejected PITR prepare. TENANT_RESTORE_PENDING means the
operation is unconfirmed or the old session has not restarted. A rejected
prepare releases quarantine and restores its old lookup index. An unrelated
restart never turns an unconfirmed prepare into a successful restore.
For an uncertain operation, inspect the current state and explicitly call
POST /tenants/:id/cancel-restore with { "confirm": "<objectId>" }. If the
old session is still running, this first restarts it and returns pending; repeat
the explicit recovery call after restart. Recovery keeps the current database
and queued work, rebuilds lookups, and stays archived. A failed pre-invocation
rollback resumes its recorded cleanup instead. Recovery owns one persisted
mode so delayed cleanup cannot race abandonment. Once abandonment is claimed a
restore can no longer be armed, but one that took effect anyway (a PITR
bookmark is irreversible once scheduled; a dump load committed) is recorded in
the control object: abandonment then discards queued non-audit outbox entries
like a confirmed restore and returns applied ({ bookmark, undoBookmark } or
{ backupKey, rows }; null when nothing applied). A restore whose arming RPC
was lost before it reached the control object cannot be recorded; such an
abandonment keeps queued work. A completed restore also stays archived;
inspect it before POST /tenants/:id/unarchive. Never automatically retry the
restore write.
Authorization epoch
The control object also holds an authorization epoch, a counter that no
restore can rewind. It is per object name: an organization object and each of
its partition objects have their own. Every restore, dump or PITR, advances it
when the restore begins, and keeps it across success, rollback and abandonment.
Read it with tenantAuthorizationEpoch(env, objectId) in src/lib/tenants.ts
(or the object's authorizationEpoch RPC), or with the
qb.epoch command, which reads it under the restore permit.
The epoch is exposed for admission checks; nothing enforces it yet. Enforcement
arrives with tenant admission (plan 032): credentials pinned to an earlier
epoch will then be refused, so a restore forces re-login even when restored
rows would grant the old access again. Restore does not call /oauth2/revoke.
Point-in-time recovery does not run under local wrangler dev. The staging
account drill remains an open release gate: write A, note time T, write B,
archive, restore to T, sync after restart, and verify A remains, B is gone,
lookup entries match the restored rows, and scheduled work resumes safely.
Do not treat the local reconciliation simulation as a completed PITR drill. The R2 dump/restore drill runs locally in the
tenant-do output-matrix fixture.
Background work
Per-tenant alarms replace global "every minute, scan every organization" sweeps. Each object has one alarm, shared by backups and tenant jobs, driven by a schedule table: a due row is moved to its retry time before the work runs, and a finished occurrence is recorded before its row is removed, so a crash retries the same occurrence up to five attempts. Handlers must make external side effects idempotent. Canceling or replacing a due occurrence prevents its stale snapshot from running; rescheduling at the same time creates a new occurrence. Unarchiving re-arms suspended work.
A tenant job is a services/queues handler that runs inside a tenant
object. Mark it tenant: true (implied when it imports an unpartitioned tenant
feature). A handler may not mix platform and tenant imports; a feature with no
tables (helpers only) is neither, so either kind may import it. Partitioned
feature imports are refused until explicit partition queue routing is supported.
Inside a tenant job, tenant on the execute context supplies scheduling,
cancellation and outbox delivery; it is absent for platform handlers.
import { eq } from "drizzle-orm";
import { defineQueue } from "@quickback/compiler";
import { reminders } from "../../features/reminders/schema";
export default defineQueue<{ type: "reminder.due"; tenantId: string; reminderId: string }>({
name: "reminderDue",
messageType: "reminder.due",
tenant: true,
execute: async ({ message, db }) => {
db.update(reminders).set({ sentAt: new Date().toISOString() }).where(eq(reminders.id, message.reminderId)).run();
return { success: true };
},
});The same handler runs three ways:
| Trigger | How |
|---|---|
| Queue | A producer sends { type, tenantId, … }; the edge consumer routes it to that tenant's object (runJob). A message literal for a tenant job without tenantId fails the build (TENANCY_QUEUE_TENANT_ID); at runtime one without it is retried into the dead-letter queue. |
| Alarm | tenantRuntime(env).schedule(key, at, message) from a tenant action or job, or scheduleTenantJob(env, tenantId, key, at, message) on the edge. One pending occurrence per key; a job reschedules its own key for the next due item. |
| Outbox | tenantRuntime(env).send(target, message) — see Partitions. |
| Platform queue | A tenant action or job calls tenantRuntime(env).enqueue(binding, message); the edge sends it to a platform queue and that queue's consumer routes it — see Platform queues. |
Tenant actions import tenantRuntime from their define-action helper:
await tenantRuntime(env).schedule("reminder:" + row.id, Date.parse(row.remindAt), { type: "reminder.due", reminderId: row.id });schedule, send and the edge helpers stamp tenantId. Each run gets a
runId, stable per occurrence or message: use it as the idempotency key for
external side effects (email, SMS, payments).
Concurrent deliveries of one runId share an in-flight execution in the live
object. Completion is recorded only on success. A crash after an external effect
but before durable completion can repeat that effect: delivery remains
at-least-once, so handlers must use that idempotency key externally.
Tenant jobs honor ack() and retry() for queues, alarms and outbox delivery.
retry() is sticky and wins over a later ack() or a successful return; no
completion marker is recorded for that attempt.
After an alarm, the object asks its edge to drain what the alarm left in the
outbox (queue sends, realtime, lookups) through the edge's
QuickbackTenantOutbox entrypoint — the tenant Worker binds it as
__QB_EDGE_OUTBOX (generated in wrangler.tenant.jsonc, each target its own
edge). While entries remain (the edge unreachable, entries backing off), an
internal alarm asks again — after 30 s, doubling to an hour while a drain
makes no progress — so delivery never waits for the tenant's next request and
no cron sweeps the objects. Without the binding it does not poll: the next
request or operator sync drains. A named entrypoint is reachable only through
a service binding, never over HTTP; tenant app code does not see the binding,
and the edge drains only registered objects (an unknown name is refused, never
woken). Keys starting qb: (qb:backup, qb:outbox-flush,
qb:schedule:<type>) belong to the runtime: schedule / cancel and the edge
helpers refuse them (TENANT_JOB_INVALID).
An alarm that throws retries the same occurrence with backoff (10 s, doubling,
capped at an hour). After five attempts it is a dead letter: a failed
delivery receipt of kind alarm (target the key,
method the job type, reason TENANT_ALARM_DEAD_LETTER: … with the last
error). A one-off occurrence is then dropped; a tenant
schedule runs again an hour later, and writes to its
watched tables do not bring that retry forward.
Tenant schedules
A tenant schedule is recurring per-tenant work with no cron and no global
scan: a tenant job with schedule:, run on each organization's own alarm. It
runs when it is due and returns next — when it is due again (ms or a
Date), or null to sleep. An organization with nothing due never wakes.
import { and, eq, isNull, lte, min } from "drizzle-orm";
import { defineQueue } from "@quickback/compiler";
import { reminders } from "../../features/reminders/schema";
export default defineQueue<{ type: "reminders.tick"; tenantId: string }>({
name: "remindersTick",
messageType: "reminders.tick",
tenant: true,
// Writes to these tables (any request, job or alarm) re-arm it.
schedule: { watch: ["reminders"], debounceSeconds: 5 },
execute: async ({ db, tenant, runId, schedule }) => {
const now = new Date().toISOString();
for (const r of await db.select().from(reminders).where(and(isNull(reminders.sentAt), lte(reminders.dueAt, now)))) {
db.update(reminders).set({ sentAt: now }).where(eq(reminders.id, r.id)).run();
tenant!.enqueue("EMAIL_QUEUE", { type: "reminder.send", reminderId: r.id });
}
const [next] = await db.select({ at: min(reminders.dueAt) }).from(reminders).where(isNull(reminders.sentAt));
return { success: true, next: next?.at ? Date.parse(next.at) : null };
},
});- Re-arming. SQL triggers on the
watched tables (tenant tables of the organization object, by export or SQL name) mark the schedule in the write's own transaction, so every write path counts — actions, jobs, bulk statements. After each request, job or alarm the object re-arms marked schedulesdebounceSecondslater (default 5, 0–3600), unless one is already due sooner or is backing off after a failure: a re-arm never postpones a run or resets a retry budget. The schedule's own writes never mark it (no self-re-arm loop). A request (its deferred work included), command or job that overlapped the run while a watched table was written (an unmuted per-schedule write counter) brings the next run forward to the debounce — at most one extra run per overlapping operation, since the counter also sees the run's own writes. A watched name that resolves to no table is skipped and logged (tenant.schedule.table_unresolved). - From the edge.
armTenantSchedules(env, tenantId, types?)re-arms an organization's schedules (all, or the given job types) for a change its writes cannot see — an edge-side row, a deploy. A deploy that adds or changes a schedule marks it on each organization's next wake; operators can arm everyone at once withPOST /tenants/schedules/arm-all(a manual, bounded one-off — never a cron) or one organization withPOST /tenants/:id/schedules/arm. An explicit arm also cuts a dead-letter cooldown short (after deploying a fix); a write does not. - At least once. A run is an alarm occurrence:
runIdis stable for it, a throw retries it with backoff and a dead letter is recorded as above; the schedule itself never stops. Completion is recorded only after the next occurrence is written; an occurrence found already complete is evaluated again after the debounce, never dropped. - Restores. A point-in-time restore rewinds the schedule rows, the marks
and the data together, so they stay consistent; the platform alarm is re-armed
from the rows when the object serves again, and every schedule is marked for
one fresh evaluation. Work due between the restore target and the restore
may have run on the abandoned timeline and runs again:
schedule.restoredAt(the last restore, ms, or null) on the execute context lets a handler skip external side effects for it. A dump restore keeps the alarm rows (they are not tenant tables) and marks every schedule the same way. - Placement. Schedules run in organization objects only.
schedule:needstenant: true, is refused on a result job, on aqb:-prefixed type, and in shared tenancy (TENANCY_SCHEDULE); it must betrueor a literal object.
Platform queues
Tenant code has no queue bindings (TENANCY_ENV_UNAVAILABLE). To hand work to
the edge — email, push, webhooks — it sends a message to one of the edge
Worker's queue bindings through the object's outbox:
import { defineAction, tenantRuntime } from "../.quickback/define-action";
// …
tenantRuntime(env).enqueue("COMMS_QUEUE", { type: "mail.send", guestId: row.id });- Same write. The call writes an outbox row with the request's writes. The
edge drains the outbox after the response and calls
env.COMMS_QUEUE.send({ ...message, tenantId, dedupeKey }). The row is written at the call: it is not dropped ifexecutelater throws or returns a dry run. To send only after a successfulexecute, useeffects.enqueue(binding, message). tenantIdcomes from the object. The edge stamps the tenant from the object's name. A message that setstenantIdordedupeKeyitself throws (TENANT_QUEUE_MESSAGE); nothing is written.- At least once. A failed send keeps the entry queued and it is retried
with backoff (outbox delivery), so a consumer can see a
message twice.
dedupeKey(a UUID fixed when the row is written) is the same on every delivery: use it as the consumer's idempotency key. - Receipts.
enqueue(binding, message, { ref })recordsref(1–200 characters) on the row's delivery receipt. Point it at your own per-send row, such as acomms_logid, instead of copying that row into the receipt. The receipt also carries thededupeKey, so a consumer that storesdedupeKeyon its row links the two the other way. - Size. A message over 120 KiB of JSON, or one that is not
JSON-serializable, throws at the call (
TENANT_QUEUE_MESSAGE_TOO_LARGE,TENANT_QUEUE_MESSAGE). A row the edge can never send (a binding missing from the edge Worker, a message the queue refuses) is dead-lettered so it does not block the outbox. - Declared bindings only. The binding must be one of the edge Worker's
queues (the primary queue and
additionalQueues). The build refuses any other name, or a name it cannot read (not a string literal), in tenant actions, tenant jobs, JS trigger handlers and the helpers they import (TENANCY_QUEUE_BINDING); at runtime the object refuses it (TENANT_QUEUE_UNKNOWN), and the drain dead-letters a row naming an unknown binding. - A tenant job uses
tenant.enqueue(binding, message), the same call.effects.enqueue(binding, message)and table-level transitiononEnter.enqueuesteps in tenant features route through it too, afterexecutesucceeds and before the response. A refusal there (TENANT_QUEUE_*) fails the response.
The consumer is a platform queue handler: it runs on the edge, with the
platform D1. If it needs the tenant's data, it calls a
result job with message.tenantId. That value is
authoritative only for messages from a tenant outbox (the edge stamped it). A
queue that edge code also produces to carries whatever that producer wrote:
edge producers must set tenantId from the caller's ctx
(ctx.activeOrgId), never from request input. A restore discards queued queue entries with the other outbox
entries that are neither audit rows nor scope revocations.
Result jobs
Edge code sometimes needs an answer from a tenant's database: a public page
showing an organization's data, or a platform action that reads tenant state.
Edge code can't query tenant tables (TENANCY_ADMISSION_DEPENDENCY), and
cross-tenant unsafe or root-path actions can't move into a tenant feature
(TENANCY_UNSAFE_ACTION, TENANCY_ROOT_ACTION). They stay on the edge, in a
platform feature, and call a result job: a tenant job declared with
result: true and a resultSchema.
import { z } from "zod";
import { eq } from "drizzle-orm";
import { defineQueue } from "@quickback/compiler";
import { reminders } from "../../features/reminders/schema";
export default defineQueue<{ type: "reminder.count"; tenantId: string; ownerId: string }>({
name: "reminderCount",
messageType: "reminder.count",
tenant: true,
result: true,
resultSchema: z.object({ open: z.number().int() }),
execute: async ({ message, db }) => {
const rows = await db.select().from(reminders).where(eq(reminders.ownerId, message.ownerId));
return { success: true, result: { open: rows.filter((r) => !r.sentAt).length } };
},
});import { callTenantJob, resolveTenant } from "../../../lib/tenants";
const tenantId = await resolveTenant(env, "inviteCode", input.code); // or ctx.activeOrgId
if (!tenantId) throw new ActionError("Unknown code", "NOT_FOUND", 404);
// The edge action is the only authorization gate: check here that this caller may ask this question.
const runId = "reminder-count:" + crypto.randomUUID(); // built on the server, never from the request
const res = await callTenantJob<{ open: number }>(env, tenantId, { type: "reminder.count", ownerId: input.ownerId }, runId);
if (res.status === "unavailable") throw new ActionError("Try again shortly", res.code, 503);
if (res.status === "failed") throw new ActionError("Lookup failed", "LOOKUP_FAILED", 422); // res.error stays server-side
return res.result;callTenantJob(env, tenantId, message, runId) returns
{ status: 'ok', result } | { status: 'failed', error } | { status: 'unavailable', code }:
| Result | When |
|---|---|
ok | The handler returned { success: true, result }. The object validated result against resultSchema and checked that it serializes to at most 256 KiB, then recorded it with the run marker in one transaction. |
failed | The handler returned { success: false, error } (that error, capped at 200 characters), threw (TENANT_JOB_FAILED; the error text is never returned, only logged as the job type and code), called retry(), or its result failed the schema (TENANT_JOB_RESULT_INVALID) or the size cap (TENANT_JOB_RESULT_TOO_LARGE). Also TENANT_JOB_RUN_CONFLICT, below. Nothing is recorded, so the same runId runs again. Writes the handler made are kept: key them by runId. |
unavailable | The registry gate shared with tenant commands (TENANT_NOT_PROVISIONED, TENANT_PURGED, TENANT_JURISDICTION_MISMATCH, TENANT_UNAVAILABLE), an archived, purged or unmigrated object (TENANT_ARCHIVED, TENANT_PURGED, TENANT_MIGRATION_FAILED), a pending restore, an RPC failure (TENANT_OVERLOADED, …), version skew, or a malformed reply (TENANT_JOB_REPLY_INVALID). The job may or may not have run. |
resultSchema must be z.object({ … }), which strips unknown keys, or a
strict object (z.strictObject, .strict()), which refuses them, written
inline or as a top-level const. A passthrough, loose or catchall schema fails
the build (TENANCY_JOB_RESULT_SCHEMA).
A repeated runId with the same message replays the recorded result without
running the handler again, for seven days; after that the runId runs again.
resultTtlSeconds (a number literal, 60 to 604800) shortens that for one job:
a result older than that is gone (with its run marker) the next time its
runId is called, and expired results are deleted whenever a result is
recorded or the object's alarm runs. Use it for read-only pulls, such as an
export that reads pages of the tenant's data, whose results should not sit in
the tenant's SQLite for a week. The 60-second floor keeps callTenantJob's
own resend after an object reset (same runId) a replay rather than a second
run. Anything else (a non-literal, out of range, or without result: true)
fails the build (TENANCY_JOB_RESULT_TTL).
The result is bound to a hash of the canonical message (sorted keys): the same
runId with any other field is failed / TENANT_JOB_RUN_CONFLICT, never
the earlier caller's result. Calls have their own run-id space (call: on the
object; runJob and deliverTenantJob refuse run ids with that prefix), so a
call never replays a queue, alarm or outbox run. Concurrent calls with one
runId and one message share one execution. Build the runId on the server
(a fresh id, or one from your own idempotency record), never from request
input. The call is retried once after an object reset or a retryable RPC
error, and drains the object's outbox after an ok.
Authorization. A result job runs with system rights in the tenant's object:
no caller, no firewall, no access rules. The edge code that calls it is the
only authorization gate, so check there that this caller may ask this
question. The build requires the tenant argument to be visibly server-verified
(TENANCY_JOB_CALL_TENANT_SOURCE): ctx.activeOrgId / ctx.tenantId, a
resolveTenant(...) or resolvePrincipalTenant(...) result (directly or
through a const), or a forEachTenant callback's tenant. If it is verified
another way, such as a signed webhook naming the organization, put
// quickback-allow-tenant-source: <reason> on the line above the call.
The object refuses, before the job runs, a message that is not a result job,
a message for another tenant, a partition or control object, or a missing
runId (TENANT_JOB_INVALID). It sets tenantId from the object's name. The
edge helper throws on the same caller bugs. Pass only a server-verified
organization id: a session's organization, or one resolved through
tenancy.lookups.
A result job is only ever called. A { type: '<result job>' } message literal
anywhere but callTenantJob's message argument fails the build
(TENANCY_JOB_RESULT_PRODUCER), and at runtime runJob, schedule, send,
deliverTenantJob and scheduleTenantJob refuse it (TENANT_JOB_CALL_ONLY).
One that reaches a queue anyway is retried into the dead-letter queue. An
outbox entry or a scheduled alarm for a job that has since become a result job
is dropped as a dead letter (logged), so it never blocks the outbox or the
alarm.
resultSchema is required with result: true, result must be the literal
true, and the handler must be a tenant job (TENANCY_JOB_RESULT_SCHEMA).
Result jobs need isolation: 'durable-object'.
Realtime broadcasts from tenant tables (realtime: { enabled: true }) are
recorded in the object's outbox with the write and replayed by the edge through
the Broadcaster as the response returns (outbox delivery).
createRealtime in tenant code has the
edge's shape, defineRealtime namespaces included: realtime.<namespace>.<event>(…)
is queued as the same broadcast('<namespace>:<event>', …) the edge sends.
Outbox delivery
Everything a tenant object hands the edge goes through its outbox, a table in the object's SQLite written in the same transaction as the writes that caused it:
- realtime frames;
audit_eventsrows;- lookup-index changes;
- jobs for other objects;
- platform queue messages;
- scope revocations (
revocationCheck: 'kv').
The edge delivers each entry at least once, even if the request's waitUntil
is cut off.
-
Claimed, never shared. An entry is claimed before anything applies it: the object reads it and leases it to that claimer for 30 seconds in one synchronous turn. Two drains, two responses, or a response and a sweep never hold the same entry, so a frame is not broadcast twice.
-
In the response. A
handle()response carries what it claimed: at most 32 entries, or 32 KiB of them, in thex-quickback-tenant-outbox-claimheader, which the edge strips. The edge does not call the object again to fetch them. Entries over those caps stay claimable. The response says how many remain, and the edge claims them withclaimOutbox. -
Realtime first. The edge publishes the claimed realtime frames before the response leaves it, in seq order within a claim, one Broadcaster call at a time. No D1 read or write happens before them. The registry report and the D1 bookkeeping run beside the frames, never in front of them.
-
Revocations beside realtime.
revokeentries are stamped into the edge's revocation KV as soon as a claim arrives, beside the realtime frames and never behind the D1 bookkeeping or the restore fence. Like the frames, the write starts before the response leaves the edge, and for a kind the tenant admits (a tenant-backed scope kind) the response does not wait for it. Every lane that admits such a claim (the namespace gate, edge actions, tickets, refresh and authorize) re-proves it against the tenant's own rows, which the same write changed, so the removed subject is refused on its very next request, whatever KV says. A revocation of any other kind (a manualrevokeScopeTokenson a platform kind, whose claims are proven against the platform database) has no such check, so a response carrying one returns once the KV is written (waiting at most 2 seconds). That makes the revocation visible at once in the location that wrote it; elsewhere it arrives within KV's propagation time (up to about a minute). Each (subject, kind, instance) is written once per claim. A KV failure or a missingKVbinding retries the entry; a revocation is never dropped. An entry whose request ended before it was written is claimed again by the next response or drain for that object once its 30-second lease lapses. -
Ordered lane. Lookup and registry entries go to D1 in one batch, guarded by
tenant_registry.outbox_seqand the restore fence (outbox_generation,restore_session), which the batch reads back. Only one claim holding them is in flight at a time, and a later one is never claimed before an earlier one, so the index sees them in order. Audit rows, jobs and queue messages are applied only when that fence holds; otherwise they are released for the reconciling drain. -
Settled, with a receipt. The edge reports each entry back as one of:
delivered: the row is deleted.retry: the row is kept and released with backoff (1 s doubling to 60 s). A realtime frame is retried three times, then recorded failed.failed: the row is dead-lettered (__qb_outbox_dead, counted intenant_registry.outbox_dead_letters), or dropped for realtime.
Settlements ride on the edge's next request to that object from the same isolate. Only if a settlement is still waiting a second later does one
settleOutboxcall carry everything pending, in calls of at most 64. Failures, fence releases, and a claim that holds later lookup or registry rows back settle at once. Any other claim holding lookup or registry rows waits only 50 ms for a request to carry it. Until it settles, the lane is blocked, and an isolate that dies inside that window leaves the lane blocked until the lease expires and the next response or sweep claims the rows. A settlement that never arrives changes nothing: its entries are claimed again after the lease, so delivery stays at least once. That covers an isolate that died, an object that refused the settlement, or a request answered503 TENANT_RPC_VERSION_MISMATCHbefore the settlement it carried was read. -
First request. On an object's first request the report write (which registers it in
tenant_registry) finishes before the claim's lookups, audit rows, jobs and queue sends are applied. Realtime does not wait for it. -
Oversized rows. A row too large for the inline header is claimed by
claimOutbox, which always hands over the oldest claimable row, whatever its size. -
Operator drains.
syncandbackupdrain an active object normally. For an archived object (and during purge, restore and unarchive) they also take entries another claim holds, and a failed realtime replay is not retried. An archived object serves no requests, so nothing else would deliver them. -
Chained jobs. A job delivered to another object of the organization drains that object's outbox too (up to 4 levels deep), after the claim that carried the job is settled.
-
When a claimer dies. Its entries stay leased until the lease expires, then the next response or sweep claims them again. A realtime frame keeps the
publishIdwritten with the row, so the Broadcaster drops a repeat it remembers, and the frameid(evt_<publishId>) lets clients drop the rest. Other kinds were already idempotent: audit rows by id, jobs by run id, queue messages bydedupeKey. -
Order. Realtime frames are ordered within one claim. Across concurrent requests they are not, as before. Realtime no longer waits for the jobs and queue sends of the same claim.
Delivery receipts
Each settled entry leaves a row in the object's __qb_outbox_receipts table:
generationandseq;kind;method: the realtime method or event, or the job or queue message type;target: the Broadcaster shard, job target, queue binding, audit row id or lookup name;ref;createdAtandsentAt(ms);attempts;outcome(delivered,retryingorfailed) andreason;detail:{ publishId, matched, duplicate }for realtime, wherematchedis the number of sockets the Broadcaster sent to;{ dedupeKey }for queue messages;{ runId }for jobs.
A receipt moves to delivered and stays there. Receipts are written in the
same object turn as the settlement, so they cost no extra request.
- Retention. Receipts older than
tenancy.outbox.receipts.retainDays(default 7), and the oldest beyondmaxRows(default 50 000), are pruned on the object's alarm, on the first settlement after each wake, and after every 256 settled entries. - Backups and restore. Receipts are not tenant data. Dumps skip them, and a dump restore keeps them, because restoring data does not undo what was already sent. A point-in-time restore rewinds the whole object, so it rewinds receipts and outbox rows (with their leases) to the bookmark as well. The reconciling sync then moves to a new outbox generation, drops the rewound non-audit rows and releases every lease. Settlements still in flight for the old generation write their receipts but touch no row. Purge erases receipts with the object.
- Reading them.
GET <api>/tenants/:id/outbox-receipts?since=<ms or ISO>&limit=<1-1000>&outcome=delivered|retrying|failed(sysadmin, oldest first). It returns{ receipts, next }; passnextassincefor the next page. In code, calllistTenantOutboxReceipts(env, name, { since, limit, outcome })fromsrc/lib/tenants.ts. - Push and email. These already keep a per-send row in your own tenant
table (for example
comms_log). Pass that row's id asrefwhen enqueueing instead of duplicating it.
Vars in tenant code
The tenant Worker has no bindings of the edge's, vars included, so tenant code
that reads env.<NAME> fails the build (TENANCY_ENV_UNAVAILABLE). List the
plain vars it needs in tenancy.vars:
tenancy: {
isolation: "durable-object",
vars: ["ATTENDEE_APP_URL"],
},
environments: {
dev: { /* … */ worker: { vars: { ATTENDEE_APP_URL: "https://app-dev.example.com" }, /* … */ } },
},Each target's tenant Worker gets the value its own edge has ([env.<name>.vars],
or the top-level [vars] without environments), so tenant code reads
env.ATTENDEE_APP_URL like edge code does; a target that does not set it gets
none. Only plain vars qualify: a declared secret (bindings.secrets, a target's
requiredSecrets, a secrets.required name of the edge) or a name that reads
like one (*SECRET*, *TOKEN*, *PASSWORD*, *_PASS/_PW/_PWD,
*PRIVATE*, *CREDENTIAL*, *APIKEY*, *SALT*, *_SIGNING*, *_HMAC*,
*PEPPER*, *BASIC_AUTH*, *_DSN, database / Redis / Mongo *_URL* and
*_URI*, *CONNECTION_STRING*, *_PEM, and *_KEY(S) / *_KEK(S) other
than *_PUBLIC_KEY(S), *_SITE_KEY(S) and *_PUBLISHABLE_KEY(S)) fails
the build (TENANCY_VARS_SECRET), as does a
name no target defines as a string var (TENANCY_VARS_UNDECLARED). Every other
env.* read in tenant code still fails.
Edge-only modules in tenant code
lib/realtime (needs the Broadcaster), lib/security-audit (needs the audit
database) and lib/auth (Better Auth) do not run in the tenant Worker. Loading
one there would drop its side effect silently, so the compiler guarantees:
- Rewritten. In tenant feature files and tenant queue handlers, a value
import of
lib/realtimeorlib/security-auditis pointed at the tenant version, which records the event in the object's outbox. This coversimport/export … fromin either quote style,import "…", andimport("…")/require("…")with a literal path (quotes or a backtick with no${}). The path is resolved against the importing file, so only paths that land on the project'slib/change.import { createAuth } from "…/lib/auth", optionally aliased and withtypenames alongside, becomes a stand-in that throws: read the caller fromctxinstead. - Refused. The build then walks every module the tenant Worker loads from
src/tenant/index.ts(again after provider build hooks, so a file a hook adds or rewrites is checked too) and fails (TENANCY_EDGE_IMPORT_IN_TENANT, with the import chain) on any value reference to one of those modules that remains: a sharedquickback/libhelper or a platform feature that imports./realtime, any other import fromlib/auth, or a string that resolves to one of them (a relative path, or one starting@/,~/orsrc/— so"src/lib/auth"itself is refused). It also refuses everyimport()orrequire()whose argument is not a plain string literal (+, a template with${}, a variable,.join()): the compiler cannot check it, and the bundler includes every file such a pattern could match. Shared modules are not rewritten because the edge Worker uses them too: importlib/realtimefrom the tenant feature itself, or keep the helper free of it. - Allowed.
import type,typespecifiers,typeof import("…"), comments and ordinary strings.
Helpers run where their caller runs
A helper a tenant action, tenant job or JS trigger handler imports runs inside
the tenant object too. After all files are generated, the compiler walks each
of them through its runtime imports, resolving relative imports and
tsconfig.json paths aliases the way the edge-bundle walk does. Every
authored module it reaches, such as a feature lib/ file, another feature's
file or a quickback/lib helper, gets the checks a tenant action gets:
- edge-only
ctxmembers andenvbindings (TENANCY_CTX_UNAVAILABLE,TENANCY_ENV_UNAVAILABLE); - async
transaction()callbacks (TENANCY_ASYNC_TRANSACTION); - tables of another object (
TENANCY_CROSS_PARTITION); - inserts that choose the id of a scope instance or proving table
(
TENANCY_ADMISSION_ID_STRATEGY).
A tenant job's own body (its top-level constants and execute) gets the same
checks. The checks are per module, not per function: a helper file is scanned
whole once anything imports it, including exports the caller never uses.
The error names the helper and the entry point that reached it. Compiler-
generated modules are not expanded, and package imports are allowed. A relative
or alias import that names no file in the build, or a computed import() /
require(), fails with TENANCY_HELPER_UNRESOLVED: the compiler can't check
code it can't find. A helper nothing in the tenant Worker imports is not checked.
JS handler: triggers on tenant tables run in the object as well. Each
handler's source gets the same checks as an action body, and its imports are
walked like a helper's. A handler receives the env a tenant action gets (a
copy of the tenant Worker's bindings without the backup bucket, which holds
every organization's dumps; tenancy.backup.bucket may not double as any
app, provider or queue binding), so
whatever it passes env to records into the object's outbox in the same way:
createRealtime(env) broadcasts, tenantRuntime(env).enqueue(…) queue
messages and the tenant revokeScopeTokens(env, …). The tenant
createRealtime logs tenant.realtime.no_outbox (an error) when it is given
an env without the outbox, instead of dropping the call silently. A handler whose source can't be extracted as one arrow
function or function expression fails with TENANCY_TRIGGER_UNVERIFIABLE.
Scope revocation
With auth.jwt.revocationCheck: 'kv', changing a scope-conferring row revokes
the carried scope claims that row backed (revoke-on-touch): the subject's
qbrev:scope:<subject>:<kind>:<instance> keys are stamped in the edge's KV
namespace. The tenant Worker has no KV, so for a tenant table the object
records the revocation in its outbox and the edge stamps it:
- In the write's transaction. Every scope-conferring tenant table (the
fromtable of a relationship a scope role is grantedvia) carries SQL triggers that write arevokeoutbox entry on everyDELETEand on everyUPDATEthat changes a column the row confers through (soft deletes included), naming the row's id and subject column and its resource (instance) column. Those columns are the subject and resource columns, each relationship'swhere:columns, every scope role'ssubKeys, every field the table's firewall names (its soft-delete column included), the tenant column and the primary key, so a backfill or data migration of other columns revokes nothing. When the compiler cannot resolve one of them, the trigger fires on everyUPDATE. An update that moves the row to another subject or instance also revokes the new pair. Being SQL triggers, they fire for every write path in the object: CRUD routes, actions,unsafeDb, jobs, trigger handlers, foreign-key cascades and compiler commands. A write that rolls back leaves no revocation.INSERT OR REPLACEdeletes a conflicting row without firing theDELETEtrigger (SQLite'srecursive_triggersstays off, so authored SQL triggers keep their semantics): the build refusesINSERT OR REPLACE/REPLACE INTOon a conferring table in tenant actions and trigger bodies (TENANCY_REVOCATION_REPLACE). Change a conferring row with anUPDATE(oronConflictDoUpdate). - Stamped by the edge, at least once, beside the realtime frames (see
Outbox delivery), at the time the edge applies the entry,
like every edge revocation: a later stamp only revokes more (a token proven
between the write and the stamp re-proves), and no stale read can lower it.
The entry keeps its write time for the receipt. The edge accepts a
revokeentry only for a declaredauthz.scopeskind; one whose instance the platform lookup index places in another organization is logged (tenant.revoke.foreign_instance) and still applied, since a wrong revocation only forces a re-prove. - Restores and purges. A dump restore drops the tables, and a purge erases
the database, without firing a trigger; a point-in-time restore rewinds the
outbox with the data. So a dump restore first queues a revocation for every
conferring row it replaces, in the load's transaction. A point-in-time
restore hands the current conferring rows and any queued revocations to the
control object before scheduling the rewind. Every reconcile
(
POST /tenants/:id/sync, abandon) and every cancel queues them again, and only a reconcile drops them once its requeue committed (an orphaned stash is pruned after a week). A purge queues them once per archive, then drains them in rounds, at most 300 rows per request (each row costs up to two KV writes, and a Worker invocation gets about 1000 KV operations). While any remain it answers409 TENANT_PURGE_DRAININGwithdetails: { remaining, more }andRetry-After: retry the purge until it completes; nothing is queued twice. An edge older than version 12 cannot purge while revocations are queued. Rows that exist again after a restore simply re-prove. - Your own revocations. In tenant code,
revokeScopeTokens(env, subject, kind, instanceId)(an import oflib/jwt-revocationin a tenant feature is re-pointed at the tenant version) records arevokeentry with your write. It takes theenvyour action, job or trigger handler received, neverenv.KV, and throwsTENANT_REVOCATION_UNAVAILABLEwithout the outbox. The other exports oflib/jwt-revocation(revokeUserTokens,revokeMemberTokens,revokeSessionTokens,checkTokenRevocation) are edge-only: in a tenant feature a call fails the build, its parameter type naming the reason. - Admission. A revoked claim stops riding on tokens; admission itself
re-reads the row. Tenant scope admission
requires the proving row's conditions (
where:) and excludes soft-deleted rows through the table's firewall (deleted_at IS NULL).
Lookup indexes
Some lookups start from a value, not an organization: an invite code, a public
slug, an SES message id, a phone number. Declare them in tenancy.lookups:
lookups: {
inviteCode: { table: "invites", column: "code", hash: true },
event: { table: "events", column: "id" },
},SQL triggers in every tenant object record each insert, update and delete of
the column (soft-deleted rows count as deleted) in the object's outbox. The
edge applies them to the platform tenant_lookups table after the response, in
order (tenant_registry.outbox_seq drops a stale batch). Read it on the edge:
import { resolveTenant, resolveTenants } from "../lib/tenants";
const tenantId = await resolveTenant(env, "inviteCode", code); // null, or throws if several
const phoneTenants = await resolveTenants(env, "phone", phone);A lookup named after a scope kind resolves tokens: an instance-bound token
(ctx.oauth.boundTo) or a scope-only principal (ctx.scope.<kind>.id) reaches
the organization whose rows hold that id. Without a matching lookup those
principals get 400 TENANT_REQUIRED. A token for a
tenant-backed kind is the exception:
it routes to the tenant it pins, and the lookup is used only to issue it.
The index converges, it is not transactional with the write: it lags until
the edge drains the object's outbox (after that object's next response, or
syncTenants(env)). Adding a lookup re-sends every current key on each
object's next wake.
Hashed lookups
Every organization's lookup rows share one D1. A lookup whose values are
credentials (invite tokens, share-link tokens, sign-in codes) should not sit
there in plaintext: set hash: true. The row's key is then h1: followed by
HMAC-SHA256(TENANT_LOOKUP_KEY, "<name>\0<value>") as 64 lowercase hex. The
h1: marker tells a hash from a value that is itself hex, so a row written
before hashing is always recognisable. The lookup name separates domains, so
the same value in two lookups gets two unrelated keys. A lookup column whose name reads like a credential (token,
code, secret, key, password) and is not hashed draws a
TENANCY_LOOKUP_PLAINTEXT compile warning. Ids and slugs are fine in plaintext.
- The secret.
TENANT_LOOKUP_KEYis required on the edge Worker (it joins the generated secret inventory and the boot guard): one or more 64-hex keys (openssl rand -hex 32), comma-separated, primary first. The tenant Worker never holds it. Its outbox carries the value to the edge, which hashes before anything reaches D1. A missing or malformed key refuses: the outbox entries retry, and resolution fails, so nothing is ever written in plaintext. - Unchanged API.
resolveTenant(env, name, value)takes the value and hashes it. Receipts record the lookup name only. A dead-lettered lookup row copied to the audit database at purge has its value withheld. - Rotation. Resolution tries every key and re-keys a row it finds under an
older one; writes use the first. Rotate in two deploys, so no isolate still
on the old list writes a row the others cannot clean up: (1) append the new
key (
old,new) and deploy; (2) promote it (new,old) and deploy; (3) runPOST /tenants/:id/rebuild-lookupson every organization, which re-publishes each object's rows under the new key; (4) drop the old key. - Turning it off. Removing
hash: true(or rolling the edge back past it) leaves the hashed rows unresolvable: rebuild every organization's lookups afterwards. - Adopting
hash: trueon an existing lookup. Rows written before it hold the value. Resolution also matches the plaintext row (never a value shaped like a stored hash) and re-keys it. Every outbox drain of an object (a request, a job call, an alarm's drain request, an operator call) and its wake report convert up to 100 of that object's plaintext rows in place first, once per object per isolate. For organizations that do not wake, runPOST /tenants/lookups/migrate(orsyncTenants(env), which also converts up to 200 rows across objects) until it reports none remaining — manually, not on a cron. Any write of a key replaces its plaintext row. Set the secret first, then deploy the tenant Worker, then the edge.
Partitions
One object per organization puts a busy organization on one thread. Shard a high-volume, naturally partitioned feature by its unit of contention:
placement: { partition: "conversation", key: "conversationId" },Each (organization, key) gets its own object, named <orgId>:<key>, so the
object name alone says which tenant it serves. The table stays
organization-scoped (the firewall still applies) and every table in the feature
uses the same key.
| Request | Result |
|---|---|
Key from ?conversationId=, the x-quickback-partition-key header, or the JSON body field | Forwarded to <orgId>:<key>; the query parameter is consumed |
| No key | 400 PARTITION_KEY_REQUIRED |
| Two different keys in one request | 400 PARTITION_KEY_MISMATCH |
| A row whose key differs from its object's | Refused by a trigger in the object (400 PARTITION_KEY_MISMATCH) |
Partition objects carry the same schema and migrations as organization
objects (their organization tables stay empty), take their own backups, and
appear in tenant_registry with partition_key. One partition kind per
project (TENANCY_PARTITION_KINDS).
No foreign key and no action crosses partitions: the build refuses
TENANCY_FK_CROSS_PARTITION and TENANCY_CROSS_PARTITION. Store the other
side's id without .references() (opt it out of the reference diagnostic with
references: { projectId: { references: false, reason } }) and send the update
through the outbox — delivered by the edge after the response, at least once,
to a tenant job in the target object:
const [row] = await db.insert(messages).values({ conversationId: input.conversationId, body: input.body }).returning();
tenantRuntime(env).send("tenant", { type: "conversation.posted", conversationId: input.conversationId });send("tenant", …) targets the organization object; send({ key }, …) a
partition object of the same organization.
Deleting the namespace
A deleted tombstone for QuickbackTenant destroys every tenant database.
The build refuses a bindings.durableObjectMigrations entry that deletes or
renames the class (TENANCY_NAMESPACE_DELETION) unless
tenancy.allowNamespaceDeletion: true is set and no table still lives in
a tenant object. Then the tenant Worker's exports carry
{ "state": "deleted" } and the edge no longer binds the class.
Registry
tenant_registry in the features D1 inventories tenant names and application
state; Cloudflare's object listing does not include those fields. Beside it,
tenant_operations holds one receipt per lifecycle operation
(Lifecycle operations). The edge
upserts a row from the object's wake report. The object repeats the
report on later responses until the edge acknowledges a successful registry
write, so a transient D1 outage does not lose inventory or migration failures.
Each report carries a generation and a strictly increasing sequence, stored atomically with the report in the object's local KV. The sequence advances on every wake and migration retry, including pre-migration progress markers, even when timestamps are identical. The edge updates an existing row only when its generation matches and the report sequence is higher. Delayed or repeated reports cannot replace newer health, schema versions, or errors. An acknowledgement consumes only the exact current report, including its generation and sequence. Unordered RPC failures preserve rows that already have an ordered health report. PITR installs a fresh report generation together with the outbox fence; reconciliation republishes the recovered object's health under that generation. Dump restore installs the same fresh generation. Failed preparation restores the previous report fence, and explicit abandonment republishes current health under its recovery generation.
| Column | Meaning |
|---|---|
state | active, migrating, failed, frozen, archived, purged. Wake reports never overwrite frozen, archived or purged. |
schema_version | Applied tenant migrations |
report_generation, report_sequence | Report ordering fence (initially initial) and last accepted sequence (initially 0) |
last_pre_migration_bookmark | PITR bookmark taken before the last migration |
last_error | The last migration error, or null |
tenant_id, partition_key | The organization, and the partition key for partition objects (id is the object name) |
last_backup_at, last_backup_key | The last finished backup |
outbox_seq | The last ordered-lane outbox entry (lookup, registry) the edge applied |
jurisdiction | The objects' jurisdiction |
Fan out over every object with forEachTenant from src/lib/tenants.ts;
syncTenants(env) drains every object's outbox with it — an operator or manual
sweep: nothing needs it on a schedule, since every alarm asks the edge to drain
its own object. It skips archived and
purged tenants, runs at most concurrency at a time (default 4), and
reports each tenant's failure instead of throwing:
import { forEachTenant } from "../lib/tenants";
const results = await forEachTenant(env, async (tenantId, stub) => {
return tenantId;
}, { concurrency: 8 });Operator routes
The edge mounts tenant operations at <api>/tenants (e.g. /api/v1/tenants).
They are plain Hono routes: they are not in the OpenAPI document or the MCP
tool list.
tenancy.operator.http decides which of them exist. Generating the
lifecycle helpers never implies exposing them over HTTP:
tenancy: {
isolation: 'durable-object',
// No generated operator routes. Your own `operator:` actions call the
// helpers in src/lib/tenants.ts (archiveTenant, restoreTenant, purgeTenant, …).
operator: { http: false },
}tenancy: {
isolation: 'durable-object',
// Only these routes are mounted; nothing else answers under <api>/tenants.
operator: { http: { allow: ['archive', 'unarchive', 'sync', 'retry-migration'] } },
}Route names are the path segments below: retry-migration, archive,
unarchive, purge, sync, rebuild-lookups, schedules (the GET plus
arm, hold and arm-all), lookups-migrate, outbox-receipts, backup,
backups (the list plus backups/counts), bookmark, restore,
cancel-restore. An unknown name, an empty list, or http set beside the
deprecated operator.purge fails the build (TENANCY_OPERATOR_HTTP). An
allowlist that mounts any mutation (every name but backups, bookmark,
outbox-receipts) needs auth.cloudflareAccess covering <api>/tenants;
without it the build fails (TENANCY_OPERATOR_AUTH): a sysadmin session cookie
is not enough authority for a lifecycle mutation. With
http: false no src/tenant/admin.ts is generated and src/index.ts mounts
nothing at <api>/tenants; src/lib/tenants.ts is unchanged. Leaving http
unset mounts every route and warns: the default becomes false in the next
breaking release.
Every route needs a sysadmin (user.role === 'sysadmin', Better Auth admin
plugin). With auth.cloudflareAccess configured, the caller must also come
through Cloudflare Access, and the build fails (TENANCY_ADMIN_NOT_BEHIND_ACCESS)
unless an Access application covers both the path and the API host (the
hostnames in providers.runtime.config.routes and domain; with none
configured, only an application without hosts covers it).
| Route | Effect |
|---|---|
POST /tenants/:id/retry-migration | Clear the reset count and migrate the object again; records the new report |
GET /tenants/operations/:operationId | The durable receipt of a lifecycle operation, settled from the object's receipt or the registry when it was left requested / uncertain. See Lifecycle operations. |
POST /tenants/:id/provision | The only route that creates a tenant object. Refuses an id that is not a Better Auth organization (404 TENANT_ORGANIZATION_UNKNOWN) and a partition name; wakes the organization's object so it migrates and reports, registers it from that report, returns the registry row. Idempotent: a registered organization is returned as is. Use it to move an existing organization into a tenant object (provision, then archive, backup, restore, sync, unarchive). Partition objects are created by their first request, never here. |
POST /tenants/:id/archive | The object answers 423 TENANT_ARCHIVED; tenant_registry.state → archived (forEachTenant skips it). Data is kept. Refuses a purged object, and refuses an unknown id (TENANT_NOT_PROVISIONED) without waking it: no lifecycle operation but provision creates an object. If archive fails after the object archived (for example the registry write failed), retry it: it re-records the object's own report, which says archived. The other operator routes (unarchive, sync, backup, backups, retry-migration, restore) likewise act on registered objects only. When the edge drains queued work from an object it has not registered, it records the state the object reports, archived included, never active by default. |
POST /tenants/:id/unarchive | Serve again; state → active. Registered objects only: an unknown id is refused (TENANT_NOT_PROVISIONED) without waking it, as is a purged one. Refuses (TENANT_RESTORE_PENDING) while a restore is pending or not yet reconciled by POST /tenants/:id/sync, or while an unconfirmed restore or stranded maintenance fence awaits POST /tenants/:id/cancel-restore — the explicit last step of restore quarantine. |
POST /tenants/:id/purge | Body { "confirm": "<id>" }. archived tenants only: records the current point-in-time bookmark in tenant_registry.last_purge_bookmark, records the durable purge marker in the control object, erases the object (deleteAlarm(), then deleteAll()), marks it purged; the object answers 410 TENANT_PURGED from then on. Resumable: after an error, retry it (interrupted purges). Any other state, or a wrong confirm, answers 409. |
tenancy.operator.purge (deprecated: name 'purge' in operator.http.allow,
or leave it out). When your own code imports purgeTenant from
lib/tenants (an organization-delete flow that archives, cleans up platform
rows and storage, then erases the object; a named import, export * or a
named re-export, or a namespace / dynamic import of lib/tenants in a file that
names purgeTenant; type-only imports and comments do not count),
the raw purge route is not mounted: an operator could otherwise erase an object behind your flow's back.
It does not exist at all then (404), and purgeTenant stays available to your
code. Set tenancy.operator: { purge: true } to mount it anyway, or
{ purge: false } to leave it out when nothing imports purgeTenant. An
operator.http.allow list that names 'purge' mounts it regardless of what
your code imports: the list is the explicit opt-in.
Purge is not immediate erasure: Durable Object point-in-time recovery keeps
the data for 30 days after deleteAll(). Purge is still final as far as these
routes go. No route brings a purged tenant back: restore and unarchive refuse it,
and the marker is never cleared. tenant_registry.last_purge_bookmark records
the bookmark from just before the purge. Getting the data back means restoring
that bookmark and clearing the control object's marker in your own code; no
generated route does either.
The same operations are functions in src/lib/tenants.ts: provisionTenantTarget,
retryTenantMigration, archiveTenant, unarchiveTenant, purgeTenant.
Lifecycle operations
Every mutation above runs through one executor, runTenantOperatorOperation
in src/lib/tenants.ts, against one registry, TENANT_OPERATIONS. The
registry is the state machine: which tenant_registry.state an operation may
start from, what it leaves, and whether the caller must confirm.
| Operation | From | To | Confirm |
|---|---|---|---|
provision | no row | the object's own report (active, migrating, failed) | |
archive | active, migrating, failed, frozen | archived | |
unarchive | archived | active | |
purge | archived | purged | { confirm: "<id>" } |
restore | archived | archived (pending) | |
cancel-restore | archived | archived | { confirm: "<id>" } |
retry-migration, sync, rebuild-lookups, backup, schedules.arm, schedules.hold | any registered state but purged | unchanged | |
schedules.arm-all, lookups.migrate | not bound to one tenant | ||
backups, bookmark, outbox-receipts, schedules.status | reads: registered, not purged |
The executor runs, in order: the registry read and the transition check
(409 TENANT_INVALID_TRANSITION, 404 TENANT_NOT_PROVISIONED), the
confirmation (409 TENANT_CONFIRM_REQUIRED), the begin audit row (a failed
write is 503 TENANT_AUDIT_UNAVAILABLE and nothing runs), the operation,
then the outcome row. All of that happens before the tenant object is woken, so
an invalid transition never creates or touches an object. Reads skip the audit.
Every mutation has an operation id and a durable receipt. Send yours as
X-Quickback-Operation-Id (8–128 characters of [A-Za-z0-9_-], else
400 TENANT_OPERATION_ID_INVALID) or let the route mint one; the response
carries it as operationId. The receipt is a tenant_operations row in the
features D1, written requested before the object is touched and settled
completed or failed after, with the operation's parameters hash, actor,
actorAuthMethod, fromState, resultingState, error code and the recorded
response (≤ 4000 characters). Every audit row of the operation carries the same
operationId in metadata, with phase: requested | completed | failed | uncertain. If the outcome cannot be recorded after the operation ran, the
receipt says uncertain, never a false completed.
Repeating an id with the same parameters replays the recorded outcome
without running again (replayed: true in the response; a recorded failure is
thrown again). The same id with different parameters is 409 TENANT_OPERATION_ID_CONFLICT; an id whose receipt is still requested or
uncertain is 409 TENANT_OPERATION_PENDING until it is reconciled.
GET /tenants/operations/:operationId (also reconcileTenantOperation(env, id) in src/lib/tenants.ts) returns the receipt and settles a requested or
uncertain one from durable evidence: the object's own last-operation receipt
(archive and unarchive record the operation id atomically with their change),
else the registry state the operation leaves (resultingState; an open
restore_session for a restore). A settled receipt records reconciledBy and
gains its missing completed audit row with reconciled: true. Without
evidence the receipt is returned unchanged (evidence: 'none') for a human to
decide.
Your own operator: action uses the same executor, so an authored control
plane and the generated routes cannot drift:
import { archiveTenant, runTenantOperatorOperation } from "../../../lib/tenants";
export default defineAction({
path: "/ops/archive-org",
method: "POST",
input: z.object({ organizationId: z.string(), operationId: z.string().optional() }),
access: { userRole: ["sysadmin"], authMethod: ["cloudflare-access"] },
operator: { target: { table: "organizations", by: "input.organizationId" } },
async execute({ env, ctx, input }) {
return runTenantOperatorOperation({
env, ctx,
tenantId: input.organizationId,
operation: "archive",
operationId: input.operationId,
execute: ({ operationId }) => archiveTenant(env, input.organizationId, { operationId }),
});
},
});With tenancy.operator.http: false this is the only way lifecycle mutations
happen, and it carries the same transition check, confirmation, audit rows and
operation id as the generated routes would.
Target resolvers. Three functions in src/lib/tenants.ts separate naming a
tenant from creating one, so no ordinary operation can wake an object for an
identifier nobody owns:
| Function | Admits | Never |
|---|---|---|
resolveTenantTarget(env, id) | A registered, unpurged object (404 TENANT_NOT_PROVISIONED, 410 TENANT_PURGED otherwise) | Creates state or wakes the object |
provisionTenantTarget(env, organizationId) | An organization that exists in Better Auth (404 TENANT_ORGANIZATION_UNKNOWN); wakes and registers it; idempotent | A partition name; an id Better Auth does not know |
resolveRecoveryTarget(env, id) | A registered row including purged (with lastPurgeBookmark), for your own exceptional recovery action | An unknown id |
A partition name <orgId>:<key> is trusted through the registry's own
ownership columns, not by parsing the name: a row registered to another
organization is 409 TENANT_TARGET_MISMATCH. runTenantOperatorOperation
applies the same check to every operation.
FGA
With authz.fga, fga_tuples is a table of every organization's
object. It is not in the platform D1.
ctx.fga(check,listObjects,expand,read,write) exists in the actions of tenant features. It reads and writes that organization's tuples in the object, for a caller who is a member of the organization. A delete is visible to the next check.- The edge hydrates no
ctx.fga. It forwards/fga/v1/*to the caller's organization object, like a tenant route (no organization:400 TENANT_REQUIRED). - Tuples never cross organizations. Two organizations can hold tuples for the same object id, and each sees only its own.
These shapes are refused, because each would read or write tuples from somewhere other than the organization object:
| Code | Cause |
|---|---|
TENANCY_FGA_SHARING | authz.fga.sharing is set. Share tokens and links are opened on the edge without a session. You can still write a user:* tuple with ctx.fga in a tenant action, but only members of that organization can read it: /fga/v1 needs a signed-in member, and platform actions have no ctx.fga. A public page must read the app's own tables instead, such as a share-token table. |
TENANCY_FGA_MEMBERSHIP | authz.fga.membership is set. Those tuples are written from Better Auth hooks on the edge. Write the tuples from a tenant action instead. |
TENANCY_FGA_ACCESS_ARM | A table, action, room binding or realtime.wsTicket declares an access: { fga } arm. Call ctx.fga.check(...) in a tenant action instead. |
TENANCY_FGA_MIXED | A platform feature's action, or an action in a partitioned feature, reads ctx.fga. A partition object's database holds no tuples. |
Moving to 'durable-object' does not copy the tuples already in the features
D1. fga_tuples leaves the platform schema, so the next generated D1
migration may drop it. If you need the existing tuples, copy them into each
organization's object before you apply that migration.
What the compiler refuses
| Code | Cause |
|---|---|
TENANT_ORGANIZATION_UNKNOWN / TENANT_TARGET_MISMATCH / TENANT_PURGED | Runtime: provision of an id Better Auth does not know; a partition name the registry attributes to another organization; a normal operation on a purged row (recovery uses resolveRecoveryTarget). |
TENANT_OPERATION_ID_CONFLICT / TENANT_OPERATION_PENDING | Runtime: a repeated operation id with different parameters; a repeated id whose receipt is not yet settled (reconcile it with GET /tenants/operations/:id). |
TENANT_INVALID_TRANSITION / TENANT_CONFIRM_REQUIRED / TENANT_OPERATION_ID_INVALID | Runtime, not compile: a lifecycle operation refused by the state machine, without its confirmation, or with a malformed operation id. See Lifecycle operations. |
TENANCY_OPERATOR_AUTH | tenancy.operator.http.allow names a mutation route but auth.cloudflareAccess is not configured. Configure Access, allow only backups / bookmark / outbox-receipts, or set http: false. |
TENANCY_OPERATOR_HTTP | tenancy.operator.http is neither false nor { allow: [...] } naming at least one known route, or it is set beside the deprecated tenancy.operator.purge. See Operator routes. |
TENANCY_RUNTIME / TENANCY_PROVIDER | The runtime is not Cloudflare, or the database is not cloudflare-d1. Postgres schemas, RLS and migrations do not port to Durable Object SQLite. |
TENANCY_ORGANIZATIONS | The Better Auth organization plugin is off. |
TENANCY_PLACEMENT_CROSS_TENANT | An organization-scoped table declares placement: 'platform'. Use { platform: true, crossTenant: true }. |
TENANCY_PLACEMENT_UNSCOPED | A table declares placement: 'tenant' but no arm of its firewall — top level, all:, any: or optional — binds an organization column. The tenant pin needs one. |
TENANCY_PLACEMENT_UNCONFIRMED | The build carries no deployed-schema facts (an older CLI, or an unreadable latest snapshot), and a table derives tenant from an any: / optional organization arm. Declare its placement. |
TENANCY_ORG_COLUMNS | A tenant table's firewall binds more than one organization column. Keep it on the platform ({ platform: true, crossTenant: true }). |
TENANCY_PLACEMENT_MOVED | A table in the deployed features D1 derives tenant from its firewall (a shared project switching to 'durable-object', or an any: / optional organization arm). Declare placement: 'tenant' after copying its rows, or { platform: true, crossTenant: true }. See Placement changes on deployed tables. |
TENANCY_PLACEMENT_MIXED_FEATURE | One feature mixes tenant and platform tables. |
TENANCY_FK_TENANT_TO_PLATFORM | A tenant table references a platform table. References to Better Auth tables (user) are ids without a foreign key and are allowed. |
TENANCY_FK_PLATFORM_TO_TENANT | A platform table references a tenant table. |
TENANCY_ASYNC_TRANSACTION | A tenant action, a helper it reaches, or a JS trigger handler passes an async or Promise-returning callback to transaction() on a db / tx handle. |
TENANCY_BATCH_ASYNC | A tenant table's create / update / upsert access, or its guards, is a function. |
TENANCY_UNSAFE_RECREATE | A new tenant migration recreates a referenced table or copies into an unsafe NOT NULL column. |
TENANCY_UNSAFE_ACTION / TENANCY_ROOT_ACTION / TENANCY_CTX_UNAVAILABLE | A tenant action is cross-tenant unsafe (unsafe: true, or an object without crossTenant: false) or sets unsafe.authWrites, mounts outside its feature's path (and outside every tenant namespace prefix), or reads ctx.auth, ctx.tenancy, ctx.mintScope or ctx.seal — or ctx.fga when the project declares no authz.fga — by property access or destructuring, including the auth execute param. They exist only on the edge. The ctx check also covers helpers tenant code imports and JS trigger handlers on tenant tables. Keep cross-tenant unsafe and root-path ingress in a platform feature and call a result job. unsafe: { reason, crossTenant: false } is allowed: see Same-tenant unsafe actions. |
TENANCY_QUEUE_BINDING | Tenant code (an action, a tenant job, a JS trigger handler or a helper they import) enqueues to a binding that is not one of the edge Worker's queues, or one the build cannot read. See Platform queues. |
TENANCY_ENV_UNAVAILABLE | A tenant action, a tenant job's body, a helper either imports, or a JS trigger handler on a tenant table reads an env binding (env.DB, c.env.AUTH_DB, const { BUCKET } = env, …). The tenant Worker has no D1, R2, KV, queue, var or secret bindings. |
TENANCY_HELPER_UNRESOLVED | A module tenant code loads imports a relative or alias path that names no file in the build, or uses a computed import() / require(). See Helpers run where their caller runs. |
TENANCY_TRIGGER_UNVERIFIABLE | A JS handler: trigger on a tenant table has a source that can't be extracted as one arrow function or function expression. |
TENANCY_JOB_CALL_TENANT_SOURCE | callTenantJob's tenant argument is not visibly server-verified (ctx.activeOrgId / ctx.tenantId, a resolveTenant / resolvePrincipalTenant result, a forEachTenant callback's tenant). Opt out with // quickback-allow-tenant-source: <reason>. See Result jobs. |
TENANCY_JOB_RESULT_SCHEMA / TENANCY_JOB_RESULT_PRODUCER | A result job without resultSchema (or a resultSchema without result: true), a resultSchema that is not a stripping or strict z.object, a result that is not the literal true, a result job that is not a tenant job or a project without isolation: 'durable-object'; or a result-job message built anywhere but callTenantJob's message argument. See Result jobs. |
TENANCY_ADMISSION_DEPENDENCY | Edge code reads a tenant-placed table: a scope kind, namespace, ws-ticket, live view, platform feature's authorization, derived JSON key, or a platform action's anchor, refs or operator target. Also a platform action, feature lib/ file, schedule, non-tenant queue handler, or edge-loaded quickback/lib helper or auth hook naming a tenant table, an edge module importing a tenant feature's module or using a computed import() / require(), an ambiguous scope instance, or an unresolved instance on a kind that is also a lookup. At runtime, an edge query on a tenant table throws TENANT_TABLE_ON_EDGE. See Admission tables stay on the platform. |
TENANCY_ADMISSION_MIXED_PLACEMENT, _PARTITIONED, _UNRESOLVED_INSTANCE, _TENANT_COLUMN, _LOOKUP, _SQL_HELPER, _SELF, _HOPS | A scope kind reads a tenant table but can't be tenant-backed. See Which kinds can be tenant-backed. |
TENANCY_LEGACY_SCOPE_ENTER, TENANCY_LEGACY_SCOPE_MINT | A scope kind reading a tenant table is admitted through POST /scope/v1/enter only, or has a subject: 'supplied' | 'both' role. Permanent: use instance-bound OAuth with subject: 'user' roles. |
TENANCY_NAMESPACE_SPLIT, _PARTITIONED, _UNROUTED, _MINT | See Tenant namespaces. |
TENANCY_EDGE_ACTION_PLACEMENT, _NAMESPACE, _CTX | An edge: true action is in a tenant feature, is not under a tenant namespace prefix, or reads ctx.<exposeAs>. See Edge namespace actions. |
TENANCY_TICKET_GATE_UNSUPPORTED | A realtime.wsTicket session gate on a tenant room isn't the tenant-backed kind's own admission, plus at most one { roles } lane of plain organization roles. See Durable Objects → Tenant rooms. |
TENANCY_ROOM_LOOKUP | A resource-backed realtime room (bindings.durableObjects[].access.resource) on a tenant table has no tenancy.lookups entry on its room key column. See Durable Objects → Tenant rooms. |
TENANCY_ROOM_UNSUPPORTED | A realtime room on a tenant table uses a form the tenant gate can't run: multi-resource access.resources, a partitioned table, or an inherited read.access with a record: predicate. |
TENANCY_ADMISSION_ID_STRATEGY | A tenant-backed scope kind's instance or proving table has generateId: false | 'short' | 'serial', allowClientIds: true, or a numeric primary key (see Tenant-backed scope kinds). Or tenant code (an action, a helper it reaches, or a trigger handler) inserts into a scope instance or proving table with an explicit id, a spread that may carry one, or a row it can't inspect. Omit id; // quickback-allow-id is honored only for crypto.randomUUID() / generateId(…). |
TENANCY_REALTIME_PLATFORM_ROOM | With a tenant-backed scope kind, a tenant table's CRUD broadcasts (realtime.scopeTable / scopeKeyFrom) target a room that is not on a tenant table, or the ws-ticket's bound kind is tenant-backed but its scopeTable is a platform table. A tenant object can address only rooms on its own tables. |
TENANCY_REALTIME_TAG_AMBIGUOUS | A tenant-backed kind proves subjects from more than one table, and a recipients resolver calls recipient.scope({ kind, subjectId }) for it (or with a kind the compiler can't read) without provingTable, so the tenant tag can't name the proving table. The error lists each call's file and line. Pass provingTable. |
REALTIME_RECIPIENT_PROVING_TABLE | A recipients resolver's recipient.scope names a literal provingTable that is not one of the kind's proving tables (a relationship from of its roles). Any tenancy mode. |
TENANCY_REALTIME_VIEW_ROOM | A live view is rooted on a tenant table while a scope kind is tenant-backed. View rooms are not tenant-qualified yet. |
TENANCY_LEGACY_SCOPE_TICKET | realtime.wsTicket accepts supplied-scope principals (acceptScope) for a tenant room or a tenant-backed kind. A legacy scope claim carries no tenant, so its ticket cannot name a tenant room. |
TENANCY_UNSUPPORTED | A tenant table declares embeddings, namespace, owns, an .encrypted() / .sealed() column, or a q.storageKey() column. |
TENANCY_TRIGGER_CROSS_OBJECT | A tenant table's sql: trigger (body or when:) names a table outside its own object — a platform table, another partition kind, or a table no feature declares — or uses SQL the compiler does not model (a schema-qualified name, a CTE, a table-valued function other than json_each / json_tree, a statement keyword such as DROP / PRAGMA, unbalanced parentheses). Also a platform table's trigger that names a tenant table. Every violation is listed. |
TENANCY_BACKUP | tenancy.backup.bucket is not an UPPER_SNAKE binding name, or every is not <n>m / <n>h / <n>d of at least 15 minutes. |
TENANCY_LOOKUP | A lookup's table is not a tenant table, or its column is not a column of that table. |
TENANCY_PARTITION / TENANCY_PARTITION_UNSCOPED / TENANCY_PARTITION_KEY / TENANCY_PARTITION_KINDS | A partition name is not a lowercase identifier, the table has no organization arm, the key is not a column (or differs within the feature), or the project declares more than one partition kind. |
TENANCY_FK_CROSS_PARTITION / TENANCY_CROSS_PARTITION | A foreign key, or an action (or a helper it reaches, or a trigger handler), spans two objects. |
TENANCY_QUEUE_PARTITION / TENANCY_QUEUE_MIXED_PLACEMENT | A queue handler imports a partitioned feature or mixes database placements. Split the job into supported handlers. |
TENANCY_QUEUE_TENANT_ID / TENANCY_QUEUE_RESERVED | A platform action, queue handler or schedule builds a tenant-job message without tenantId, or a handler uses the reserved type qb:backup. |
TENANCY_NAMESPACE_DELETION | See Deleting the namespace. |
TENANCY_JURISDICTION_CHANGED / TENANCY_JURISDICTION_UNKNOWN | tenancy.jurisdiction differs from the deployed wrangler.toml's # quickback:tenancy.jurisdiction= line, or wrangler.toml binds QuickbackTenant without that line. Add it with the jurisdiction the objects were created in. A build without wrangler.toml only warns. |
TENANCY_ROUTE_RESERVED / TENANCY_ADMIN_NOT_BEHIND_ACCESS | A feature mounts at /tenants, or Access is configured and no application covers the operator routes. |
TENANCY_TENANT_WORKER_UNKNOWN | The existing wrangler.toml references QuickbackTenant in a form the build cannot resolve to an edge and tenant Worker. Rewrite it as plain [[durable_objects.bindings]] tables under a named edge table. |
TENANCY_TENANT_WORKER_CHANGED | A deployed edge (per the existing wrangler.toml) would bind a different tenant Worker than it does now. See Two Workers. |
TENANCY_ENVIRONMENT_WORKER / TENANCY_ENVIRONMENT_SHARED / TENANCY_ENVIRONMENT_BACKUP | A named environment has no tenantWorker.name, shares a tenant Worker or backup bucket with another target, or names a tenant Worker like an edge or files Worker. A target that does not deploy as the project name reuses the single-target tenant Worker or bucket. A Worker takes a reserved <name>-unselected, <name>-files-unselected or <name>-tenant-unselected placeholder name. backupBucketName is missing while tenancy.backup is set, or set while it is not. See Named environments. |
TENANCY_PURGE_DO / TENANCY_CHANGES / TENANCY_FGA | tenancy.purge, changes, or a top-level fga block (declare it at authz.fga) together with isolation: 'durable-object'. |
TENANCY_FGA_SHARING / TENANCY_FGA_MEMBERSHIP / TENANCY_FGA_ACCESS_ARM / TENANCY_FGA_MIXED | An FGA shape that would reach tuples from outside the organization object. See FGA. |
Fence a result job to an authorization epoch
callTenantJob(env, tenantId, message, runId, { expectedAuthorizationEpoch })
optionally requires a safe nonnegative integer epoch. The tenant compares it
with its live admission permit before executing or replaying a result. A
mismatch returns { status: 'unavailable', code: 'TENANT_AUTHORIZATION_EPOCH_CHANGED' }.
The expected epoch also participates in the durable run hash: reusing a run ID
with a different fence produces TENANT_JOB_RUN_CONFLICT once admitted.
Result handlers receive readonly authorizationEpoch?: number from that same
permit, including ordinary four-argument calls. This value never comes from the
message. Preserve it in a server-owned proof when subsequent work must be
invalidated by restore. Other queue handlers may omit it. This does not change
archive, restore or purge admission. RPC version 8 is required; deploy the tenant
worker before the edge worker. Older workers fail closed on the new contract.
Fence purge to a retained backup
Operators retaining records before erasure can use
purgeTenant(env, tenantId, tenantId, { expectedBackup: { key, generation, bytes } }).
Pass the completed archived backup receipt whose contents were retained. The
options and receipt accept exactly those keys. The receipt must name this
organization's backup and contain a positive safe integer byte count. This
fence does not download bytes and has no backup-download size cap.
Under exclusive control admission, before permanent purge marking, the tenant
requires that receipt to match its latest completed archived backup, current
generation and clean state. A completed restore, subsequent writes after
unarchive, or a replacement backup refuse with TENANT_PURGE_BACKUP_STALE.
Nothing is erased and a known rejected fence is released. In-flight work or
restore continues to refuse under the existing maintenance contract.
The permanent purge marker retains the receipt identity outside erased tenant storage. A retry with that same receipt can finish after an interrupted erase; a different receipt, or a fenced retry of an originally unfenced purge, refuses. The original three-argument trusted-operator API remains available, including finishing an already marked purge without supplying a receipt. It cannot alter the permanent receipt or re-enable the tenant. Applications requiring retention must consistently supply the fourth argument before their first irreversible purge. RPC version 9 is required: deploy the tenant worker before the edge worker; older workers fail closed. No room-deletion callback or general maintenance callback is exposed by this API.
Hold admission through a resource room's native commit
Result jobs and outbox delivery do not keep a permit around their consumer's
later native Durable Object commit. For a declared, locally authored room on a
non-partitioned tenant resource, call acquireTenantRoomPermit inside the
actual room operation, while holding its own work gate:
import { tenantRoomSession } from "../../../lib/tenant-room-session";
import { acquireTenantRoomPermit, releaseTenantRoomPermit } from "../../../lib/tenants";
// Method on the existing room class. No FIFO, admission, name initialization,
// or native storage I/O here: control calls back while this actor awaits it.
getTenantRoomSession(version: number) {
return tenantRoomSession(this.ctx, version);
}
// Inside the actual authorized room operation, after taking its work gate:
const permit = await acquireTenantRoomPermit(this.env, verifiedTenantId, this.ctx, {
binding: "PAGE_ROOM",
key: roomKey,
operationId, // stable server-owned id for THIS operation's exact retries
expectedAuthorizationEpoch,
});
// All SQL authority reads, binary checks and native transaction retries must
// settle here. Keep the existing resource-pointer/native CAS checks.
await commitNativeCheckpoint();
await this.ctx.storage.sync();
await releaseTenantRoomPermit(this.env, this.ctx, permit);Do not put release in an unconditional finally: a timeout, failed sync or
unsettled continuation is not proof that no native commit can still happen.
An ambiguous acquire/release retains quarantine; retry that exact operation or
prove the actual room actor was replaced. A new operation must have its own ID.
Released/denied identities remain terminal in individual durable records; the
shared index contains only active operations. Delayed acquisition cannot
resurrect them. Exact concurrent retries share the final ownership outcome;
late failed cleanup cannot release a grant another retry already received.
Duplicate requests wait on their own bounded timer, never another request's
I/O promise. Timing out does not release admission.
The tenant verifies the live actor session through the declared existing room
namespace, the current epoch/generation, and exactly one undeleted resource row
using its parsed organization/key/deletion columns. It returns captured
authorizationEpoch, generation and the permit token. This is ownership and
restore admission, not user authorization: keep the application's operator/user
checks and resource-pointer CAS. Ordinary admitted jobs can still change a
resource while this permit is held; the permit blocks exclusive restore, purge
and backup maintenance, not ordinary application edits.
The public helper takes the actual native DurableObjectState and derives its
private session; it does not accept a supplied session string. An edge caller
with a made-up object gets a different UUID and cannot borrow the live room
session merely by reading the callback. Release also requires that same actual
state/session. A session is private to that actor instance, never persisted or
a module-wide UUID shared by rooms. tenantRoomSession uses a WeakMap keyed by
its actual DurableObjectState and performs no I/O. Pruning compares the live
room session, not the tenant actor's session. Tenant/control restart, elapsed
time, a missing method or a probe error never retire a live room permit. An
actual room replacement proves that its old continuation cannot access native
storage. It may still finish external I/O; immutable R2/artifact safety remains
a separate application protocol. See DO shutdown semantics
and storage sync/output gates.
Room records and an armed compatibility guard are stored atomically outside the tenant's restored database. Older control code sees a non-purge operation with an unknown recovery kind and refuses restore, purge and recovery. It cannot prune the separate room records. Only the room-aware release or a proven room restart clears the last active record and its guard together. Unknown or inconsistent state blocks admission and exclusive maintenance. RPC version 10 is required; room permit calls never fall back to an older protocol. Rolling back while a room permit is held intentionally retains quarantine.
The compiler adds private tenant-worker bindings to existing edge room
classes, with the matching edge script for each named environment. It adds no
room class, namespace or migration. External-script rooms, partitioned resource
rooms and rooms without a provable parsed organization column cannot use this
API. Authored tenant handlers still cannot read the private bindings. Deploying
a tenant that references a previously undeployed edge class may require the
existing edge class to be deployed first. For an upgrade, a compatibility phase
can add the zero-I/O getTenantRoomSession method using a private per-instance
UUID to the already existing edge room class before introducing the tenant's
reverse binding or the full room-permit configuration. Then the room-aware
tenant can bind that existing class, followed by the full edge configuration. The full room class must then use
the same generated WeakMap helper for both its callback and acquisition; a
compatibility-only private UUID must not remain a different identity source.
This is an artifact/rollout plan, not a deployment performed by compilation.
A completely fresh pair has circular edge TENANT and tenant ROOM references.
Its account bootstrap/provisioning order needs a separate account-stage drill;
local generation and Wrangler dry-run do not prove that one deployment creates
such a pair without extra setup. Named-environment bindings must target that
specific environment's existing edge Worker/class, and the deployment must
verify those physical namespaces before enabling room commits. No raw SQL
opener or arbitrary callback bridge is added.
Local fault-adapter tests can drop an RPC reply before helper validation or
reject an application acknowledgement after native storage.sync() completed.
Those seams do not establish physical network loss or a native storage engine
failure; the corresponding native/account failure drills remain open.
Tenant Purge
tenancy.purge erases one organization — every organization-firewalled table, the R2 objects its storage keys reference, and optionally the Better Auth organization — from a list the compiler derives from the firewall.
Using the API
Overview of the API generated by Quickback. CRUD endpoints, filtering, batch operations, views, and custom actions.