Frequently asked questions¶
Short answers to common questions from developers who deploy services and operators who run a node. Each answer links to the page that explains the mechanism in full.
General¶
What is 21 Cloud Functions Runtime?¶
It is a self-hosted, multi-tenant serverless runtime. You run one binary (rt) in one container with one data
volume, and your tenants deploy TypeScript/JavaScript services (run in Deno V8 isolates) or WebAssembly components
(run by wasmtime) through a REST API or the 21ci CLI.
Every service gets an HTTP route, interval and cron schedules, a durable queue with a dead-letter queue, its own KV, SQL (SQLite) and blob storage, environment variables, sealed secrets and calls to other services. The platform runs on your servers and you are the operator. See Architecture for how the pieces fit together.
Who is it for?¶
Two groups of people use it:
- Operators who host other people's code: a product that lets its users add webhooks, bots, integrations and automations, or an internal platform that needs many small services without a container per function. Operators install the node, define plans and quotas, and create tenants.
- Developers (tenants), including their AI agents, who write and deploy services through the API the operator gives them.
If you only need to run a handful of your own functions and do not need tenants, plans or quotas, a single-tenant setup works too: create one tenant for yourself. Start with Get started.
Is it ready for production?¶
The core path is implemented and tested: both executors, all triggers, storage, the scheduler, the control API and the CLI. The current version is 0.1.0 and there is no tagged release yet. Some hardening from the design is not in this release: the V8 sandbox, the supervisor/executor process split and a seccomp filter. The node is also a single machine with no clustering or failover.
That makes it a good fit today for trusted or semi-trusted tenants and for platforms where you control who deploys code. Read Status and roadmap before you put mutually hostile tenants and sensitive data on one node.
What do I need to run it?¶
A Linux x86_64 or arm64 host with Docker (Compose v2 and BuildKit), or a Linux host where you run the rt binary
under systemd. macOS works for development with make dev, but not as a production host. The node detects the
container's or cgroup's CPU and memory limits and sizes itself from them; Resource requirements
explains the sizing rules and how to measure the cost of your own services. Building the image the first time
downloads V8 and compiles the workspace, which takes 20 minutes or more. See Install
and Self-hosting.
Running code¶
Deno or WASM: which should I use?¶
Use Deno for services you want to write quickly: webhooks, bots, API glue and code an AI assistant writes for you.
You get TypeScript, standard Web APIs, npm: and jsr: packages, and the std:* platform modules. Use WASM for
hot paths that need low per-request overhead and a small memory footprint: gateways, auth checks and stateless
high-load microservices, written in Rust, Go (TinyGo), C, Python, or JavaScript compiled with jco componentize.
| Deno | WASM | |
|---|---|---|
| You write | TypeScript or JavaScript | Anything that builds a WASI Preview 2 component exporting wasi:http/incoming-handler |
| Dependencies | npm:, jsr:, allowlisted https: imports, bundled at build time |
Whatever you compile into the component |
| What a fresh task costs | A new V8 isolate plus evaluating your module top level | A new instance from a pre-compiled component, plus your start-up code |
| What warm reuse skips | Isolate creation and top-level evaluation | Instantiation and start-up code |
| Typical artifact size | Kilobytes of source; the bundle includes your npm dependencies | Tens of KB (Rust) to tens of MB (embedded language runtimes) |
You can mix both runtimes in one tenant, and services call each other the same way regardless of runtime. Performance characteristics explains the costs, Languages and frameworks lists what works on each runtime, and the Deno guide and the WASM guide cover each one.
Does npm work?¶
Yes, for most packages that run in a browser-like environment. You import npm:pkg@range or jsr:@scope/pkg@range,
and the runtime resolves it at build time through esm.sh (build.npm_cdn, ?target=es2022), records every
fetched module's hash in the version's rt.lock, and vendors the modules into the artifact store. Nothing is fetched
at request time, and rebuilds reproduce the same dependencies. Bare specifiers such as zod work through an rt.json
import map, and https: imports only from hosts in build.allowed_hosts.
What does not work: node: imports in your own code are rejected at deploy with 422, and Node built-ins exist only
as far as esm.sh polyfills them for that target. Packages that need real sockets, the file system or processes
(node:net, node:fs, child_process and similar) do not work, and native addons are not supported. A Node
compatibility layer is still an open decision. Test your dependencies early: Deno languages and
frameworks has a table of common frameworks and SDKs and a checklist for deciding whether a
package will work, and sample/deno/npm-zod/ in the runtime repository is a working example. Details:
Services, versions and builds.
Which languages can I use for WASM?¶
Any toolchain that produces a WASI Preview 2 (0.2) component exporting wasi:http/incoming-handler: Rust with the
wasm32-wasip2 target and JavaScript through jco componentize are the paths the samples use; Go through TinyGo,
Python through componentize-py and C through wasi-sdk are possible too. Core modules, components without that
export, components built for WASI 0.3, and components that use threads or shared memory are rejected with
422 invalid_artifact. The upload limit is 50 MB. WASM languages has the per-language
requirements and limits.
Languages that ship a large runtime inside the component (for example Python compiled to WASM) pay their start-up
cost on every fresh instance, because build-time pre-initialization is not in this release. Turn on instance reuse
(wasm.reuse_instances) for such services. See the WASM guide and the std:host WIT
reference.
Do I need the CLI?¶
No. Everything goes through the REST control API, which is described by OpenAPI 3.1 (GET /v1/openapi.json) and
browsable in the Scalar UI at /docs on the control listener. curl and any HTTP client work, and so does
sample/scripts/deploy.sh in the runtime repository.
21ci is a convenience: one small binary with no runtime dependencies that covers every control API operation
(contexts, doctor and whoami to check a setup, push with archive building and --wait, rollback, logs -f,
--json output and --retry for scripts, and so on). You build it from the runtime repository with
make cli-install. See the CLI reference and the API reference.
Can an AI assistant deploy and manage my services?¶
Yes. The node serves the Model Context Protocol at POST /mcp on the control listener. Claude Code and Cursor connect
to it as an HTTP MCP server with a tenant key in the Authorization header; Claude Desktop and other clients that
start a local program use the 21ci mcp bridge. The assistant gets tools to list and create projects and services,
deploy TypeScript or JavaScript, roll back, read logs and stats, set env variables and secrets, and manage tokens,
limited to what its key's scopes allow. Give it a token made for it with 21ci token create (narrow scopes, an
expiry). See AI assistants (MCP).
How do I organise many services, or staging and production?¶
With projects and environments. A project groups services and lists the environments they may be in; every tenant
starts with a default project and its production environment. Create 21ci project create web --env
production,staging, then create services with --project web --env staging. Services of the same project in the same environment may
call each other by default (within the same project). Slugs are unique on the node, so name copies api and api-staging. See
Projects and environments.
Can services call each other?¶
Yes. A Deno service calls another with call(slug, request) from std:services, a WASM service with std:host/call.
The call runs in-process, without an HTTP hop; the child's wall-time limit is the smaller of its own and the caller's
remaining time, and call chains stop at depth 15 (508 call_depth). A failing child returns a 5xx response rather
than throwing. Services can also enqueue work for each other with std:queue / std:host/queue.
By default a service accepts calls and enqueues from services of the same tenant and project in the same environment
(environment label, default production). The target can list exactly who may call it in call.allowed_callers
(slugs, svc_… ids or env:<name>); everyone else gets call_not_allowed. See
Service calls and queues.
Can a service call a service of another tenant?¶
Only if you, the operator, allow it. Cross-tenant calls are refused with cross_tenant_call unless the caller's plan
sets allow_cross_tenant_calls = true, and even then the target must list the caller by its service id in
call.allowed_callers. Both sides have to agree. See Tenants, plans and keys.
Does it support WebSockets?¶
Yes. A service accepts an upgrade with sockets.accept(req) (std:sockets); the node holds the socket and runs
one task per message, close or error, so no isolate stays alive between messages. A durable service keeps
state per object (a room, a document) with its own storage and alarm, much like Cloudflare Durable Objects, and
sockets survive deploys. Guests can also connect out with new WebSocket(url). See
WebSockets and durable objects. For one-way live updates, streaming the response body or
long polling, as the live queue tutorial does, still works too.
Are there email triggers?¶
No. The triggers are HTTP (path route, custom host or {slug}.{base_domain}), interval and cron schedules, the
durable queue, service calls, WebSocket events and durable-object alarms. Email triggers are not part of this release. See Triggers.
Isolation and security¶
Why not a Docker container per function?¶
Because a container per function makes every idle service pay for a process, a runtime and an image, and every cold request pay for starting them. Here a service that is not running costs a row in a database and some files; a request creates a V8 isolate or a WASM instance inside the already-running node. A deploy is an upload, with no image to build or registry to run.
The trade-off is the isolation boundary: a container or microVM is a stronger wall than an isolate inside a shared process, and it runs any Linux program. If some tenants must be strictly separated, run them on separate nodes. The full comparison, with the rows that matter (boundary, cold start, density, compatibility, operations), is on Isolation compared.
How is tenant code isolated?¶
Every task runs in its own sandbox: a fresh V8 isolate for Deno (no heap shared across requests or tenants) and a fresh wasmtime instance for WASM. When you turn on warm reuse, an isolate or instance is only reused for the same tenant, service and version, and it is discarded after any error, kill or limit. Around that:
- Limits per task: CPU time, memory and wall time, enforced by the runtime (
502 cpu_limit,502 oom,504 timeout). - No escape hatches in the guest: Deno guests have no
Worker,SharedArrayBuffer,Atomics.waitorShadowRealm,WebSocketonly connects out through the egress guard, andDenois reduced toDeno.env. WASM components get no threads, no preopened directories and no sockets. - Frozen clock:
Date.now(),performance.now()and WASI clocks advance only at I/O and timer boundaries, which removes local high-resolution timers. - Egress guard: every outbound request goes through an SSRF guard that denies loopback, private and metadata addresses and mail ports.
- Data separation: each service has its own SQLite file and blob directory; secrets are sealed with AES-256-GCM and are write-only through the API; API keys are stored hashed.
The Security model page explains each layer.
What isolation features are missing in this release?¶
Three parts of the design are not in this release:
- V8 sandbox. The node uses a prebuilt V8 without the V8 sandbox, and
GET /v1/node/statusreportsv8_sandbox: false. A bug in V8 would expose the whole process. planned - Supervisor/executor split. Everything runs in one process (
--role=all), so the master key and decrypted secrets share an address space with tenant code. planned - seccomp filter and Landlock. With
security.seccomp = truethe node only appliesPR_SET_DUMPABLE=0,NO_NEW_PRIVSandRLIMIT_CORE=0on Linux. A full syscall filter is not on the current plan.
Until the first two ship, the per-task isolate is the main boundary. Do not mix mutually hostile tenants with sensitive operator data on one node. Status and roadmap tracks these items.
Can tenant code reach my internal network?¶
Not through fetch, wasi:http or sockets. Outbound HTTP passes the SSRF guard, which resolves the host once, pins
the address and denies loopback, private ranges, link-local and cloud metadata addresses (including IPv6 forms that
embed them), the node's own listeners and configured endpoints, and ports 25, 465 and 587. You can deny more with
egress.denied_hosts, egress.denied_cidrs and egress.denied_ports, and plans or services can restrict egress
further. When tenants should reach your own platform, give them an operator SDK instead: it runs in the runtime and calls
your API through a binding, which adds your credentials upstream; no guest ever sees them. See
Operator SDKs and bindings.
Performance and scaling¶
What about cold starts?¶
A request can be cold at three levels, and each has its own fix:
| Where the version is | What the request pays | How to avoid it |
|---|---|---|
| Not compiled yet (T3, source only) | A build: transpile, fetch dependencies, bundle, compile. Seconds, not milliseconds | Build at deploy: ?compile=eager, 21ci push --wait, or the min_latency profile (eager by default) |
| Compiled on disk (T2), not in memory | A load from the disk cache with an integrity check | min_latency preloads the most used services at startup and keeps them in memory longer |
| In memory (T1) | Creating a fresh isolate or instance for the task | Warm reuse: deno.reuse_isolates, wasm.reuse_instances |
| Idle long enough to be scaled to zero | No warm instance is left | scaling.min_instances keeps that many warm, and the node pre-loads the live version at start and on deploy |
With the default lazy builds (balanced, max_density), the first request after a deploy compiles the version; if
that build fails, the request gets 502 build_failed and the live pointer reverts to the previous healthy version.
After a restart, the first request per service is usually a T2 hit.
The per-request cost matters most for Deno: a fresh isolate costs CPU on every request, and the pre-created isolate
pool hides that latency but not the CPU. Warm reuse (deno.reuse_isolates) removes most of it. WASM instances are
cheap to create from a pre-compiled component, but a component that embeds a language runtime pays its start-up code
per instance. Every response carries a server-timing header with queue, wait_runtime, init and exec, so you
can see where the time went. Details: Performance characteristics,
Caching and Performance tuning.
Why do I get 503 queue_timeout and never 429 for capacity?¶
Because busy capacity is not an error. When the node, the tenant's plan or the service is at its concurrency limit,
a new request waits in its service's FIFO queue instead of being rejected; tenants take turns as slots free up,
then the services within a tenant. A request that waits longer than sched.queue_timeout (1 s in min_latency, 5 s
in balanced, 10 s in max_density) gets 503 queue_timeout with a Retry-After header estimated from how fast the
queue drains. If the queue itself is full (sched.max_queued, sched.max_queued_mb, or a service's
scaling.max_queued), you get 503 overloaded right away.
429 is reserved for limits someone chose: plan quotas (rps, invocations_per_day), a service's ingress
rate_limit, repeated failed authentication, and control API quotas such as build_quota and queue_full. If you
see 503 queue_timeout often, look at the service's admission state (21ci stats <svc>, or admission in
GET /v1/services/{s}/stats: its adaptive limit, queued requests and queue wait) and, as the operator, at
21ci node scheduler. Usually the node is short of CPU or memory for its load: turn on warm reuse for Deno services,
or give the node more resources. 21ci --retry N rides out short busy periods in scripts. See
Scheduler and autoscaling and the error reference.
How do I scale?¶
Vertically, on one node. Give the container more CPU and memory: with sched.processors = 0 and
memory.budget_mb = 0 (the defaults) the node uses the cgroup's CPU quota and 85 % of its memory limit. Within the
node, each service's concurrency limit is adaptive: it grows while requests wait and shrinks under memory pressure,
rising latency or saturated processors. A service owner can bound it with scaling.min_instances (kept warm) and scaling.max_instances (hard cap). Warm pools follow
recent demand within sched.warm_share of the memory budget and scale to zero when a service is idle.
Pick a profile to set the overall trade-off (max_density, balanced, min_latency), per node or per service. For
Deno, the biggest single lever is isolate reuse. See Performance tuning and
Configuration.
Can I run several nodes or get high availability?¶
Not in this release. The runtime is a single node: there is no clustering, replication, shared metadata or failover,
and schedules and queues live on that node's volume. Two nodes you start are fully independent: they share no tenants,
services or data. State survives restarts (services, versions, live pointers, schedules, queues, env and secrets are
all on the volume), so the practical setup is a container restart policy, health checks on /healthz and /readyz,
and regular backups. The data layout is meant to keep a later multi-node design possible. See
Production deployment.
Storage and state¶
Can I keep state between requests?¶
Not in memory, by default. Each task gets a fresh isolate or instance, so module top-level code runs once per task
and globals start empty every time. With warm reuse (deno.reuse_isolates, wasm.reuse_instances, both on by
default only in min_latency), top-level code runs once per isolate and globals or linear memory persist, but only
as a cache: an isolate is replaced after any error, kill or leftover timer, after an env or secret change, after 1,000
requests, when idle, and under memory pressure, and consecutive requests may land on different isolates.
Keep anything that matters in the per-service storage: std:kv for small values (with optional TTL), std:sql (or
its synchronous form std:sqlite, also importable as bun:sqlite) for relational data and transactions, std:blob for files and large objects. WASM services use the same stores through
std:host/kv, std:host/sql and std:host/blob. See Storage and the
guest API reference.
How do I keep a counter that is safe under concurrency?¶
Use kv.incr. It adds to an integer counter and returns the new value atomically: the read and the write run in one
transaction on the service's single SQLite writer, so concurrent requests never lose an update. A missing key starts
at 0, and ttlSeconds applies when the call creates the counter, which gives you fixed windows for rate limits:
import { kv } from "std:kv";
export default async function (req: Request): Promise<Response> {
const hits = await kv.incr("hits:home"); // 1, 2, 3, … across all requests
const minute = Math.floor(Date.now() / 60_000);
const recent = await kv.incr(`rl:${minute}`, 1, { ttlSeconds: 120 });
return Response.json({ hits, recent });
}
A plain get followed by set is not atomic and can lose updates. For counters that must change together with
other rows, use an upsert inside an std:sql transaction. WASM components get the same operation as incr in
std:host/kv.
Where is the data stored? Can I use S3?¶
On the node's local disk, under the data volume: one SQLite file per service for KV and SQL, a blob directory per
service, and the content-addressed artifact store for uploaded sources. S3 backends for artifacts and blobs are not in
this release; the [s3] section and storage.blob_backend are accepted but unused. Storage per tenant is capped by
the plan's storage_mb. See Storage.
Operations¶
How do I back up a node?¶
Back up meta.db, queue.db, artifacts/, services/ and blobs/ from the data volume, and the master key,
stored separately. cache/ and deps/ can be rebuilt and may be skipped. A hot cp -r of SQLite files is not
consistent: stop the node first, or take sqlite3 <db> ".backup" copies of meta.db and queue.db. A tenant (or
you) can also export one service's KV and SQL data as a SQLite file with GET /v1/services/{s}/data/export (scope
data:admin).
docker compose stop rt
docker run --rm -v rt_rt-data:/data -v "$PWD":/backup debian:bookworm-slim \
tar czf /backup/rt-data.tgz -C /data .
docker compose start rt
There is no built-in backup schedule or off-site copy; that is the operator's job. See Production deployment.
Where do logs and metrics go?¶
- Service logs:
console.*in Deno and stdout/stderr in WASM go to a per-service log store on the volume (kept forlogs.retention_days, 7 by default, up tologs.max_mb_per_service, 50 MB). Read them withGET /v1/services/{s}/logs, stream them live over SSE with?follow=1, or run21ci logs <svc> -f. Build logs have their own stream (21ci build-logs <svc> <version>). - Per-service stats and events:
GET /v1/services/{s}/stats(counts, error rate, p50/p95/p99 and the admission state) andGET /v1/services/{s}/events(deploys, rollbacks, env and secret writes, host, schedule and DLQ operations, data exports). - Node logs: JSON on stdout, filtered with
RT_LOG, so any container log collector picks them up. - Metrics: Prometheus text at
/metricson the metrics listener (:9090), next to/healthzand/readyz. Per-service series are off unless you setmetrics.per_service = true.
OTLP trace export is not in this release: the [otel] section is accepted but nothing is exported. Requests carry a
traceparent header into the guest. See Observability and
Metrics.
How do I rotate a secret or an API key?¶
Set the secret again. Secrets are write-only and apply to the next task, with no redeploy; warm isolates and instances that saw the old value are discarded.
API keys rotate the same way whoever manages them: create a new key, switch clients over, then revoke the old one.
With a key that has keys:write you do it yourself (21ci token create, 21ci token revoke, /v1/keys); otherwise
the operator does (21ci key revoke, DELETE /v1/tenants/{t}/keys/{key}). Revocation takes effect at once. See
Tenants, plans and keys and the CLI reference.
How do I rotate the master key?¶
The master key seals tenant secrets and derives the API-key pepper and the compiled-cache integrity key. Rotation takes two restarts and no downtime beyond them:
- Back up
meta.dband generate a new key withopenssl rand -hex 32. - Restart with
RT_MASTER_KEY=<new>andRT_MASTER_KEY_PREVIOUS=<old>. New secrets are sealed with the new key, old ones still open, existing API keys keep working, and the compiled cache rebuilds on demand. - Run
rt secrets resealwith the node's configuration (docker compose exec rt rt secrets reseal). It is safe to re-run and exits1if a value opens with neither key. - Have tenants move to new API keys: key hashes cannot be re-sealed, so old keys only work while
RT_MASTER_KEY_PREVIOUSis set. - Restart without
RT_MASTER_KEY_PREVIOUSand destroy the old key.
If you lose the master key, sealed secrets cannot be recovered. See Production deployment.
Does the runtime terminate TLS?¶
No. rt serves plain HTTP/1.1 and h2c. Put a reverse proxy such as Caddy or nginx in front of the public
listener (:8080) to terminate TLS, and set ingress.trusted_proxies to the proxy's address so client IPs from
X-Forwarded-For are trusted. Keep the control (:8081) and metrics (:9090) listeners on loopback or a VPN: they
have no network access control of their own. See Production deployment and
Custom domains.
Troubleshooting¶
Why does my deploy fail with 422?¶
validation_failed means the upload was rejected before anything was built: the source does not parse, an import
specifier is not allowed (for example node: in your own code), or a file or archive is over a size limit.
details.diagnostics in the response lists file, line and column. invalid_artifact means a WASM upload is not a
WASI 0.2 component that exports wasi:http/incoming-handler. See Deploy and roll back and the
error reference.
Why is the first request after a deploy slow, or 502 build_failed?¶
With lazy builds the first request compiles the version (see What about cold starts?). If
that build fails, the request gets 502 build_failed and the live pointer goes back to the previous healthy version.
Read the build logs (21ci build-logs <svc> <version>), fix the source and upload again. 21ci push --wait
builds at deploy time so you see the failure immediately.
My service answers 500, 502 or 504. What happened?¶
When the platform itself ends a task, the response carries an x-rt-reason header (a handler's own responses never
do): guest_error (500) means the handler threw or returned something that is not a Response, cpu_limit and oom
(502) mean the CPU or memory limit was exceeded, timeout (504) means the wall-clock limit passed. Read
21ci logs <svc> for the stack and the response header x-rt-task-id to find the task. What each outcome does to
the isolate and to in-flight work is on Failure behavior.
Why does my service return 404?¶
Either the slug or host is wrong, the service has no live version, triggers.http is off, or the service has
access: internal and you are calling the public listener. GET /v1/services/{s} shows the live version and the
access mode. See Triggers.
Why does Date.now() not advance while my code runs?¶
The clock is frozen for synchronous code on purpose: it removes local high-resolution timers as a side-channel
source. It advances at I/O and timer boundaries. To measure your own latency use server-timing and the stats, not
Date.now() around CPU work. See Execution and isolation.
A package works locally but fails after deploy¶
The build resolves npm: and jsr: imports through a CDN for a browser-like target, so a package that imports Node
server modules deploys fine and throws when it calls them. The checklist and a table of common packages are in
Deno languages and frameworks.
I get 403 call_not_allowed between my own services¶
A service accepts calls only from the same tenant, project and environment by default. Put the caller in the target's
call.allowed_callers (slug, service id or env:<name>). See Configure a service and
Service calls and queues.
Can I reuse code written for another fetch-based platform?¶
Often with small changes. The default handler shape, export default async function (req) { return new Response(…) },
is a plain web-standard handler. The object form export default { fetch, scheduled, queue } looks like a Workers-style
module, but handlers receive (request, ctx) and read configuration from Deno.env, not from an env argument.
Replace platform-specific storage with std:sql, std:kv and std:blob, and do not rely on module state surviving
between requests. See the Deno guide and the guest API reference.
Licensing¶
What is the license?¶
The runtime is licensed under the Apache License 2.0 (SPDX identifier Apache-2.0). The full text is in the
LICENSE file of the repository.
Can I use it in a commercial product?¶
Yes. The Apache License 2.0 allows commercial use, modification and redistribution, subject to its terms (license and notice requirements, no trademark grant). See About.
What is in the open core, and what is paid?¶
| Capability | Open core | Paid |
|---|---|---|
The runtime itself: both executors, triggers, storage, scheduler, control API, 21ci CLI |
Yes | Yes |
| Supervisor/executor split and the sandboxed V8 build, once they ship | Yes | Yes |
| Executor sharding: several executor processes per node, tenant groups pinned to them | No | Yes |
| KMS or Vault secrets backend | No | Yes |
| Durable, exportable audit log | Events API and logs only | Yes |
| SSO, OIDC and RBAC for the control API | No | Yes |
| Web console and fleet management | No | Yes |
Paid modules live outside the core and plug into it through extension points, so the open core is complete on its own.
See also¶
- Status and roadmap: what is in this release and what is not.
- Concepts: how the runtime works, one mechanism per page.
- Isolation compared: containers, microVMs, VPSes and this runtime.
- Error reference: every
x-rt-reasonand control API error code.