Performance tuning operator¶
This guide lists the knobs that change where a request's time goes, and what each change costs in memory or isolation. It is for operators tuning a node and for developers who want to know why their service is fast or slow. The cost model behind it is in Performance characteristics; how to size a node and measure the cost of your own services is in Resource requirements.
Where the time goes¶
Every request passes the same phases, and every response reports them in the server-timing header:
flowchart LR
A["queue: wait for a run slot and a processor"] --> B["wait_runtime: build or load the code"]
B --> C["init: prepare isolate / instance"]
C --> D["exec: run the handler"]
$ curl -sS -o /dev/null -D - "$RT_INVOKE/s/hello/" | grep -i server-timing
server-timing: queue;dur=0.004, wait_runtime;dur=0.000, init;dur=0.120, exec;dur=0.310
(Example values, in milliseconds.) Read it like this:
| Phase | What it measures | Usually large because |
|---|---|---|
queue |
From admission to start: waiting for a run slot (the service's concurrency limit, the plan's tenant cap, sched.max_inflight) and for a free processor |
Load above capacity; a low scaling.max_concurrency; a plan max_concurrency cap |
wait_runtime |
Waiting for the version's compiled code: a build (first request after a lazy deploy or an upgrade) or a load from disk into memory | Cold version, see First-request cost |
init |
Preparing the isolate or WASM instance for this task | Deno without a warm isolate |
exec |
Your handler running | Your code, its fetch calls and storage queries |
21ci stats <svc> (or GET /v1/services/{s}/stats) shows p50/p95/p99 per phase over 5 minutes, 1 hour or
24 hours, without Prometheus.
Deno: the isolate is the cost¶
By default each Deno request runs in a fresh V8 isolate, so no state leaks between requests. Creating one costs CPU
on every request (read the histogram rt_deno_isolate_create_seconds on your hardware). The pre-created pool
(deno.isolate_pool_per_p) only moves that cost off the request path: the CPU is still spent, so the per-core
throughput of a trivial service is bounded by it in the default configuration.
Warm reuse removes the cost: a cleanly finished isolate serves the next request of the same service version. The
_count of rt_deno_isolate_create_seconds then stays small compared with the request count. See
Warm reuse for what that changes for your code.
WASM: instantiation is cheap¶
A WASM component is compiled once, ahead of time, to native code, and each request gets an instance from a pre-sized pooling allocator. That is cheap, and it does not grow with the size of your handler. For WASM services the handler itself, its start code (large for components that embed a language runtime) and its host calls dominate.
Profiles¶
A profile is a bundle of defaults. node.default_profile picks one for the node; a service can pick its own with
profile (max_density, balanced, min_latency).
| Profile | Pick it when |
|---|---|
max_density |
Many mostly idle services per node; memory matters more than tail latency |
balanced (default) |
General purpose |
min_latency |
Few hot services; you trade memory for latency and accept warm state in isolates |
The knobs that differ (from config/schema.json):
| Key | Scope | max_density |
balanced |
min_latency |
|---|---|---|---|---|
sched.spin_us |
node | 0 | 10 | 50 |
sched.time_slice |
node | 20 ms | 10 ms | 5 ms |
sched.pin_threads |
node | false | false | true |
sched.handoff_on_hog |
node | false | false | true |
sched.queue_timeout |
service | 10 s | 5 s | 1 s |
sched.warm_share |
node | 0.05 | 0.15 | 0.3 |
sysmon.max_tick |
node | 20 ms | 10 ms | 2 ms |
rebalance.interval |
node | 250 ms | 100 ms | 50 ms |
memory.high / memory.critical |
node | 0.90 / 0.97 | 0.80 / 0.92 | 0.70 / 0.88 |
memory.purge_min_interval |
node | 5 s | 30 s | 120 s |
cache.t1_share |
node | 0.10 | 0.20 | 0.35 |
cache.idle_ttl |
service | 60 s | 600 s | 3,600 s |
cache.preload |
service | false | false | true |
deno.max_bound_per_m |
node | 128 | 64 | 32 |
deno.isolate_pool_per_p |
node | 0 | 1 | 4 |
deno.reuse_isolates |
service | false | false | true |
wasm.linear_memory_keep_resident (MB) |
node | 0 | 1 | 8 |
wasm.reuse_instances |
service | false | false | true |
wasm.max_unused_warm_slots |
node | 0 | 100 | 1000 |
build.compile |
service | lazy | lazy | eager |
The autoscaler has four more profile constants with no config key: the per-service concurrency ceiling
(256 / 1,024 / 1,024, replaced by an explicit scaling.max_instances or the plan's max_instances), the start and minimum concurrency limit
(4 and 1 / 16 and 2 / 32 and 4), the latency tolerance before it shrinks a service's limit (300 % / 200 % / 150 % of
its baseline), and the warm-instance headroom and maximum per processor (100 % and 1 / 125 % and 4 / 150 % and 16).
Explicit keys in rt.toml beat every profile
A profile fills only keys you left unset. The files in config/defconfigs/ (and make defconfig) state every
key, so with such a file a service's profile and a runtime switch of default_profile no longer change those
keys: the file pins them. Keep only the keys you mean to pin: make savedefconfig CONFIG=rt.toml saves just the
non-default values (to .config). GET /v1/services/{s} shows each effective value with its source
(service, config, profile:<name>, plan, default).
Warm reuse¶
Three knobs keep work warm between requests:
| Key | Scope | What it keeps |
|---|---|---|
deno.isolate_pool_per_p |
node | N blank isolates per processor, built between tasks and when idle. Single-use: they hold only the base snapshot, no tenant state. Dropped under memory pressure |
deno.reuse_isolates |
service | An isolate that finished a request cleanly, with your evaluated module, for the next request of the same service version on the same machine |
wasm.reuse_instances |
service | A WASM instance that finished cleanly, with its linear memory, for the next request of the same service version on the same processor |
Turn reuse on for the whole node in rt.toml, or per service in its deno / wasm block:
curl -sS -X PATCH -H "Authorization: Bearer $RT_KEY" -H 'Content-Type: application/json' \
-d '{"deno": {"reuse_isolates": true}}' "$RT_API/v1/services/svc_01J…"
curl -sS -X PATCH -H "Authorization: Bearer $RT_KEY" -H 'Content-Type: application/json' \
-d '{"wasm": {"reuse_instances": true}}' "$RT_API/v1/services/svc_01J…"
A service's own setting wins over the node's, which wins over the profile.
What reuse changes for your code¶
Reuse never crosses tenants, services or versions, even for identical code. Within one service version it changes semantics:
- Module state persists. Top-level code runs once per isolate, and globals survive between requests. Data one request leaves in a global is visible to the next request, possibly from another user. Do not keep per-request or per-user data in module scope.
- It is a cache, not storage. Requests land on different isolates, and any isolate may be discarded at any time.
Keep state in
std:kv,std:sqlorstd:blob. - Only clean finishes are kept. An isolate is reused only if the handler, the response body and every
waitUntilsucceeded, no timer or pending operation is left, the task was not killed, and its memory stayed within ¾ ofmemory_mband did not grow more than max(16 MiB, 50 %) beyond what it held after its first request. It is replaced afterdeno.reuse_max_uses(1,000) requests and disposed afterdeno.reuse_idle_ttl(60 s) idle. - Env and secrets are read per request. If your code read the environment and it changed since (a rotated secret), the isolate is discarded.
- WASM: linear memory (statics, allocator state, caches) persists the same way. An instance is dropped after a trap, kill or timeout, after 1,000 requests, and when the service's environment changed.
- Memory: kept isolates and instances hold memory (
rt_mem_committed_bytes{category="deno_heap"}and{category="wasm_reuse"}). They are dropped first when memory pressure reachesHigh.
How many are kept: the autoscaler publishes a warm target per service from recent demand, capped by
sched.warm_share of the memory budget. Without a target, Deno keeps up to deno.reuse_max_idle (4) per service
version and machine, deno.reuse_max_idle_total (16) per machine overall; WASM up to wasm.reuse_max_idle (8) per
service and processor.
First-request cost¶
A version moves through cache tiers: source (T3) is compiled into native code or a V8 code cache on disk (T2), which
is loaded into memory (T1). A request that finds its version cold waits in wait_runtime. See
Caching for the mechanism.
| Situation | What helps |
|---|---|
| First request after a deploy (lazy build) | Deploy with eager compile: the build runs at upload and the version goes live only when it is ready. Per upload ?compile=eager (21ci push … --wait), per service or node build.compile = "eager" (on in min_latency) |
| First request after a restart | Nothing to do for most services: compiled code on disk survives restarts, so this is a load, not a build. With cache.preload (on in min_latency) the node loads the cache.preload_top_n (100) most used versions at startup |
| First request after an idle period | The in-memory entry was evicted after cache.idle_ttl (600 s in balanced). Raise it per profile, or give the cache more room with cache.t1_share |
| First request after an upgrade | A new engine fingerprint invalidates compiled code; versions rebuild lazily. Rebuild hot ones ahead of traffic: 21ci rebuild <svc> <N> |
A Deno build also fetches new npm/jsr dependencies. They are cached on the node, stored with the version's artifacts
and pinned by hash in its rt.lock, so a rebuild after an upgrade does not depend on the CDN.
Queueing and timeouts¶
The node does not answer 429 because capacity is busy: requests over capacity wait in a per-service FIFO queue,
fair across tenants. A request runs when it gets a run slot, which needs room in three limits:
- the service's adaptive concurrency limit: it starts at the profile's start value, grows while requests wait
and latency holds, and shrinks on latency regressions, memory pressure or saturated processors. An explicit
scaling.max_instanceson the service (at most the plan'smax_instances) is a hard cap, andscaling.min_instancesa floor; - the tenant's plan
max_concurrency(in-flight tasks across all its services; 0 = no cap); - the node's
sched.max_inflight(10,000).
Waiting is bounded twice:
- Room to wait:
sched.max_queuedtasks (0 = 4 ×max_inflight) andsched.max_queued_mbbytes (0 = 1/16 of the memory budget) node-wide; one service and one tenant may each hold half. A service'sscaling.max_queuedlowers its share. Over it:503 overloaded. - Time to wait:
sched.queue_timeout(1 s / 5 s / 10 s by profile). A request that does not start in time ends with503 queue_timeout. The same timeout covers waiting for code and for memory.
Both carry a Retry-After computed from the recent completion rate (1 to 60 s, with jitter). A client that
disconnects while waiting leaves the queue at once.
The trade-off: a short queue_timeout fails fast and keeps tail latency bounded, but turns bursts into 503s; a long
one absorbs bursts and lets latency grow. For interactive APIs behind clients that retry, min_latency's 1 s is a
reasonable start; for webhooks and batch work, 5 to 10 s.
Watch rt_admission_queued, rt_admission_queue_wait_seconds and, with per-service metrics on,
rt_service_concurrency_limit and rt_service_queued (Metrics).
Memory budget and pressure¶
All memory that scales with load is reserved against one budget, memory.budget_mb (0 = 85 % of the container's
memory limit). Two watermarks, as fractions of the budget, drive a pressure ladder:
| Level | When | What the node does |
|---|---|---|
| Normal | below memory.high |
Idle entries expire after cache.idle_ttl |
| High | at memory.high (0.80 balanced) |
Evicts in-memory code down to high − 10 %, drops idle reused WASM instances and the pre-created isolate pools, halves warm targets, cuts services' concurrency limits by 25 % |
| Critical | at memory.critical (0.92 balanced) |
All of the above, halves concurrency limits, warm targets go to zero, the node enters shed mode (requests for versions not already in memory get 503 shed), new memory reservations wait (bounded by queue_timeout). It leaves shed after 5 s below high |
rt_mem_pressure_level (0, 1, 2) and GET /v1/node/status show the level. Lower watermarks (min_latency) react
earlier and keep headroom for latency; higher ones (max_density) use more of the budget before shedding.
WASM pool size
The WASM pooling allocator is sized at startup: slots = (memory budget ÷ 4) ÷ the largest
limits_max.memory_mb of any plan (at least 128 MB), at most 10,000. Every running or idle WASM instance holds a
slot, and rt_wasm_pool_slots shows the count. On a 2 GiB container that is (1,740 MB ÷ 4) ÷ 128 MB = 3 slots.
If WASM services keep many requests in flight on a small node, give it more memory or keep plan memory maxima
tight.
Processors per runtime kind¶
The scheduler runs a fixed set of processors (sched.processors, 0 = CPU quota or cores), each serving one runtime
kind. sched.kinds.deno and sched.kinds.wasm set a min and max per kind (max = 0: processors minus the other
kinds' minimums). Every rebalance.interval, a processor moves to the busier kind when that kind's load per
processor exceeds rebalance.up_threshold (2.0) and the other kind has an idle processor above its minimum, at most
once per rebalance.cooldown (1 s). A kind with no live services has an effective minimum of 0.
Pin the split only when you know your mix, for example to reserve processors for a latency-sensitive WASM fleet:
Builds run on their own processors (build.threads, default 1) at low priority, so compiling a large component does
not take CPU from running services.
Deno code cannot be preempted in the middle of JavaScript. A task that runs past sched.time_slice is a hog: the
monitor moves the other work queued on its processor elsewhere, and with sched.handoff_on_hog (on in min_latency)
hands the processor to a spare thread while the hog finishes. WASM tasks yield at every slice.
Storage: the blocking pool¶
KV, SQL and blob calls run on a separate thread pool, blocking.threads (0 = max(4, cores)), with per-tenant FIFO
queues served round-robin. Under contention one tenant may occupy floor(blocking_share × threads) threads, at least
one (blocking_share is a plan key, default 0.25), so a tenant's heavy SQL cannot starve the others' KV reads. The
pool is work-conserving: while threads are idle, a tenant may use more than its share, always leaving one thread free
for an arriving tenant.
A node with few, storage-heavy tenants may need a higher blocking_share in the plan or more blocking.threads; watch
the service's exec phase and the pool's queueing before changing either.
Other storage knobs, all restart-only:
| Key | Default | Effect |
|---|---|---|
storage.readers_per_db |
4 | Read-only connections per service database, opened under concurrent reads, so reads do not queue behind writes |
storage.max_open_dbs |
256 | All service-database connections together; the least recently used idle service is closed when it is reached |
storage.sql_cache_mb |
512 | SQLite page cache of all connections together, split evenly |
storage.sql_budget_ms |
1,000 | Per-statement time budget; over it the statement fails with sql_timeout |
Host-side work is charged: SQL thread time counts against the task's CPU limit.
Live tuning¶
Two ways to change performance knobs without a restart:
SIGHUPre-readsrt.tomland applies the live keys:sched.spin_us,sched.time_slice,sched.handoff_on_hog,sched.max_inflight,sched.max_queued,sched.max_queued_mb,sched.queue_timeout,sysmon.max_tick,rebalance.interval,deno.max_bound_per_m,memory.high,memory.critical(and the[plans]). See Production deployment.PUT /v1/node/configswitchesdefault_profile(the profile's live keys apply at once, except keys yourrt.tomlsets explicitly) anddegradation_mode:shedserves only versions already in memory,drainstops taking work for good,normalreturns fromshed.
Pool sizes, processor counts, cache shares and sched.warm_share need a restart.
Benchmark your own node¶
Most "the runtime falls over at N requests per second" reports measure something else, and a number from a laptop, a debug build or a shared machine says nothing about capacity. Before you trust a number:
- Release build.
make release(target/release/rt) or the Docker image.make devis a debug build onconfig/defconfigs/dev.toml, which also capssched.max_inflightat 100, turns off the isolate pool and uses one blocking thread. - A plan without quotas. The built-in
freeplan allows 50 requests per second and 100,000 a day per tenant, and 10 tasks in flight: beyond those you measure429 rate_limitedor queueing, not the runtime. Use a plan withrps = 0andinvocations_per_day = 0(0 = unlimited) and a highmax_concurrency. - Enough connections. One load generator is one client IP: raise
ingress.max_conns_per_ip(256), and for burstsingress.backlogplus the kernel'ssomaxconn(kern.ipc.somaxconnon macOS, 128 by default). - Check the status codes, not only requests per second. A fast stream of 429s or 503s is not throughput.
- Warm up, then measure. The first requests build and load the code.
- Separate machines for the node and the load generator when you can; on one machine they compete for CPU.
The runtime repository has a ready setup in bench/: bench/rt.toml is a private node with an unlimited plan,
[ingress]
max_conns_per_ip = 4096
[plans.bench]
rps = 0
invocations_per_day = 0
max_concurrency = 100000
max_services = 100
bench/rt-warm.toml adds deno.reuse_isolates, deno.isolate_pool_per_p = 4 and wasm.reuse_instances, and
bench/bench.sh starts a node, deploys three services and runs oha against them:
make release && make fixtures
bench/bench.sh target/release/rt mine # bench/out/mine.txt
BENCH_CONFIG=bench/rt-warm.toml bench/bench.sh target/release/rt warm
For your own service, the same idea with oha directly:
While it runs, watch rt_admission_queue_wait_seconds, rt_mem_pressure_level,
rt_deno_isolate_create_seconds_count (how many isolates were created) and the server-timing phases of a few
requests. Resource requirements turns these measurements into per-service costs and a capacity plan.
See also¶
- Performance characteristics: what costs what, the model behind these knobs.
- Resource requirements: sizing a node and measuring per-service cost.
- Scheduler and Execution: the mechanisms behind these knobs.
- Configuration reference: every key with its profile defaults.
- Limits and quotas: per-task limits and plan quotas.
- Metrics: what to watch while tuning.