Scheduler and autoscaling¶
Every request, schedule fire, queue batch, service call and build becomes a task that the node's scheduler
places on a CPU. This page explains how that scheduler works, why a busy service makes requests wait instead of
answering 429, and how per-service concurrency and warm capacity follow demand. It is written for developers who
want to understand latency and for operators who tune a node.
Short version
- Capacity never returns
429. A request over the current limit waits in its service's queue and runs when a slot frees up. Only when the wait runs out does it get503 queue_timeoutwithRetry-After. - Each service's concurrency limit is adaptive. Set
scaling.max_instancesonly if you need a hard cap. - Warm instances follow recent demand and scale to zero when a service goes idle, unless
scaling.min_instanceskeeps a floor. - The profile (
max_density,balanced,min_latency) decides how long requests may wait and how much the node keeps warm.
The model: tasks, processors and machines¶
The scheduler follows the design of Go's runtime scheduler (the "GMP" model). If you have read Go's proc.go, the
vocabulary will look familiar.
| Term | What it is | Go equivalent |
|---|---|---|
| Task | One unit of work: an invocation (HTTP, interval, cron, queue, call) or a build. Kinds deno, wasm, build |
goroutine (G) |
| Processor (P) | A run slot with a kind. A P only runs tasks of its kind and owns a local run queue | P (GOMAXPROCS) |
| Machine (M) | An OS thread that executes the tasks of the P it holds | M |
| sysmon | A monitor thread without a P: timeouts, limits, preemption, re-kinding, autoscaling, memory | sysmon |
flowchart TB
subgraph DENO["Deno processors"]
direction LR
P0["P0 + local queue"] --- M0["M0 (thread)"]
P1["P1 + local queue"] --- M1["M1 (thread)"]
end
subgraph WASM["WASM processors"]
direction LR
P2["P2 + local queue"] --- M2["M2 (thread)"]
end
GD["Global queue: deno"] --> DENO
GW["Global queue: wasm"] --> WASM
BP["Blocking pool: SQL, KV, blob files"]
IO["I/O threads: fetch, timers"]
SYS["sysmon"] -. timeouts and slices .-> DENO
SYS -.-> WASM
M0 -- blocking op --> BP
M2 -- outbound HTTP --> IO
Processors per executor kind¶
sched.processors sets the number of Ps (0, the default, follows the container's CPU quota or the core count).
The Ps are split between the two executor kinds within sched.kinds.deno.{min,max} and sched.kinds.wasm.{min,max}
(defaults: at least 1 each, max = 0 means "all Ps minus the other kind's minimum"). Builds get their own
low-priority Ps (build.threads, at most 2) outside sched.processors, so compiling a service never takes CPU from
requests.
The split is elastic. Every rebalance.interval, sysmon compares the pressure of each kind (runnable + running +
memory-waiting tasks per P). When one kind is over rebalance.up_threshold (2.0) and the other has an idle P above its
minimum, that P switches kind. A P re-kinds at most once per rebalance.cooldown (1 s), and a P whose thread still
holds started Deno tasks is never re-kinded. A kind with no live services can shrink to zero.
Machines and thread pinning¶
A Machine is an OS thread. It never blocks on a system call: SQL, KV and blob file work goes to the blocking pool, outbound HTTP and timers to the I/O threads. While a task waits for one of those, the P runs other tasks.
V8 isolates are bound to the thread that created them. Once a Deno task has started, it is bound to its M and
resumes only there; a WASM task can resume on any WASM P. One M holds at most deno.max_bound_per_m started Deno tasks
(64 in balanced); at that cap it only resumes the ones it has, and new tasks stay in its local queue where other
Ps can steal them. With sched.pin_threads (on in min_latency) each M is pinned to a core.
Run queues and work stealing¶
Each P has a local run queue (256 entries) plus a one-task runnext slot; each kind has a global run queue.
- New work (a request, a schedule fire, a queue batch) goes to the shorter local queue of two randomly chosen Ps of its kind, or to the global queue when both are full.
- A service call of the same kind goes into the caller's
runnext, so the child starts next on the same P with warm caches. - An idle P looks for work in this order: resumable Deno tasks of its M,
runnext, its local queue, a batch from the global queue, then it steals half of another P's local queue of the same kind. A WASM P never steals Deno work and the reverse. Every 61st pick it checks the global queue first, so nothing starves there. - With nothing to do, an M spins for
sched.spin_us(0 / 10 / 50 µs by profile) and then parks. An idle node uses almost no CPU.
Time slices, preemption and limits¶
A task may run for one time slice (sched.time_slice: 20 / 10 / 5 ms) before the scheduler intervenes. What
"intervene" means depends on the executor:
| WASM | Deno | |
|---|---|---|
| Time slice used up | Real preemption: the instance yields at the next epoch check and goes to the back of the local queue | V8 cannot yield mid-JavaScript. Hog handoff: sysmon moves the hogging P's queued tasks to the global queue so other Ps start them |
sched.handoff_on_hog (on in min_latency) |
sysmon gives the hogging P to a spare M, so the queue keeps moving while the hog finishes on its old thread | |
CPU limit (limits.cpu_ms) |
Trap at the next epoch check: 502 cpu_limit |
The isolate is terminated: 502 cpu_limit |
Wall limit (limits.wall_ms) |
Cancelled: 504 timeout |
Terminated or dropped: 504 timeout |
Memory limit (limits.memory_mb) |
Growth refused, trap: 502 oom |
Heap or ArrayBuffer limit: 502 oom (a resizable buffer throws RangeError) |
CPU time is measured per task on its own thread, and time a blocking-pool job spends on the task's behalf is added to it. The execution model page covers limits per trigger.
sysmon¶
sysmon is a dedicated thread with no P. Its tick starts at 20 µs, doubles after 1 ms without anything to do, and stops
growing at sysmon.max_tick (20 / 10 / 2 ms). Any action resets it to 20 µs.
| Duty | How often | What it does |
|---|---|---|
| Queue timeouts | every tick | Tasks that did not start within their queue_timeout end with 503 queue_timeout |
| Limits | every tick | CPU and wall checks of running tasks |
| Hog detection | every tick | A task past its time slice: WASM epoch bump, Deno hog handoff |
| Lost-wakeup guard | every tick | Global queue has work, a P is idle, nobody spins: wake a P |
| Re-kinding | rebalance.interval (250 / 100 / 50 ms) |
Move idle Ps to the busier kind |
| Autoscaler | rebalance.interval |
Adapt concurrency limits, publish warm targets, refresh the Retry-After estimate (on its own thread) |
| Memory | memory.check_interval (250 ms) |
Wake memory waiters; at Critical pressure switch the node to shed mode, and back once below High |
| Stats | 1 s | Update gauges and run a self-check |
Blocking pool for service data¶
SQL statements, KV operations and blob file I/O run on the blocking pool, not on a P. The pool has blocking.threads
threads (0 = max(4, cores)) and is fair between tenants:
- each tenant has its own FIFO lane, and lanes are served round-robin;
- under contention one tenant occupies at most
floor(plan.blocking_share × threads)threads (blocking_shareis 0.25 by default, at least 1 thread); - the pool is work-conserving: when threads are idle, a busy tenant may borrow more, always leaving one thread free for a tenant that arrives;
- the thread CPU time of each job is charged to the calling task, and a single SQL statement is aborted after
storage.sql_budget_ms(1,000 ms) withsql_timeout.
So a tenant that runs a heavy cross join slows down its own queries, not another tenant's KV reads.
blocking_share is a plan setting.
Admission: queueing instead of 429¶
When a request arrives, admission decides whether it runs now, waits, or is refused. A busy service does not get
429: capacity problems turn into waiting, and only a wait that runs out becomes an error.
| # | Check | Outcome |
|---|---|---|
| 0 | Call depth over 15 | 508 call_depth |
| 1 | Node in drain mode |
503 draining |
| 2 | Node in shed mode and this version is not in RAM (T1) |
503 shed + Retry-After |
| 3 | A run slot is free: the service's adaptive limit (at most scaling.max_instances), the tenant's plan cap (max_concurrency) and the node's sched.max_inflight (less a ⅛ reserve) |
Run now |
| 4 | Otherwise: room in the queues (node sched.max_queued / sched.max_queued_mb; one service and one tenant may each hold half of either; scaling.max_queued can only lower the service's share) |
Wait in the service's FIFO queue, else 503 overloaded + Retry-After |
Then, while it waits:
- A waiting request that gets no slot within its queue timeout ends with
503 queue_timeout+Retry-After. The timeout comes from the service's profile:min_latency1 s,balanced5 s,max_density10 s. An explicitsched.queue_timeoutinrt.tomloverrides it node-wide. The same budget covers waiting for a P, for the artifact to load or compile, and for memory. - A client that disconnects while waiting is taken out of the queue at once. A request that has started runs on.
Retry-Afteris the time the current waiters need to drain at the recent completion rate, between 1 and 60 s, plus up to 50 % random jitter so refused clients do not come back in lockstep.
The full flow for a request that has to wait:
sequenceDiagram
autonumber
participant C as Client
participant I as Ingress :8080
participant A as Admission
participant Q as Service queue
participant P as Processor and executor
C->>I: GET /s/hello/
I->>I: route, access, ingress rules, plan quotas
I->>A: submit task
A->>A: service limit reached
A->>Q: wait (FIFO)
alt a slot frees within queue_timeout
P-->>A: earlier task finished
A->>Q: grant the oldest waiter
Q->>P: place on a P of its kind
P-->>C: 200, server-timing shows the wait
else timeout passes
A-->>C: 503 queue_timeout + Retry-After
end
The time a request spent waiting shows up in the queue phase of its server-timing response header.
Where 429 still comes from¶
429 is reserved for quotas the operator configured in the tenant's plan: the tenant-wide rps token bucket,
invocations_per_day, and a service's own ingress.rate_limit. These are checked at ingress, before admission. The old
tenant_concurrency and service_concurrency reasons are no longer produced. See
error reasons.
Slot classes and fairness¶
- Requests (HTTP, interval, cron, queue) are held by all three limits and may use
sched.max_inflightminus ⅛. - Service-to-service calls run on behalf of a request that already holds slots. They may use the whole
max_inflight(the ⅛ reserve is for them and for builds), count against the tenant cap without waiting for it, and are served before the target's waiting requests. Nested calls therefore cannot deadlock behind their parents. One task has at most 16 calls in flight; further calls wait within its deadline. - Builds count in
max_inflightonly and never wait. - Fairness: FIFO within a service. When slots free up, tenants take turns, then the services of a tenant, one task per turn.
- Queue consumption: a service consumes its durable queue with at most
queue.max_concurrency(4) tasks at once, within its concurrency limit.
Plan caps are per tenant
The plan's max_concurrency (10 on the built-in free plan) bounds all running tasks of a tenant. Requests over it
wait like any other. A plan change applies with the next request.
Instances and autoscaling¶
An instance is one concurrently running copy of a service on the node: a Deno isolate or a WASM instance. A service's concurrency limit is therefore its instance target, and the autoscaler moves it between two bounds you can set per service. Scaling is per node; there is no multi-node scaling.
| Setting | Meaning | Default | Upper bound |
|---|---|---|---|
scaling.min_instances |
Instances kept warm. The service never scales to zero, and the concurrency limit never falls below it | 0 (scale to zero) | the plan's max_instances |
scaling.max_instances |
Most instances running at once. Requests beyond it wait in the service's queue; they are not refused | the plan's max_instances, else the profile ceiling (256 / 1,024 / 1,024) |
the plan's max_instances |
The plan's max_instances is a per-service limit the operator sets (0 = the node default; the example plans in the
configuration use free 4, pro 32, business 256). A larger value on a service is clamped to it, and the service's
effective block shows plan as the source. min_instances above max_instances is 422 validation_failed. The
tenant-wide plan cap max_concurrency bounds all of a tenant's services together on top of that.
A PATCH replaces the whole block you send (21ci service update merges your fields into the current block for you).
max_queued bounds this service's queue; over it new requests get 503 overloaded. scaling.max_concurrency is the
old name of max_instances: the API and 21ci --set still accept it, and it is never returned.
How the limit moves¶
Once per rebalance.interval (100 ms) the autoscaler, on its own thread, adjusts each service's limit. It works like
TCP congestion control (slow start, then additive increase, multiplicative decrease), guarded by latency and by the
arrival-rate trend. Scale-up is fast (one step per tick, doubling during slow start); scale-down is slow (one ⅛ step
per 10 s cooldown).
| Signal (first match wins) | New limit | Metric cause |
|---|---|---|
Node memory pressure Critical |
halve | memory |
Memory pressure High |
−25 % | memory |
| Mean latency above the service's baseline × tolerance | −25 % | latency |
| The kind's Ps are saturated (p90 run-queue wait > 2 × time slice) and the service runs more than its start limit | −25 % | cpu |
| Requests waited and the limit was reached (queue length) | grow: double below the slow-start threshold, then +√limit, never more than the queue asked for. When requests waited longer than a quarter of the queue timeout (queue wait), double regardless |
queue |
| The arrival rate rose fast and the limit is at least 75 % used, before a queue forms (trend) | grow like the queue rule, to at most rate × latency × 1.25 once the latency is known |
trend |
| No queue, less than half the limit in use, last change older than the cooldown | decay by ⅛ toward the start limit | idle |
- Bounds. The limit stays within
[min, max]:maxismax_instances(else the plan's limit or the profile ceiling) and the tenant cap,minis the profile floor raised tomin_instances. The start value is raised tomin_instancesand capped bymax_instances. A service update that moves the bounds pulls the limit inside them at its next request (causebounds). - Arrival rate is the requests that reached admission per second (queued and refused ones count, builds do not), kept as a fast (0.5 s) and a slow (10 s) moving average. It counts as rising when the fast one is at least 1.5 × the slow one and at least 5 req/s above it, so the trend rule grows the limit before requests start to queue.
- Latency is measured from slot grant to finish, without artifact waits. The baseline is a moving average that follows improvements fast and regressions slowly, so a lasting change (a slower upstream API) becomes the new normal instead of throttling the service forever. A tick needs at least 3 completions and more than 2 ms above the baseline to count as unhealthy.
- Decreases cut what the service actually uses (at least its start limit), so a service that did not cause a node-wide signal settles near its start value, not at its floor. A CPU cut only hits services running more than their start limit.
| Profile | Start | Min | Ceiling | Latency tolerance |
|---|---|---|---|---|
max_density |
4 | 1 | 256 | 300 % |
balanced |
16 | 2 | 1,024 | 200 % |
min_latency |
32 | 4 | 1,024 | 150 % |
These are profile constants, not config keys. To bound a service yourself, set min_instances and max_instances.
What you can see¶
status.admission of GET /v1/services/{s} (also admission in …/stats, 21ci service stats, and services[] of
GET /v1/node/scheduler) is the service's admission view, null while the service has not run recently:
| Field | Meaning |
|---|---|
limit, min, max |
The adaptive concurrency limit and its current bounds |
instances |
Instances running now |
min_instances, max_instances |
The effective bounds |
target_instances |
The warm target the autoscaler wants kept |
running, queued, max_queued, queue_wait_ms, queue_timeouts |
What runs, what waits, the smoothed wait of queued requests, 503 queue_timeout count |
rps, demand, latency_baseline_ms, warm_per_proc |
Smoothed arrival rate, decaying peak of running tasks, the latency baseline, warm instances per processor |
last_change |
{reason, from, to, ago_ms} with the reasons queue, trend, latency, memory, cpu, idle, bounds |
Metrics: rt_autoscale_adjustments_total{direction,cause} and, for labelled services, rt_service_min_instances,
rt_service_max_instances, rt_service_target_instances and rt_service_arrival_rate.
Warm targets and scale to zero¶
From the same tick, the autoscaler publishes a warm target per service and version: how many ready instances each P of its kind should keep, so the next request does not pay for creating one.
- Demand is a decaying peak of the service's running tasks (half-life 15 s).
- Per P:
ceil(demand × headroom / Ps of the kind), capped by the profile (headroom 100 % / 125 % / 150 %, at most 1 / 4 / 16 per P formax_density/balanced/min_latency). Below half an instance of demand there is no target. - Node cap: all warm instances together stay within
sched.warm_shareof the memory budget (5 % / 15 % / 30 %); the services with the lowest demand are trimmed first. Targets are halved atHighmemory pressure and dropped atCritical. - Scale to zero: once a service has been idle longer than its idle TTL, its target is 0 and the executors drop what they kept.
The executors converge on these targets between tasks: WASM pre-instantiates fresh instances (or keeps reusable ones
when wasm.reuse_instances is on), Deno keeps reusable isolates when deno.reuse_isolates is on. Warm instances are
kept per service and are never shared between services or tenants, even when two services deploy identical code. The
caching page explains how this differs from the code cache.
The min_instances floor¶
min_instances is a floor under all of the above: each P of the service's kind keeps
ceil(min_instances / Ps of the kind) ready instances, so at least min_instances in total. The floor is not cut by
the profile cap, the idle TTL, the memory share or High pressure; only Critical memory pressure releases it. A
service with a floor also keeps its admission state while idle.
The floor does not wait for traffic. The node pins every routable service with min_instances > 0 when it starts,
when a version goes live, and when the service's scaling, profile or plan changes: the live version is loaded (or
built) into memory off the request path and the executors pre-warm it. A service that loses its floor, is deleted or
stops being routable is released. Warm instances cost memory whether or not requests come, so keep floors for
latency-sensitive services.
Degradation modes¶
The node is always in one of three modes, shown as mode in GET /v1/node/status:
| Mode | Effect | How it starts and ends |
|---|---|---|
normal |
Everything is admitted as described above | default |
shed |
Requests whose version is in RAM (T1) are served; anything that would need a load or compile gets 503 shed + Retry-After |
automatically at Critical memory pressure (back to normal once below High), or set by the operator |
drain |
Every new task gets 503 draining; /readyz turns 503; running tasks finish |
POST /v1/node/drain, degradation_mode: "drain", or shutdown. One-way: restart to resume |
A mode the operator sets is kept: automatic shedding never overrides it.
How profiles change scheduling¶
A profile fills every knob you did not set explicitly. The node's node.default_profile covers node-wide knobs; a
service's profile covers its own (queue timeout, concurrency constants, warm headroom, reuse).
| Knob | max_density |
balanced |
min_latency |
|---|---|---|---|
sched.queue_timeout |
10 s | 5 s | 1 s |
sched.time_slice |
20 ms | 10 ms | 5 ms |
sched.spin_us |
0 | 10 | 50 |
sched.pin_threads, sched.handoff_on_hog |
off | off | on |
sysmon.max_tick |
20 ms | 10 ms | 2 ms |
rebalance.interval |
250 ms | 100 ms | 50 ms |
deno.max_bound_per_m |
128 | 64 | 32 |
sched.warm_share |
5 % | 15 % | 30 % |
| Concurrency start / ceiling | 4 / 256 | 16 / 1,024 | 32 / 1,024 |
| Warm headroom / max per P | 100 % / 1 | 125 % / 4 | 150 % / 16 |
max_density packs more services onto a node and lets requests wait longer; min_latency spends CPU (spinning,
pinned threads) and memory (warm instances) to cut tail latency. The
performance guide shows how to choose, and
profiles at a glance lists every profile-controlled key.
Some scheduler keys can change without a restart: sched.max_inflight, sched.max_queued, sched.max_queued_mb,
sched.queue_timeout, sched.spin_us, sched.time_slice, sched.handoff_on_hog, sysmon.max_tick,
rebalance.interval and deno.max_bound_per_m are re-read on SIGHUP (see
applying changes). sched.processors, sched.warm_share and thread counts
need a restart.
Observing the scheduler¶
GET /v1/node/scheduler (operator token; 21ci node scheduler --json prints it raw) returns mode, procs (per P:
kind, status, lrq_len, busy_ns, steals, preemptions, re-kinds), machines (per M: held P, bound_count,
spinning, parked), the global queue lengths grq, idle, spinning, waiting_runtime, waiting_memory,
inflight and task counters, plus the node-wide admission bounds and services: one admission view per service
that has run recently.
A service's own admission view is status.admission in GET /v1/services/{s} and admission in
GET /v1/services/{s}/stats (scope logs:read): the adaptive limit with its min and max, running, queued,
max_queued, the smoothed queue_wait_ms, queue_timeouts, the autoscaler's demand, latency_baseline_ms,
warm_per_proc and idle_ms. It is null while the service is idle. This is the first place to look when a
service answers 503 queue_timeout.
For a running trace, start the node with RT_SCHEDTRACE=<ms>. It prints one line per interval to stderr:
SCHED 1000ms: procs=4 deno=3 wasm=1 idle=1 spinning=0 grq=[deno:12 wasm:0 build:1] lrq=[3 0 5 | 0] bound=[2 0 1] waitrt=4 waitmem=0
waitrt counts tasks waiting for their artifact to load or compile, waitmem tasks waiting for memory.
Prometheus series on the metrics listener (the metrics reference has the full list):
| Series | Use it to see |
|---|---|
rt_admission_running, rt_admission_queued, rt_admission_queued_bytes |
Slots in use and requests waiting, node-wide |
rt_admission_queue_wait_seconds |
How long queued requests waited (histogram) |
rt_admission_rejects_total{reason} |
queue_timeout, overloaded, shed, draining, … |
rt_autoscale_adjustments_total{direction,cause} |
Why limits went up or down |
rt_warm_target_instances{kind} |
Warm instances the autoscaler asks for |
rt_sched_processors{kind}, rt_sched_rekinds_total{from,to} |
The P split and how often it moves |
rt_sched_grq_len{kind}, rt_sched_lrq_len{p}, rt_sched_steals_total{kind} |
Run-queue depth and stealing |
rt_sched_preemptions_total{kind}, rt_sched_hog_handoffs_total |
Long-running tasks |
rt_task_phase_seconds{kind,phase} |
Time in queue, wait_runtime, init, exec |
rt_service_concurrency_limit, rt_service_running, rt_service_queued, rt_service_queue_wait_seconds |
Per service, only with metrics.per_service = true |
Under the hood
- Per-service and per-tenant admission state lives in sharded maps of atomics: a request whose slots are free takes them with three compare-and-swaps. The queues sit behind one mutex that is only taken when something waits.
- A waiting request is charged 4 KiB plus its known body size against
sched.max_queued_mb.sched.max_queued = 0means 4 ×max_inflight;sched.max_queued_mb = 0means 1/16 of the memory budget. - Local run queues are lock-free rings. Overflow moves half of the queue plus the new task to the global queue.
runnextis never stolen. - Before a task starts, it reserves memory (Deno:
deno.isolate_base_mb, WASM: the declared initial memory). If the memory governor refuses, the task waits asWaitingMemory(FIFO) until memory frees up, within its queue timeout.
See also¶
- Execution and isolation: isolates, instances and limits per trigger.
- Caching: why the first request after a deploy can be slower.
- Configuration:
[sched],[rebalance],[blocking], servicescaling. - Performance tuning and running in production.