Request lifecycle¶
This page follows one HTTP request from the socket to the last byte of the response, then shows how the other triggers (schedule, queue, service call) join the same path. Each phase says what happens, which part of the node does it, why it sits where it does, and how a failure in it reaches the caller. For the big picture first, read Architecture.
The whole path¶
sequenceDiagram
autonumber
participant C as Client
participant I as Ingress and router
participant A as Admission
participant S as Scheduler (P and M)
participant K as Cache T1 T2 T3
participant E as Executor
participant H as Host APIs
C->>I: HTTP request
I->>I: route, plan, access, ingress rules, quotas
I->>I: header hygiene, build the task spec
I->>A: submit
alt slot free
A->>S: runnable
else busy
A-->>A: wait in the service FIFO queue
A->>S: slot granted, runnable
end
S->>K: lookup prepared artifact
K-->>S: ready, or pending then woken
S->>E: start on a machine
E->>E: acquire isolate or instance, load, run handler
E->>H: kv, sql, blob, fetch, call, log
H-->>E: results
E-->>I: response head, then body chunks
I-->>C: response with x-rt-task-id, x-rt-version, server-timing
E->>S: finish, release slots and memory
S->>I: usage and stats recorded
The same sequence in words, with the cost of each step in the server-timing header:
| Phase | Component | server-timing phase |
|---|---|---|
| 1 to 5: accept, route, checks, quotas, task spec | Ingress (rt-api, route table in rt-triggers) |
before queue starts |
| 6: admission and queueing | Scheduler admission (rt-sched) |
queue |
| 7: artifact lookup, build or load | Cache tiers and builder | wait_runtime |
| 8: placement and start | Scheduler processors and machines | queue (until a machine picks it) |
| 9: sandbox acquisition and module load | Executor | init |
| 10: handler and host calls | Executor and host APIs | exec |
| 11: response streaming | Executor and ingress | inside exec |
| 12: teardown and accounting | Scheduler and executor | after the response |
1. Accept and parse¶
The public listener accepts connections on dedicated I/O threads (io.threads). Limits that protect the node from
slow or abusive clients apply here, before any routing: a request head of at most 16 KB and 100 headers, a header
timeout, an idle timeout, a per-IP connection cap, and HTTP/2 stream caps with a rapid-reset guard. These are
enforced where the listener is built, not per request. A request that breaks them gets 431 or a closed connection,
and closures are counted in rt_ingress_conn_rejected_total.
Why here: connection-level attacks have to be stopped before they cost scheduler time. See Security model.
2. Route¶
The router looks up the request in an immutable route table held behind an atomic pointer (ArcSwap). First match
wins:
Hostequals a verified custom host: the path passes through unchanged.Hostis{slug}.{routing.base_domain}(when a base domain is set): the path passes through.- The path is
/s/{slug}/{rest}: the handler sees/{rest}andx-rt-base-pathcarries/s/{slug}.
On the public listener, a service with access: internal does not resolve. A missing route, an unknown slug, a service
without a live version and a service with triggers.http off all answer 404 not_found without saying which.
Why an atomic table: a deploy or a live flip builds a new table and swaps the pointer, so a request in flight
sees either the old or the new route, never a gap. If the table names a service the service table does not hold yet
(a control change is mid-apply), the request gets 503 route_stale with Retry-After: 1: it is a race, not overload.
3. Plan and access¶
The tenant's plan must allow the http trigger (403 trigger_not_allowed). The plan lookup is an in-memory cache, not
a database read. Then the service's access mode applies:
| Mode | Rule | Failure |
|---|---|---|
public |
Anyone | none |
key |
A tenant key of the same tenant with the invoke scope (and, if the key is limited to services, listing this one), or the root token |
401 unauthorized; repeated failures from one client are rate-limited (429 rate_limited) |
internal |
Only the internal listener (a Unix socket, peer UID checked) | 404 on the public listener |
The internal listener skips this phase and the next one: it is trusted.
4. Ingress rules and quotas¶
For the public listener the service's ingress block runs: allowed methods (405 method_not_allowed), allowed paths
(403 forbidden), ip_allow against the resolved client address (403 forbidden) and a per-service token bucket
(429 rate_limited). The client address is the TCP peer, or, when the peer is in ingress.trusted_proxies, the first
X-Forwarded-For hop that is not itself a trusted proxy.
Then the tenant's quotas: the rps token bucket and invocations_per_day (429 rate_limited or 429 quota_exceeded,
both with Retry-After).
Why before admission: these are limits someone chose. A refusal here costs almost nothing and never occupies a run slot or a queue place. Busy capacity, in contrast, is not an error and is handled by waiting (phase 6).
5. Header hygiene and the task spec¶
The headers the handler receives are the client's, minus:
- hop-by-hop headers and
Host; - every
x-rt-*header (only the runtime sets them; on the internal listenerx-rt-invokeris kept); Authorizationwhen it carries a runtime credential (anrtk_bearer, or the root token that passed anaccess: keycheck). Any otherAuthorizationheader is the handler's own and arrives unchanged.
The runtime then adds x-rt-trigger: http, x-rt-base-path (for /s/{slug} routes) and x-rt-client-ip, and a
traceparent header if the client sent none.
The ingress layer then builds the task spec: the service, the live version, the resolved limits (service
settings capped by the plan), the trace context and the request. A request whose Content-Length is over
limits.max_body_bytes is refused here with 413 too_large. A streamed body that grows past the limit later fails
inside the handler's read (request_too_large), because the body is passed to the guest as a stream, not buffered.
6. Admission¶
Scheduler::submit is the single entry for every trigger. It applies these checks in order:
| # | Check | Outcome |
|---|---|---|
| 0 | Call depth over 15 | 508 call_depth |
| 1 | Node in drain mode |
503 draining |
| 2 | Node in shed mode and the artifact is not in T1 |
503 shed |
| 3 | A run slot: the service's adaptive limit, the tenant's plan cap, the node's sched.max_inflight less the reserve for calls |
run now, or wait in the service's FIFO queue |
| 4 | Room to wait: the node's sched.max_queued and sched.max_queued_mb, with a service and a tenant each limited to half |
503 overloaded with Retry-After |
If no slot is free, the task waits in a per-service FIFO queue. When slots free up, services are served round-robin
within their tenant and tenants round-robin among themselves, one task per turn, so a tenant with many busy services
gets no more than a tenant with one. A task that waits longer than the service's queue timeout ends as
503 queue_timeout with Retry-After. If the client disconnects while its request waits, the place is freed at once;
a task that has already started keeps running.
Why queue instead of reject: a burst above the current concurrency limit is normal and not the client's fault. A
short wait is almost always better than an error that clients retry in lockstep. The wait is bounded by the profile
(short in min_latency, long in max_density) so a saturated node fails fast enough for callers to react. The
mechanism and the adaptive limit are in Scheduler and autoscaling.
7. Artifact lookup¶
When the task is admitted, the scheduler asks the cache for the prepared artifact of the service's live version:
- T1 hit (in memory): continue at once.
- Miss: the cache starts a load (from the disk cache T2 with an integrity check) or a build (from source T3), with
single-flight semantics: the first task triggers it, concurrent tasks for the same version wait on the same work. The
waiting task is parked as "waiting for runtime" and woken when the result is recorded. This waiting is
wait_runtime. - Build failure: the task ends
502 build_failed, and the live pointer reverts to the previous healthy version (auto-rollback).
A task whose memory reservation is refused (the node is near its budget) waits for memory, bounded by the same queue timeout. See Caching and Services, versions and builds.
8. Placement and start¶
A runnable task is placed on a processor (P) of the right kind (deno or wasm). An external submit (ingress,
schedules, queue) picks two random processors of the kind and pushes to the one with the shorter local run queue; if both
are full it goes to the kind's global queue. A child task spawned by a service call of the same kind goes to the
calling processor's runnext slot. Idle processors of the kind steal half of a busy one's queue. A machine (M, an OS
thread running a single-threaded async runtime) picks the task and starts it. From then on the task lives on that
machine: Deno isolates are thread-pinned, and in this release WASM tasks stay on the machine that started them too. The
scheduler hands the executor a memory reservation, the wall-clock deadline, the current time slice and the CPU budget.
9. Sandbox acquisition and module load¶
The executor gets a sandbox for the task:
- A kept isolate for this service version, if reuse is on, its environment is still current and its memory fits.
- Else a pre-created isolate from the processor's pool, or a fresh one from the base snapshot.
- The V8 heap limit is topped up through the memory governor in 8 MB steps as the task allocates, up to
limits.memory_mb; a refusal terminates the task withoom. - The module loader serves only from the prepared bundle. Anything else fails with
module_not_in_bundle: there is no network or disk access at run time. - The entry module is evaluated (already evaluated in a kept isolate) and the handler is called.
- An idle instance for this exact
(tenant, service, version, artifact, processor)built with the same environment: a pre-instantiated warm one, or, with reuse on, a used one. - Else a new store instantiated from the pre-linked component (
InstancePre), taking a slot from the pooling allocator, thenwasi:httphandleis called. - Linear memory may grow up to
limits.memory_mb.
Environment variables and secrets are fetched when the guest first reads them (Deno) or before the instance is created
(WASM); a failure to fetch them throws env_unavailable instead of blocking. This phase is init.
10. The handler and host calls¶
The handler runs under three limits at once: the wall-clock deadline, the CPU budget (judged on the thread's CPU time, so waiting on I/O does not consume it) and the memory limit. The scheduler's monitor thread enforces them from outside: it terminates a running isolate, or bumps the epoch to make a WASM instance yield or trap. A running isolate cannot be paused mid-slice; WASM yields at every slice.
Everything the handler does to the outside world is a host call:
| Call | Path |
|---|---|
kv, sql, blob |
The blocking pool (per-tenant fair queues), against the service's own files. SQL CPU counts against the task's CPU budget |
fetch, wasi:http outbound |
The egress guard: resolve once, validate every address, connect to the validated IP, apply size, time and quota limits |
queue enqueue |
The blocking pool, against queue.db, after the call-access and plan checks |
std:services / std:host/call |
An in-process child task: it goes through admission itself (calls are served first and cannot deadlock behind their parent), runs under wall = min(own, parent's remaining), and its CPU is counted to the child service |
platform.fetch |
The binding's checks, credential injection and the upstream request |
console.* and WASI stdout/stderr |
The log store, at most 1,000 lines per task, written in batches |
11. Response¶
The response is decided by the response head. For Deno, the handler runs inside the task's own future until the
head exists, so CPU is measured and billed; then the body streams through a bounded channel, which gives backpressure.
For WASM, a guest that finishes in that first run hands over a buffered body; otherwise the head is returned and the
same driver keeps running the guest, forwarding chunks through a bounded channel up to
limits.max_response_stream_bytes. The CPU limit still applies to the tail, and the tail's CPU is charged to the
tenant.
A failure after the head (an exception, a trap, a kill, the size limit) cannot change the status that was already
sent. The body is aborted instead, so the client sees a broken stream, not a clean 200.
The ingress layer adds x-rt-task-id, x-rt-version and server-timing to every response, including platform errors.
Platform errors are text/plain with the reason repeated in x-rt-reason; a response your handler returns never
carries x-rt-reason.
12. Teardown and accounting¶
When the task finishes, exactly one finisher releases its run slots, its memory reservation and its deadline entry, and delivers the outcome once. Then:
- Sandbox: with reuse on and a clean finish, the isolate or instance is kept for the next task of the same service version; otherwise it goes to a dispose queue that is drained between tasks and when the processor is idle, so disposal is not part of the response time.
- Usage: an invocation, the task's CPU milliseconds and (for egress) bytes are recorded against the tenant and
service, for quotas and for
21ci tenant usage. Outcomes that count: a normal reply, a guest error and a kill. - Stats, metrics, logs: latency percentiles per phase, outcome counters and admission state are updated; log lines are persisted in batches.
ctx.waitUntil: the isolate waits for registered promises, bounded by the wall limit, before it is disposed.
Other triggers¶
Schedules, queue batches and service calls skip phases 1 to 5 (there is no client, route or ingress rule) and enter at
phase 6 through the same Scheduler::submit:
| Trigger | Who builds the task | What differs |
|---|---|---|
| Interval and cron schedules | One timer wheel in the schedule engine, with a small jitter per schedule | A tick is skipped (and counted) while the previous run of the same schedule is still running, so runs never overlap. The plan's allowed_triggers is rechecked on every fire. Missed ticks are not backfilled after a restart. Default limits are 300 s wall and 10 s CPU |
| Queue | The queue engine leases a batch (up to queue.batch_size) per service under queue.max_concurrency |
The visibility timeout is the task's wall limit plus 10 s. 2xx (or Form B msg.ack()) acknowledges; anything else retries with exponential backoff and jitter up to queue.max_attempts, then the dead-letter queue |
| Service call | The calling task's host call | The child is a task of its own with depth + 1, placed next to the caller when kinds match, in the same tenant unless the plan allows cross-tenant calls, and admitted only if the target's call.allowed_callers (or, when empty, its environment) allows the caller |
| Internal listener | Same ingress code | Skips access checks and ingress rules; x-rt-invoker is passed to the handler |
Where each failure shows up¶
| Phase | Failure | Status and x-rt-reason |
|---|---|---|
| Accept | Head too large, too many connections, slow headers | 431, or the connection is closed |
| Route | No route, no live version, access: internal on the public listener |
404 not_found |
| Route | Control change mid-apply | 503 route_stale |
| Plan, access | Plan forbids http |
403 trigger_not_allowed |
| Plan, access | Missing or wrong key | 401 unauthorized (repeated failures 429 rate_limited) |
| Ingress | Method, path, IP | 405 method_not_allowed, 403 forbidden |
| Ingress, quotas | Rate or daily quota | 429 rate_limited, 429 quota_exceeded |
| Task spec | Body over the limit | 413 too_large |
| Admission | Draining, shedding | 503 draining, 503 shed |
| Admission | Queue full | 503 overloaded |
| Admission | Waited too long | 503 queue_timeout |
| Artifact | Build failed on first use | 502 build_failed |
| Handler | Threw, trapped, returned a non-Response |
500 guest_error |
| Handler | CPU, memory, wall limits | 502 cpu_limit, 502 oom, 504 timeout |
| Handler | Call chain too deep | 508 call_depth |
| Response | Failure after the head | Body aborted; status already sent |
Full descriptions and remedies: Error reasons. What a failure does to the sandbox, to in-flight work and to stored state: Failure behavior.
See also¶
- Architecture: the components this path crosses.
- Scheduler and autoscaling: admission, queues and processors in depth.
- Execution and isolation: the sandboxes and their limits.
- Caching: the tiers behind phase 7.
- Observability: reading
server-timing, logs, stats and metrics for a request.