What it is
A runner is a host that runs your fleets. It takes work fromagentsfleetd, runs it inside a sandbox, and reports the result back.
You enrol every runner from the dashboard. The host holds one token and nothing else. It never holds a database credential, a cache credential, or vault access.
Why it exists
Runners run fleet code separately from the control plane. Their tokens reach assigned work without direct access to the control plane’s data stores. Runners are replaceable. When one stops answering, unfinished work can be assigned again after its lease expires. External actions must be safe to repeat if the first runner acted before losing contact.How it behaves
Enrolling a host
Open Admin → Runners → Add runner and fill in the policy.agentsfleetd mints a runner token and shows it once. Copy it before you close the dialog, because it is never shown again.
Put that token on the host, then start agentsfleet-runner. Install a runner covers the binary and its toolbox. The host reads two required variables and six optional ones:
An unreadable log level falls back to
info. A missing or malformed address or token stops the runner at start and names the variable.
No policy value is read from the host. A runner that carried its own isolation setting could disagree with the dashboard, and nothing would compare the two.
What the runner sends to a collector
The runner holds no telemetry credential. SettingOTEL_EXPORTER_OTLP_HEADERS, or a signal’s own header variable such as OTEL_EXPORTER_OTLP_TRACES_HEADERS, stops the runner at start and names the variable. A user name or password in the endpoint stops it too, so give the credential to the collector on the host.
The runner sends uncompressed. Any OTEL_EXPORTER_OTLP_COMPRESSION variable, a signal’s own included, stops it at start.
With an endpoint set, the runner sends one trace for each run. The trace holds the run, each model turn, and each tool call, with the lease and event identifiers.
It also sends eight counts and timings with no tenant or fleet name on them:
- model turn time and retries, by provider
- sandbox start time
- live updates it could not deliver
- memory saves that failed
- tool call time, by tool
- processes the kernel killed inside a sandbox for lack of memory
- sandboxes kept between runs, by what became of them
agentsfleet_runner_otlp_entries_discarded_total. It never sends its log lines, a prompt, a reply, a tool’s output, or a credential. Log lines stay on standard error, for the host’s log store.
The policy you assign
Policy travels down from the dashboard. Change it under Admin → Runners → Edit and the host applies it on its next heartbeat. Nobody logs in to the host.
The three isolation settings differ in what enforces the boundary:
Labels
Labels are short names you give a runner when you enrol it. Change them under Admin → Runners → Edit, or sendPATCH /v1/fleets/runners/{runner_id} with a labels list.
The new list replaces the old one, and an empty list clears every label. A runner holds at most 32 labels of at most 64 bytes each. agentsfleetd reads the labels each time the runner asks for work, so the host needs no restart.
The sandbox each run gets
A run whose fleet declares a sandbox tool gets its own sandbox. No fleet setting picks its size yet, so every sandbox gets these defaults.
A runner refuses a run that asks for a size outside these ranges, before it builds a sandbox.
Commands share the memory limit minus 64 MiB, which the sandbox keeps for itself, and there is no swap. When commands need more, the kernel kills one of their processes. The command it belonged to reads
Process killed: out of memory with the code out_of_memory, and the sandbox and its files stay for the next command.
/tmp lives on the workspace disk, so the two share one size limit. A write past it fails with No space left on device, and the sandbox keeps running. The kernel refuses a new process or thread past 512.
Shared memory, /dev/shm, holds at most a quarter of the memory limit, 512 MiB by default. A write past it fails with No space left on device, and later commands keep the rest of the memory.
After a run finishes without failing, the runner keeps its sandbox for 10 minutes for the fleet’s next run. A kept sandbox is frozen, and it still holds its memory and disk. The runner keeps at most one per worker, drops the oldest first, and drops other fleets’ sandboxes when every worker is busy.
While a runner keeps a fleet’s sandbox, only that runner takes the fleet’s next run, unless it goes offline. What a follow-up message keeps lists what ends the wait early.
What a sandbox can reach
The network policy decides what each sandbox reaches.
Under Allowlist egress, the runner builds the allowlist when a run starts. It joins the policy’s Allowlist with the hosts the fleet lists in
network.allow in its Definition. A fleet that sets read_only keeps its hosts off the list and reaches them only through http_request.
An empty Allowlist means the runner’s default registry set. That set is registry.npmjs.org, pypi.org, files.pythonhosted.org, crates.io, index.crates.io, static.crates.io, proxy.golang.org, and sum.golang.org.
The runner looks up each host’s Internet Protocol version 4 (IPv4) addresses on the host itself. The sandbox finds those names in its own /etc/hosts and has no Domain Name System (DNS) resolver. Traffic to port 53 is dropped, so a name outside the list never resolves.
Each sandbox joins the host through its own virtual Ethernet pair. nf_tables rules on the host side of that pair admit only the allowlist’s addresses. The rules live in the host’s network namespace, so code inside the sandbox cannot see or change them.
The runner refuses a run before its sandbox starts when a listed host has no IPv4 address. It also refuses a run whose allowlist holds more than 256 addresses. The run then fails with “the egress allowlist could not be resolved into addresses this runner can admit”.
A host from network.allow may not resolve to a private or reserved address, such as 10.0.0.0/8, 100.64.0.0/10 or the cloud metadata address 169.254.169.254. Such a run fails with “the fleet allows an egress host at a private or reserved address”. Hosts in the runner’s Allowlist are exempt, so an operator can point one at a package mirror on its own network.
In both cases the runner’s log names the host and never an address.
Allowlist egress needs three things from the host:
- a kernel with
nf_tables - IPv4 forwarding on,
net.ipv4.ip_forward=1 - no host firewall whose forward policy drops traffic
egress_enforcement: false. A runner assigned Allowlist egress then reads degraded with the reason egress enforcement unavailable.
The states a runner shows
The runner page shows two facts side by side: what you have told the runner to do, and whether the host is answering. The first is the admin state, which the API returns asstatus. You set it with the actions below.
Only active runners can send authenticated requests.
The second is liveness. The host reports it through its heartbeat; you cannot set it.
When a runner stops taking work
A runner reports what its kernel can actually enforce on every heartbeat.agentsfleetd compares that report against the policy you assigned.
When the host cannot meet the assignment, the runner is marked degraded and is sent no work. The reason names the one missing piece, such as cgroup controllers not delegated. Fix that piece, or lower the assignment, and the next heartbeat clears the mark.
The runner page carries the verdict of those checks beside the runner’s state and labels. It reads checks passed, N checks failed, checks stale, checks pending, or checks never run. Select it to read the full report, check by check.
You can also stop a runner deliberately:
- Cordon stops new work reaching it.
- Drain requests a drain and stops new work reaching it.
- Revoke ends its token, so the host’s next request is refused.
Lease history
Each runner page lists the leases the runner held, newest first. A lease is one piece of fleet work the runner took.GET /v1/fleets/runners/{runner_id}/leases returns the same list.
A finished lease stays in the history for 30 days, counted from when it finished. Deleting its fleet does not remove it. A lease still running is never removed.
The API returns fleet_name: null once the fleet is gone; the dashboard still shows its callsign from the saved identifier.
The event type and actor are blank once the event is gone.
Each lease carries destination, which says whether you can open its fleet:
The runner’s lifetime totals count every lease it took, so they can exceed the history the page lists.
Limits
- Allowlist egress is IPv4 only. The runner refuses a run whose listed host has only Internet Protocol version 6 (IPv6) addresses.
- A run keeps the addresses it started with. A host whose address changes during a run stays unreachable until the next run looks it up again.
- None is refused outside development. A release build of
agentsfleetdrejects it at startup. - A run sends at most 256 spans, and the runner at most 128 a second. The rest are counted in
agentsfleet_runner_spans_suppressed_total, not sent. Every run keeps its top-level span, so each run stays one trace. - A host can lie about what it enforces. The capability report is self-declared, so treat placement as something you decide, not something the host proves.
- Use matching runner and server releases. Runners older than
agentsfleet-runner0.61.0 cannot accept work from the current server.