runTaskOnRuntime in apps/web/server/lib/executorRunner.ts, drives one existing board task through one runtime from claim to finish. Given a task id, an assignee and an adapter factory, it claims the task, opens an execution row, sets up a worktree when the task changes files, assembles a tiered prompt, drains the adapter’s event stream while recording spend and running the circuit breakers, rotates the session when it runs out of room, runs the verification gate, and writes a handoff for the next runtime.
It is one of two execution paths, and not the default one:
- The executor runner serves
POST /api/runtimes/:id/runand agent Routine fires on Clawboo Native, Claude Code, Codex and Hermes. WithCLAWBOO_ENABLE_MOCK_RUNTIME=1, the run route and Routine fires onclawboo-mockuse it too. When it claims a task it saves its inputs on the task (tasks.runner_inputs: runtime, agent, repository, kind, model), and a person’s Retry, Assign or reply on that task runs it through the runner again (lib/runnerRerun.ts;lib/runnerRoute.tsdecides). The team engine’s ready list leaves such tasks out, and the server’s dispatch pump re-fires them through the runner instead (pumpRunnerReruns). - The team engine (
serverDeliver, driven by the team orchestrator) runs every team-chat turn, every delegation, and every other board task a person starts, including a rerun of a task with no saved runner inputs. It provisions no worktree. As the working directory for Claude Code, Codex, Hermes and Clawboo Native it passes the task’s own folder (<work folder>/teams/<teamId>/tasks/<taskId>, fromtaskRunFolderFor, with the team folder asreadRoot) for a run that holds a task, and the team’s folder (<work folder>/teams/<teamId>, fromteamWorkdirFor) for every other turn. It has no verification, memory auto-injection or circuit breakers. See How a team works.
Where it sits
The runner talks to a runtime only through theRuntimeAdapter trait and an injected adapter factory (adapterFactoryFor(runtimeId) in production). It is not:
- the team engine, which decides what to delegate and runs team chat;
- the OpenClaw path: OpenClaw runs over its live Gateway connection, so the runner refuses it before claiming anything, and OpenClaw routine fires use a separate dispatcher (see How a Routine fire picks its path);
- the verification critic: it triggers the verification gate, which reuses the runner’s adapter factory to drive an independent reviewer.
runTaskOnRuntime(input) returns a RunTaskResult and never throws for an expected board outcome. A refusal is { ok: false, reason }; runtimesRunPOST maps it to an HTTP status:
A completed run answers
200 with the result (doneReason, final status, summary, costUsd, usedWorktree, degradations). An unexpected throw answers 500. The request blocks for the whole run, and the handler aborts the run if the client disconnects.
The lifecycle
Before the claim
Everything that can refuse a run without changing the board runs before the claim, so a misrouted or over-budget call never opens an execution row or spawns a process. In order:- Aborted waiter. A run first takes one of the server’s run slots (below), then, for a persistent-home runtime, waits its turn on a per-agent lock (
homeDispatchMutex, below). The slot wait has no time limit, since waiting for capacity is not a failure. The lock wait is bounded byHOME_MUTEX_ACQUIRE_MS(10 minutes of awake time,CLAWBOO_HOME_MUTEX_ACQUIRE_MS); past it the run rejects withMutexAcquireTimeoutError. If the caller disconnected or a person moved the card during either wait, the runner returnsconflictwithout touching the board. - Task exists, else
not_found. - Depth. A task with more than
MAX_SPAWN_DEPTH(2) ancestors is refused withtoo_deep. Creation and dispatch read the sameDEFAULT_MAX_DEPTHfrom@clawboo/governancebut compare differently: a create is refused when the parent is already at the cap, a dispatch only when the task itself is past it. So anything that can be created can also run. - Connected runtime.
resolveRuntimeIntegration(caps)gives OpenClaw aconnectedhome, and the runner returnsconnected_substrate. - Budget preflight and hold. A paused budget in
capmode on the agent, the mission (the root task of the delegation tree) or the team returnsbudget_paused. Otherwise the run takes a hold on every one of those budgets (reserveRunBudget,lib/budgetHold.ts): its estimated cost, kept inbudget_reservationsuntil the attempt ends. A cap-mode budget that other runs already hold refuses a run whose estimate does not fit beside them withbudget_held; a run with no other run going against the budget is let start as before. Budgets inwarnmode, the default, never block. The hold is checked again right before the run’s first model call: a refusal there closes the execution rowcancelledwithbudget_paused:<scope>orbudget_held:<scope>, comments on the task and releases it totodo. Every cost the run charges passes the hold’s id, so its spend replaces the estimate, and the hold is released when the attempt ends, however it ends. Each fix cycle takes a hold of its own. Before a run rotates to a fresh session, it reads the same three budgets again: a hard cap another run paused meanwhile ends it as a budget pause (budget_paused:<scope>) instead of starting the successor. See Runs that go at the same time.
homeDispatchMutex, keyed on the home path. The team engine and 1:1 chat take the same lock for their own runs of that agent. An ephemeral home (Claude Code) is never keyed.
Above the per-agent locks sits one process-wide cap on runs executing at once (runSlots.ts, CLAWBOO_MAX_CONCURRENT_RUNS, 8 by default). Every run start takes a slot: this runner, team chat’s serverDeliver, 1:1 chat, the peer exchange and the OpenClaw Routine dispatcher. The order is always slot first, then the agent’s lock, so a run waiting for a slot holds nothing another run needs; the reverse order deadlocks a lead whose delegate waits on a lock held by a run that is itself waiting for the lead’s slot. Here the slot wait comes before the claim, so a queued task stays in To do, and the slot is held until the run, its completion and its review are over. The verification reviewer takes no slot of its own; it runs inside this one. Each run names its agent lock as the slot’s key, and the queue never gives a slot to a run whose agent already holds one, so several runs of one busy agent wait without taking slots other agents could use. The queue serves a person waiting on a turn first (a chat message, a peer exchange), then background work by its task’s priority, one level higher per 15 minutes waited, oldest first within a level; while both kinds wait, one slot in every four goes to background work. A request also carries a check for a pause, read when the queue is about to grant, so a paused run keeps its place and blocks nobody. A pause this runner meets before the queue refuses the run with paused, and one placed while it waits or runs ends it through stopPausedTaskRuns: its execution closes cancelled with errorClass Paused, and the task returns to To do to start again on resume. Team chat cannot wait for a delegated task’s slot inside deliver, because the engine awaits deliver from within a lead’s drain: such a delivery that finds every slot taken settles at once and starts later, it tells the engine how it ended only while its session still holds that task, and the engine’s isRunQueued keeps its idle watchdog off it. Other team-chat deliveries (a person’s or a Routine’s message, a task update) are never awaited from a drain, so they wait for their slot before they settle.
The claim
The claim is one conditionalUPDATE through the board’s claimTask (see the atomic claim). A lost claim returns conflict and is never retried: another runner legitimately owns the task. A verification fix cycle is the one dispatch that does not claim; it checks that the task is still in_progress and still assigned to this agent, and returns conflict otherwise.
Right after the claim the runner opens an execution_processes row and emits execution_started. The row’s run reason is verify-fix on a fix cycle, resume-via-handoff when the runtime cannot resume natively, and run otherwise. A liveness heartbeat (startTaskHeartbeat, every 30 seconds) starts before the slot and mutex waits and stops in a finally, so it spans the waits, the run and every fix cycle; that beat is what keeps the stale sweep from releasing a live run.
The workspace
acquireWorkspace sets up a workspace only when the adapter declares capabilities().worktrees. Clawboo Native, Claude Code, Codex and Hermes all do; the test-only clawboo-mock does not, so its runs get no working directory. Whether a task needs a worktree comes from its kind (isolationForTask in @clawboo/worktrees): research and review get folder, which means no worktree; code, the default, and any unknown kind need one. There are four outcomes:
- Reuse. The task already has a worktree, which is the cross-runtime continuation path. If garbage collection reaped it (the row is
stale, the directory is missing, or git does not list it as a worktree), the runner rebuilds it from the retained branch withresumeTaskWorkspace. Whenever the directory is there, astalerow included, the runner first asksworktreeRegistration, which answersunknownwhengit worktree listitself fails; that is refused (below) rather than read as reaped, because the rebuild may replace the folder.resumeWorktreeasks git once more before it touches anything; when that check goes unanswered it throwsRegistrationUnknownError,resumeTaskWorkspacereturnsregistration_unknown, and the run is refused the same way, with no fresh provision. A registered checkout of the task branch with no uncommitted changes is kept as it is. Any other folder is replaced only when it holds nothing to lose: missing, empty, or a checkout of the task’s repository at a commit a branch holds, with nothing ingit status --ignored. OtherwiseresumeTaskWorkspacereturnsunsafe_to_rebuildand nothing falls back to a fresh provision, which would replace the folder the same way. - Provision. A file-changing task with a
repoPathgets a fresh checkout on aclawboo/task-<id>branch fromprovisionTaskWorkspace, with the handoff scaffold files. - Folder. A
researchorreviewtask gets no worktree. Its run starts in the task’s own folder fromtaskRunFolderFor(apps/web/server/lib/runtimes/teamWorkdir.ts):<work folder>/teams/<teamId>/tasks/<taskId>, orteams/_no-team/tasks/<taskId>for a task with no team, created owner-only (0700) first byensureTeamWorkdir. The run reports that it is working in the team folder that holds the task folder (startFolderRun), so a team-chat work check waits for it, and the run context passes that team folder asreadRoot: Clawboo Native’sread_fileandlist_filesmay reach it whilewrite_filestays in the task folder, and Claude Code keeps reading theCLAUDE.mdfiles from the team folder down. A team folder the work folder move kept at its old place,<CLAWBOO_HOME>/workspaces/<teamId>, is used where it is (see The work folder). If the folder cannot be created, the run is refused as below, withCould not create the working folder <path> for this run: <error>as the reason. - Refused. A file-changing task with no
repoPath, or one whose worktree cannot be provisioned, does not run. The runner adds a[blocked: no isolation] <reason>system comment, closes the execution asfailedwithworkspace_unavailable: <reason>, releases the task totodo, and returnsworkspace_unavailable. Two refusals differ only in their comment and where the task goes. A registration check git did not answer adds[not started: worktree unchecked]and releases the task totodo. A checkout that cannot be rebuilt without losing work adds[blocked: worktree kept], naming the folder and why, and moves the task toblocked, so it waits in Needs you instead of starting again and refusing the same way.
POST /api/runtimes/:id/run defaults it to code, so a request without repoPath is refused; the Routines wake bridge runs an agent Routine that has no repoPath as research, so it runs without a worktree instead.
When a worktree exists, the runner reconstructs a ResumeState from its AGENT_HANDOFF.json and scaffold files, so a different runtime picks the task up where the last one left it. See Worktrees and handoff.
Prompt assembly
The prompt goes throughassembleTiers from @clawboo/executor/tiers, in three tiers ordered so the frozen part forms the cacheable prefix:
The inbox digest packs up to 20 undelivered mailbox rows for the assignee into the addressed section, as the team-chat path does, and marks only the rendered rows delivered once the run has started.
Memory auto-injection (
buildMemoryInjection) seeds the most relevant team facts for the task into the volatile tier, so the runtime starts with the team’s knowledge without calling the Memory MCP tools (it still can). It searches the same memory store the /api/memory routes use, keeps the top 5 facts within 1,500 characters, drops any candidate the injection scanner flags, scrubs secrets, and injects nothing when memory is empty. It runs once per task and is reused across rotations. disableMemoryAutoInject turns it off (eval runs set it). See Memory.
A persistent-home runtime also gets homeDir, <Clawboo state dir>/runtimes/<runtime>/<agent>, created owner-only (0700) before the driver touches it; the critic’s own run gets none, so the reviewer never shares the builder’s memory. The run context (RuntimeRunContext) carries the cwd (the worktree, else the task’s own folder), the readRoot (the team folder, on a run in a task’s own folder), the model, the native resume id (only when the last runtime on this worktree was the same one), the MCP base URL, the memory scope (team, agent, the attach secret, and provenance stamps), the home directory and any provider keys. The run starts with session key runtime:<runtime>:task:<taskId>, the task title as its message, and childToolBlocklist: ['sessions_send'].
The event drain
adapter.start returns a RunHandle, and the runner drains adapter.events(run) through withIdleTimeout until a terminal done:
text-delta: accumulated into the run’s text. Thereasoningchannel is skipped, so the summary shows what the agent did, not its private reasoning.cost: recorded against budgets (below) and emitted as acostevent.tool-call(settled calls only, not each streaming-input delta),tool-resultanderror: emitted as observability events and fed to the circuit breaker. Anerrorgoes through the error taxonomy: an unknown class is a harness bug and raises an alert; a recognized policy denial is expected governance, not a bug.done: captures the reason and summary and ends the pass.
CLAWBOO_RUN_SILENT_TIMEOUT_MS), then allows 5 seconds for the adapter’s own terminal. Time the machine spends asleep is left out, except a sleep of under about a minute. A silent timeout is recorded as a failure with the cause stated (“runtime silent: no events for N minutes”), never as a clean abort and never as a reason to rotate. A pass that finishes without reporting any usage logs a spend_unreported warning, because budgets then under-count that run.
An external abortSignal (the REST handler wires it to the response’s close) aborts the current run, even across a rotation, and forces the aborted terminal.
Stopped by a person
runTaskOnRuntime folds the caller’s signal and a person’s stop into one AbortController and registers the run with the per-task stop registry (registerRunStop, apps/web/server/lib/runStops.ts) for its whole life, fix cycles included. After a board PATCH writes a move, noteHumanMove calls stopTaskRuns(taskId, note) for every move except into in_progress or in_review. That aborts the run and leaves a note such as Stopped: a person moved the card to Done., which becomes the run’s system comment. The result carries stoppedByPerson: true, so a caller treats the move as the outcome rather than a failure (a Routine fire stopped this way stays armed). The OpenClaw routine dispatcher registers with the same registry; the team engine stops the runs it started itself.
Steering
The live run is also registered with the per-task steer registry (registerRunSteer, runSteers.ts), re-pointed on every rotation and fix cycle. A person’s comment on the task reaches it through steerTaskRun after the comment is written to the agent’s mailbox, which is where a task the runner runs reads comments, since it has no ticket block. How each runtime takes a steer is described on the RuntimeAdapter trait page.
Spend and budgets
On everycost event the runner records spend against the agent, mission and team budgets with recordSpend. A runtime that reports usage but no USD (costUsd: null) is priced from its exact token counts with estimateRunCostUsdFromUsage, which falls back to the shared default rate for an unpriced model; a reported costUsd is used as is. An event flagged local, or a null-USD event whose model isLocalModel recognises (an ollama/ id without a cloud tag), costs an exact $0. A `done` that carries `costUsd` is charged only when its pass had no `cost` event. A pass that ends with no spend at all is priced from characters with `estimateRunCostUsd`, from the prompt it was sent and the text it produced, at the model the run was given (`input.model`), or $0 when that model is local. An aborted pass is charged this way too; only a pass that ended on its own logs the spend_unreported warning. The per-run ceiling accumulates whole cents while the budget ledger takes fractional cents, so sub-cent events are carried, not lost. The execution row’s costSource comes from the run’s least trustworthy part: char-estimate if any pass was priced from characters, else token-estimate if any was priced from tokens, else reported if any part was paid, else local. When the run recorded a cost, recordRunTokens (runTokenLog.ts) also writes one cost_records row for Tokens Used, with tokens guessed from characters for the passes that reported none.
Budgets default to warn mode. A warn budget records spend and writes a budget audit row when it crosses 80% and 100%, but never pauses. Only a cap budget that crosses 100%, or the per-run maxNodeCents ceiling (set only by a Routine template through the API), stops the live run, and not at once: the crossing leaves a pending stop. The runner aborts the run at its next event that means more work (any event except a done, a fatal error, or a status outside thinking and running), or when BUDGET_STOP_GRACE_MS (5 seconds) passes first. If the run reaches a done other than aborted first, the report that crossed was the run’s last, and the run keeps its result (see Cap reached at the end below). That is the usual case for Claude Code and Codex, which send their one cost report beside their done. The explicit mode === 'cap' check backs up the database layer’s rule that a warn budget never reads paused. See Governance.
Circuit breakers
The circuit breaker halts a run that makes no progress, repeats a failing tool or burns tokens, before the dollar ceiling. It reads typed signals only: token counts oncost, a tool signature on tool-call and tool-result (correlated by toolCallId), and the typed code of a policy-denied error. The defaults (BREAKER_DEFAULTS in @clawboo/governance):
POST /api/runtimes/:id/run accepts a partial breakerConfig to override them. The breaker is fed only while no budget stop is pending, and its teardown runs after the budget teardown, so the budget wins a tie and a run gets at most one adapter.abort.
Session rotation
The runner’s unit of context is the run, not the model turn: the runtime owns its inner loop. When a pass ends out of room (an explicitdone with reason max_turns, which Claude Code also sends when its context window is full, or a non-success done whose input tokens crossed 85% of the declared context window) and rotations are left (maxRotations, default 3), the runner calls rotateSession. That serializes the predecessor best-effort, builds a short handoff note (the last summary, never the transcript), and starts a fresh successor session (<sessionKey>:r<n>) whose prompt carries the note in its context tier.
The watermark reads input tokens alone, because the prompt is what occupies the window. A runtime that declares no context window and never emits max_turns never rotates. Successors always start fresh (ctx.resume = null): resuming the exhausted session would exhaust it again. Budget and breaker state are cumulative across rotations; only the per-pass counters reset. Each rotation records session lineage and emits session_rotated. A budget, breaker, cancel or silent-timeout stop ends the run and never rotates, and neither does a pass whose last cost report crossed a cap.
Terminal handling
After the loop exactly one terminal branch runs. Every branch closes the execution row only if it is stillrunning (a stale sweep may have closed it first), and emits execution_completed only when its close landed.
Budget pause. Audit a budget row with reason auto_pause, add the comment Auto-paused: <scope> budget reached. Raise the cap (or resume) to continue., close the execution cancelled with budget_paused:<scope> and errorClass BudgetRefused (no person stopped it, so a saved wake it carried goes back to waiting, uncounted), and release the task to todo, ready to retry once someone raises the cap.
Cap reached at the end. Not a branch of its own. When the cost that crossed a cap came as the run ended (the pass’s done was not aborted, or its characters were priced after it ended on its own), the runner audits a budget row with reason cap_reached_at_end and adds the comment Budget cap reached (<scope>) by this run's final cost report. The run had already ended, so what it produced is kept; the next run is refused until you raise the cap or resume. For the per-run ceiling the comment reads This run's cost ceiling was reached by its final cost report. The run had already ended, so what it produced is kept. The run then takes the success or failure branch like any other, and the paused budget refuses the next run at pre-flight.
Breaker trip. Audit a circuit_break row, add [stopped: <reason>] <detail> Released to todo for re-planning., close the execution cancelled with circuit_broken:<reason>, and release to todo. The worktree stays intact, so a retry resumes from it.
Success. The summary is scrubbed of secrets before it lands in a board comment, the execution row, the handoff or the HTTP response. The runner then writes the agent’s comment and closes the execution succeeded, unless the work goes on to the check, whose execution stays open until the verdict (below). Then:
-
The agent blocked its own task (
block_task, orupdate_task_statustoblocked): the task isblockedand no person put it there, which the runner reads from the stop registry and the task’spersonMovedAt. The ending is the team engine’s for the same thing. The execution closessucceeded, the card stays blocked with no late result, the steps waiting on it stay queued, and the event log getsstatus_changedwith the reasonagent_blocked. The delegator (or the team leader) gets a mailbox notice with the reply, unless a person assigned the task. No check runs, and a worktree stays as it is, with the reply aswhyBlockedandnextBestStepinAGENT_HANDOFF.json, for a Retry to carry on from. Rotation does not start a fresh session on such a task. -
With a worktree: persists the native session id into
AGENT_HANDOFF.json(best-effort) and, if a previous attempt crashed, records that its outcome is unknown. WithkeepForResumeit releases the task totodofor another runtime to pick up. Otherwise it callsactOnTaskWorkspace(taskId, 'complete', { runId }), which runs the verification gate and routes the status, and closes the execution only after that:The attempt budget isCLAWBOO_MAX_FIX_CYCLESplus one, 2 by default. Fix cycles run outside the home mutex, each taking it fresh, because the mutex is not re-entrant. The gate runs the team’s verify command when the team has one (team-verify-command:<teamId>); otherwise it reads its command fromVERIFY_CMDin the worktree’sinit.sh(or the- Verify:line inVERIFICATION.md) and records whether that command is still the scaffold’s or the agent’s (commandSource). Without a team command the scaffold writes a placeholder there, so a task whose agent never sets a real command fails verification; the run’s context and the worktree’sTASK.mdtell the agent to set one. If a person moved the task while the check ran, their move stands and the verdict stays only as a record. The execution staysrunningthrough the check, with the taskin_reviewand beating (startReviewHeartbeat), so a crash leaves a row recovery can find: the boot reap, or the stale sweep after a quick restart, moves the task toblockedin Needs you and tells the delegator (see Orphan and stale reconciliation). A check that lost its execution that way, or to a newer run, writes no verdict, moves nothing and asks for no fix cycle (supersededon the result). A run that lost its execution before the check even starts (the sweep released it while it worked) touches neither the worktree nor the card, and its work waits in the worktree for the next run. If a check throws, or has not answered afterWORK_CHECK_TIMEOUT_MS(30 minutes of awake time), the task moves toblockedfirst, and only then does the execution closefailed: if the move does not land, the row stays open for the sweep. A person who moved the card meanwhile keeps their move, and the execution closes when the check answers or at that limit. The critic reuses this run’s adapter factory on a detached checkout with a fresh session and no builderhomeDir;CLAWBOO_REVIEWER_MODELcan make it a different model. See Verification. -
Without a worktree (a
researchorreviewtask):updateStatus(db, taskId, 'done'). If the board refuses (the task was released, the verification gate holds a non-promotable verdict, or the row is gone), the runner keeps the result as a[late result ...]system comment that says which.
Run <reason>: <summary>), the execution closes cancelled for an abort or failed otherwise, and a done with reason error is classified through the taxonomy and alerts on an unknown class. A run that attempted a native resume and failed clears the stored session id, so the next dispatch falls back to the prose handoff instead of retrying a stale id. Finally releaseTask returns a task still in_progress to todo; a task a person already moved stays where they put it, and the result reports its real status.
Degradation notes
planDegradations(caps) (apps/web/server/lib/degradation.ts) derives three flags from the adapter’s capabilities:
describeDegradations renders the notes into the prompt’s context tier and the result’s degradations, so a gap is visible. The flags do not change tool handling: Clawboo adds no approval gate for a runtime that lacks one. Every real adapter declares resume, toolApproval and streaming, so in practice no run gets a note.
Tracing
Every dispatch runs insidewithTaskSpan. The trace id is the mission root, so a multi-agent task renders as one trace. The run’s span id is derived from its task id, and its parent span is either an explicit cross-process parentTraceparent or the parent task’s span. Lifecycle events (execution_started, cost, tool_call, tool_result, error, session_rotated, execution_completed) go through emitEvent, which is best-effort. The Org chart’s live overlay and the trace view read this same event log. See Observability.
Design rationale
“Drive one runtime through one task” is one testable function with one set of invariants, instead of being spread across the team engine, the worktree code and each adapter. It buys three things:- Capability-driven routing. The connected-runtime refusal, home serialization, degradation notes and the rotation watermark all come from declared capabilities, so a new runtime is a new adapter and driver, not a runner change.
- One teardown per run. Cumulative budget and breaker state above the rotation loop, the breaker yielding to a budget stop, and never rotating over a stop guarantee one
adapter.abortand one terminal, even across a rotation chain. - Recoverability. Every terminal closes the execution row and either sets a real status or releases the task; the board’s orphan and stale passes clean up after a crash (see Orphan and stale reconciliation).
Related
- The RuntimeAdapter trait: the interface the runner consumes
- Board internals: the claim, the transitions and the recovery passes
- How a team works: team chat versus the task runner
- What agents can do: the coding runtimes’ safety posture
- Verification and Governance: the gate and the budgets this runner enforces
- Runtimes API:
POST /api/runtimes/:id/run