Observability API

REST surface over the durable orchestration event log: query the raw event feed, reconstruct a single trace with metrics, pull the harness-bug error feed, read the fleet-health triage, fold the event-sourced delegation graph, live-tail the log over Server-Sent Events, and mirror client-observed runtime events back into the log. This group also covers POST /api/eval/smoke, the on-demand deterministic eval run. Observability is always on; there is no feature gate, and these handlers serve unconditionally. The event log is the single read source: a trace is all events sharing a traceId, the graph is a projection of the ordered stream, and metrics/error-taxonomy/fleet-health all fold from the same rows. All POST routes read a JSON body parsed by express.json({ limit: '2mb' }).

Every payload-bearing response redacts the event’s JSON data field before it is sent; credential-shaped keys/values are masked with ••••. This is display-layer defense in depth on top of the storage-layer secret scrub; numeric telemetry (token counts, cost) survives the mask. GET /api/obs/stream is Server-Sent Events, not request/response; it is documented below with an event-stream catalog, not a JSON response body.

Routes

Method	Path	Summary	Stream?
GET	`/api/obs/events`	Query the orchestration event feed	No
GET	`/api/obs/traces/:traceId`	One reconstructed trace + aggregate metrics	No
GET	`/api/obs/errors`	The error-taxonomy feed (harness-bug filter)	No
GET	`/api/obs/health`	Fleet-health triage (working/idle/stalled/zombie)	No
GET	`/api/obs/graph`	Event-sourced delegation/status/cost graph projection	No
GET	`/api/obs/stream`	SSE live-tail of the log, scoped by team/task/agent	SSE
POST	`/api/obs/ingest`	Mirror client-observed runtime events into the log	No
POST	`/api/eval/smoke`	Run the deterministic eval smoke suite	No

`GET /api/obs/events`

Returns rows from the event log filtered by the query params, ordered by the monotonic seq cursor. The default order is asc (causal order, so a feed reads chronologically); pass order=desc for a recent-first feed.

Path params: none.
Query params:

Param	Type	Notes
`teamId`	string	Match events for one team.
`taskId`	string	Match events for one board task.
`agentId`	string	Match events for one agent (the per-agent activity scope).
`traceId`	string	Match events sharing a trace.
`kinds`	string	Comma-separated list of event kinds; empty/blank values dropped.
`since`	number	`ts >=` (wall-clock ms), a recent-window read.
`afterSeq`	number	`seq >` cursor (strictly monotonic, collision-free).
`limit`	number	Row cap, clamped to 1-5000 (defaults to 500 when omitted).
`order`	`asc` \| `desc`	`desc` only when literally `"desc"`; anything else is `asc`.

Request body: none.

Responses

200 OK: the matched events. Each row is the full stored shape with its data field redacted (still a JSON string after masking):

{
  events: Array<{
    seq: number
    id: string
    ts: number
    kind: OrchestrationEventKind
    teamId: string | null
    taskId: string | null
    agentId: string | null
    runtime: string | null
    traceId: string | null
    spanId: string | null
    parentSpanId: string | null
    correlationId: string | null
    data: string // JSON string, credential keys/values masked with ••••
    tenantId: string | null
    createdAt: number
  }>
}

OrchestrationEventKind is one of: task_created, task_claimed, status_changed, comment_added, dep_linked, execution_started, execution_completed, tool_call, tool_result, cost, approval_requested, approval_resolved, error, span_start, span_end, session_rotated, routine_fired, routine_dispatched, routine_completed, routine_error, team_chat_post, speaker_selected, turn_bound_hit. See Events & errors reference for the per-kind data shapes. 500 Internal Server Error: any read failure:

{ "error": "<message>" }

Example

# Last 100 cost + error events for a team, recent first
curl "http://localhost:18790/api/obs/events?teamId=<team-uuid>&kinds=cost,error&order=desc&limit=100"

`GET /api/obs/traces/:traceId`

Reconstructs one trace, every event sharing the traceId, ordered seq ASC (causal). The full multi-agent task renders from this (leader → specialists → tool spans). Aggregate metrics are computed from the un-redacted events first, then each event’s data is masked for display, so numeric cost/token telemetry in metrics is accurate.

Path params: traceId (string).
Request body: none.

Responses

200 OK: the trace events plus the aggregate metrics:

{
  traceId: string
  events: Array<{/* same redacted row shape as GET /api/obs/events */}>
  metrics: {
    totalCostUsd: number
    inputTokens: number
    outputTokens: number
    toolErrorRate: number // failed tool results / total tool results (0 when none)
    toolCalls: number
    toolErrors: number
    eventCounts: Partial<Record<OrchestrationEventKind, number>>
    activeAgents: number
    tokensPerMinute: number // output tokens/min over the window (0 when window < 1s)
  }
}

The trace read is capped at 5000 events. An unknown traceId returns 200 with an empty events array and zeroed metrics. 500 Internal Server Error:

{ "error": "<message>" }

Example

curl http://localhost:18790/api/obs/traces/<trace-id>

`GET /api/obs/errors`

The error-taxonomy feed: every error event, recent-first, projected to a compact shape. An error whose class is Unknown is a harness bug; pass harnessBug=true to filter to those.

Path params: none.
Query params:

Param	Type	Notes
`harnessBug`	string	`"true"` filters to harness-bug errors only; any other value returns all.
`since`	number	`ts >=` (wall-clock ms).

Request body: none.

The read is fixed to kinds: ['error'], limit: 500, newest-first by ts. Wall-clock order is what a display feed wants, and it is the order the only index over kind can serve, so the cost stays proportional to the 500 rows returned rather than to every error ever recorded. The harnessBug filter is applied after projection.

Responses

200 OK: the error rows plus the unfiltered harness-bug count. The whole payload is run through redactObject:

{
  errors: Array<{
    seq: number
    ts: number
    taskId: string | null
    agentId: string | null
    runtime: string | null
    errorClass: string // from event data; defaults to 'Unknown'
    harnessBug: boolean // from event data
    message: string // from event data; defaults to ''
  }>
  harnessBugCount: number // count across ALL errors in the window, before the filter
}

harnessBugCount always reflects the full window (harness bugs found regardless of the harnessBug filter), so a UI can badge the alert count even while showing the unfiltered feed. 500 Internal Server Error:

{ "error": "<message>" }

Example

# Harness-bug alerts in the last hour
curl "http://localhost:18790/api/obs/errors?harnessBug=true&since=$(($(date +%s%3N) - 3600000))"

`GET /api/obs/health`

Fleet-health triage, a per-agent state folded from the event log, time-sensitive (computed against Date.now()). An agent with an open execution is working if a recent event landed, stalled if quiet past 5 minutes, and zombie if quiet past 30 minutes (the process is almost certainly dead, what orphan reconciliation reaps); an agent with no open execution is idle.

Path params: none.
Query params: teamId (string), scope to one team.
Request body: none.

The read is capped at the most recent 5000 events for the requested scope. orchestration_events is append-only and is never pruned, so the triage folds a trailing window rather than the whole history: rows are selected newest-first, then folded in causal order. An agent whose activity has scrolled out of that window stops appearing.

Responses

200 OK: one entry per agent that appears in the window:

{
  agents: Array<{
    agentId: string
    status: 'working' | 'idle' | 'stalled' | 'zombie'
    lastEventTs: number
    activeTaskId: string | null
    openExecutions: number
    costUsd: number
  }>
}

500 Internal Server Error:

{ "error": "<message>" }

Example

curl "http://localhost:18790/api/obs/health?teamId=<team-uuid>"

`GET /api/obs/graph`

Folds the ordered event stream into a delegation/status/cost graph projection. The team graph is a pure projection of the log (replaying the same log always reproduces the same graph), so the surface cannot drift from reality. Two views fold from one ordered list: the task-delegation graph and the derived agent-to-agent delegation graph.

Path params: none.
Query params: teamId (string), scope to one team.
Request body: none.

The read is capped at the most recent 5000 events for the requested scope. orchestration_events is append-only and is never pruned, so the projection folds a trailing window rather than the whole history: rows are selected newest-first, then folded in causal order. A task whose events have all scrolled out of that window stops appearing, and one only partly inside it projects from the events that remain.

Responses

200 OK: the projected graph (this response is the projection output verbatim, not wrapped or redacted; it carries no raw event data):

{
  tasks: Array<{
    id: string
    title: string | null
    status: string
    assigneeAgentId: string | null
    parentTaskId: string | null
    runtime: string | null
    teamId: string | null
    costUsd: number
  }>
  taskEdges: Array<{
    id: string
    source: string
    target: string
    kind: 'delegation' | 'dependency'
  }>
  agents: Array<{
    id: string
    costUsd: number
    taskIds: string[]
  }>
  agentEdges: Array<{
    id: string
    source: string
    target: string
    kind: 'delegation' | 'dependency' // agent→agent edges are always 'delegation'
  }>
}

500 Internal Server Error:

{ "error": "<message>" }

Example

curl "http://localhost:18790/api/obs/graph?teamId=<team-uuid>"

`GET /api/obs/stream`

Server-Sent Events live-tail of the event log, scoped by team/task/agent. The handler opens an text/event-stream response and polls the log every 750 ms on the monotonic seq cursor, pushing any new rows. It is cross-process correct (it catches writes from the standalone MCP stdio bins, not just the in-process server) and indexed. Resume from a known position via the standard EventSource Last-Event-ID header or the ?since=<seq> query param.

Path params: none.
Query params:

Param	Type	Notes
`teamId`	string	Scope the tail to one team.
`taskId`	string	Scope the tail to one board task.
`agentId`	string	Scope the tail to one agent.
`since`	number	Resume cursor (`seq`). Negative/non-finite → `0`. The `Last-Event-ID` header takes precedence when present.

Request body: none.

This is an SSE route, not request/response. There is no JSON response body; the catalog below describes the wire frames. Each polled batch reads up to 500 rows past the cursor (order: 'asc'). The cursor only advances forward, so resuming with a stale since/Last-Event-ID replays from there. The stream is cleaned up on req/res close.

Connection

On open, the handler writes HTTP/1.1 200 with:

Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no

then emits a : connected comment frame and flushes any rows already past the cursor.

Event catalog

Frame	When	Shape
`: connected`	Immediately on open	SSE comment (ignored by `EventSource`)
event data	Per new log row (poll every 750 ms)	`id: <seq>` line + `data: <json>` line (the full event row, `data` field redacted)
`: keepalive`	Every 20 s while open	SSE comment (keeps the connection warm)

The data: payload is one stored event row, identical in shape to a GET /api/obs/events row, with its inner JSON data field masked (••••). The id: value is the row’s seq; the browser sends it back as Last-Event-ID on auto-reconnect, so the tail resumes without gaps. Example frames:

: connected

id: 4187
data: {"seq":4187,"id":"...","ts":1718450000000,"kind":"tool_call","taskId":"...","agentId":"...","runtime":"claude-code","traceId":"...","data":"{\"toolName\":\"read_file\"}","tenantId":null,"createdAt":1718450000000}

: keepalive

Example

curl -N "http://localhost:18790/api/obs/stream?taskId=<task-uuid>"

# Resume from a known seq
curl -N "http://localhost:18790/api/obs/stream?taskId=<task-uuid>&since=4187"

`POST /api/obs/ingest`

Mirrors client-observed runtime events into the durable log. The OpenClaw runtime is observed in the browser (the server never sees those frames), so the SPA forwards them here to keep the activity terminal uniform across runtimes. Ingestion is whitelisted to the three per-tool kinds the browser legitimately observes: tool_call, tool_result, error. Board lifecycle events (task_created, status_changed, execution_*, …) are emitted server-side by the board REST handlers and are never accepted here. Each event is best-effort: a malformed row is skipped, never failing the batch.

Path/query params: none.
Request body:

{
  events?: Array<{
    kind: 'tool_call' | 'tool_result' | 'error'  // any other kind is dropped
    ts?: number              // must be within [now - 24h, now + 60s], else server time is used
    teamId?: string | null
    taskId?: string | null
    agentId?: string | null
    runtime?: string | null   // defaults to 'openclaw' when omitted
    data?: Record<string, unknown>  // non-object → {}
  }>
}

A non-array events is treated as empty. At most 200 events are accepted per call (the rest are sliced off). Any event whose kind is missing or not in the whitelist is skipped. A supplied ts is clamped to a band around server time (60 s ahead for clock skew, 24 h behind) and replaced with server time when it falls outside. ts is not just metadata: fleet health derives staleness from it, so a future timestamp would pin an agent at working and mask a genuine zombie. A mirror reports what a browser just observed, so a timestamp far from now is wrong regardless of intent.

Responses

200 OK: the count of events actually appended (after whitelist + slice):

{ "ok": true, "count": 3 }

400 Bad Request: an unexpected throw while ingesting (the error string is redacted):

{ "error": "<message>" }

Example

curl -X POST http://localhost:18790/api/obs/ingest \
  -H 'Content-Type: application/json' \
  -d '{"events":[{"kind":"tool_call","taskId":"<task-uuid>","agentId":"<agent-id>","data":{"toolName":"web_search"}}]}'

`POST /api/eval/smoke`

Runs the deterministic eval smoke suite (SMOKE_TASKS), the exact subset CI runs, and returns the real SuiteReport. It uses no live model, no provider keys, no executor/RuntimeAdapter, and no network; each trial gets its own temp-dir SQLite board (disjoint from the real clawboo.db), and the contexts are cleaned up after the run. The ablation self-test (variants over the harness’s own subsystem flags) runs only from the manual evals.yml workflow; it is explained in the UI but never driven from this route, and no live-model judge is wired into it yet.

Path/query params: none.
Request body:

{
  trials?: number  // clamped to [1, 3]; non-finite/missing → 1
  k?: number       // clamped to [1, 3]; non-finite/missing → trials
}

Both inputs are floored and clamped so the route can never be turned into a load generator.

Responses

200 OK: the suite report:

{
  tasks: Array<{
    taskId: string
    suite: 'capability' | 'regression'
    kind: 'coding' | 'research' | 'coordination'
    trials: Array<{
      taskId: string
      passed: boolean
      score: number
      graders: Array<{/* GraderResult */}>
    }>
    passAt1: number // per-trial success rate = empirical pass@1
    passPowK: number // probability all k trials pass = passAt1^k
    meanScore: number // mean partial-credit score across trials
  }>
  passAt1: number // macro-averaged pass@1 across tasks
  passPowK: number // macro-averaged pass^k across tasks
  k: number
}

500 Internal Server Error: the run threw (contexts are still cleaned up in finally):

{ "error": "<message>" }

Example

curl -X POST http://localhost:18790/api/eval/smoke \
  -H 'Content-Type: application/json' \
  -d '{"trials":3,"k":2}'

Error envelope

Every error response on these routes is the standard envelope { error: string }. On the obs routes the error string is run through the display redactor before it is sent. The two success-shaped exceptions are /api/obs/ingest ({ ok: true, count }) and /api/eval/smoke (the bare SuiteReport).

Reference

REST API

Packages

Observability API

Routes

`GET /api/obs/events`

Responses

Example

`GET /api/obs/traces/:traceId`

Responses

Example

`GET /api/obs/errors`

Responses

Example

`GET /api/obs/health`

Responses

Example

`GET /api/obs/graph`

Responses

Example

`GET /api/obs/stream`

Connection

Event catalog

Example

`POST /api/obs/ingest`

Responses

Example

`POST /api/eval/smoke`

Responses

Example

Error envelope

See also

​Routes

​GET /api/obs/events

​Responses

​Example

​GET /api/obs/traces/:traceId

​Responses

​Example

​GET /api/obs/errors

​Responses

​Example

​GET /api/obs/health

​Responses

​Example

​GET /api/obs/graph

​Responses

​Example

​GET /api/obs/stream

​Connection

​Event catalog

​Example

​POST /api/obs/ingest

​Responses

​Example

​POST /api/eval/smoke

​Responses

​Example

​Error envelope

​See also

Routes

`GET /api/obs/events`

Responses

Example

`GET /api/obs/traces/:traceId`

Responses

Example

`GET /api/obs/errors`

Responses

Example

`GET /api/obs/health`

Responses

Example

`GET /api/obs/graph`

Responses

Example

`GET /api/obs/stream`

Connection

Event catalog

Example

`POST /api/obs/ingest`

Responses

Example

`POST /api/eval/smoke`

Responses

Example

Error envelope

See also