> ## Documentation Index
> Fetch the complete documentation index at: https://docs.claw.boo/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability API

> REST reference for the orchestration event log: feed, traces, errors, fleet health, graph projection, SSE live-tail, client mirror, and eval smoke run.

REST surface over the durable orchestration event log: query the raw event feed, reconstruct a single trace with metrics, pull the harness-bug error feed, read the fleet-health triage, fold the event-sourced delegation graph, live-tail the log over Server-Sent Events, and mirror client-observed runtime events back into the log. This group also covers `POST /api/eval/smoke`, the on-demand deterministic eval run.

Observability is always on; there is no feature gate, and these handlers serve unconditionally. The event log is the single read source: a trace is all events sharing a `traceId`, the graph is a projection of the ordered stream, and metrics/error-taxonomy/fleet-health all fold from the same rows. All POST routes read a JSON body parsed by `express.json({ limit: '2mb' })`.

<Info>
  Every payload-bearing response redacts the event's JSON `data` field before it is sent; credential-shaped keys/values are masked with `••••`. This is display-layer defense in depth on top of the storage-layer secret scrub; numeric telemetry (token counts, cost) survives the mask. `GET /api/obs/stream` is **Server-Sent Events**, not request/response; it is documented below with an event-stream catalog, not a JSON response body.
</Info>

## Routes

| Method | Path                       | Summary                                               | Stream? |
| ------ | -------------------------- | ----------------------------------------------------- | ------- |
| GET    | `/api/obs/events`          | Query the orchestration event feed                    | No      |
| GET    | `/api/obs/traces/:traceId` | One reconstructed trace + aggregate metrics           | No      |
| GET    | `/api/obs/errors`          | The error-taxonomy feed (harness-bug filter)          | No      |
| GET    | `/api/obs/health`          | Fleet-health triage (working/idle/stalled/zombie)     | No      |
| GET    | `/api/obs/graph`           | Event-sourced delegation/status/cost graph projection | No      |
| GET    | `/api/obs/stream`          | SSE live-tail of the log, scoped by team/task/agent   | SSE     |
| POST   | `/api/obs/ingest`          | Mirror client-observed runtime events into the log    | No      |
| POST   | `/api/eval/smoke`          | Run the deterministic eval smoke suite                | No      |

***

## `GET /api/obs/events`

Returns rows from the event log filtered by the query params, ordered by the monotonic `seq` cursor. The default order is `asc` (causal order, so a feed reads chronologically); pass `order=desc` for a recent-first feed.

* **Path params**: none.
* **Query params**:

| Param      | Type            | Notes                                                            |
| ---------- | --------------- | ---------------------------------------------------------------- |
| `teamId`   | string          | Match events for one team.                                       |
| `taskId`   | string          | Match events for one board task.                                 |
| `agentId`  | string          | Match events for one agent (the per-agent activity scope).       |
| `traceId`  | string          | Match events sharing a trace.                                    |
| `kinds`    | string          | Comma-separated list of event kinds; empty/blank values dropped. |
| `since`    | number          | `ts >=` (wall-clock ms), a recent-window read.                   |
| `afterSeq` | number          | `seq >` cursor (strictly monotonic, collision-free).             |
| `limit`    | number          | Row cap, clamped to 1-5000 (defaults to 500 when omitted).       |
| `order`    | `asc` \| `desc` | `desc` only when literally `"desc"`; anything else is `asc`.     |

* **Request body**: none.

### Responses

**`200 OK`**: the matched events. Each row is the full stored shape with its `data` field redacted (still a JSON string after masking):

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  events: Array<{
    seq: number
    id: string
    ts: number
    kind: OrchestrationEventKind
    teamId: string | null
    taskId: string | null
    agentId: string | null
    runtime: string | null
    traceId: string | null
    spanId: string | null
    parentSpanId: string | null
    correlationId: string | null
    data: string // JSON string, credential keys/values masked with ••••
    tenantId: string | null
    createdAt: number
  }>
}
```

`OrchestrationEventKind` is one of: `task_created`, `task_claimed`, `status_changed`, `comment_added`, `dep_linked`, `execution_started`, `execution_completed`, `tool_call`, `tool_result`, `cost`, `approval_requested`, `approval_resolved`, `error`, `span_start`, `span_end`, `session_rotated`, `routine_fired`, `routine_dispatched`, `routine_completed`, `routine_error`, `team_chat_post`, `speaker_selected`, `turn_bound_hit`. See [Events & errors reference](/reference/events-and-errors) for the per-kind `data` shapes.

**`500 Internal Server Error`**: any read failure:

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
# Last 100 cost + error events for a team, recent first
curl "http://localhost:18790/api/obs/events?teamId=<team-uuid>&kinds=cost,error&order=desc&limit=100"
```

***

## `GET /api/obs/traces/:traceId`

Reconstructs one trace, every event sharing the `traceId`, ordered `seq` ASC (causal). The full multi-agent task renders from this (leader → specialists → tool spans). Aggregate metrics are computed from the un-redacted events first, then each event's `data` is masked for display, so numeric cost/token telemetry in `metrics` is accurate.

* **Path params**: `traceId` (string).
* **Request body**: none.

### Responses

**`200 OK`**: the trace events plus the aggregate metrics:

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  traceId: string
  events: Array<{/* same redacted row shape as GET /api/obs/events */}>
  metrics: {
    totalCostUsd: number
    inputTokens: number
    outputTokens: number
    toolErrorRate: number // failed tool results / total tool results (0 when none)
    toolCalls: number
    toolErrors: number
    eventCounts: Partial<Record<OrchestrationEventKind, number>>
    activeAgents: number
    tokensPerMinute: number // output tokens/min over the window (0 when window < 1s)
  }
}
```

The trace read is capped at 5000 events. An unknown `traceId` returns `200` with an empty `events` array and zeroed `metrics`.

**`500 Internal Server Error`**:

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
curl http://localhost:18790/api/obs/traces/<trace-id>
```

***

## `GET /api/obs/errors`

The error-taxonomy feed: every `error` event, recent-first, projected to a compact shape. An error whose class is `Unknown` is a harness bug; pass `harnessBug=true` to filter to those.

* **Path params**: none.
* **Query params**:

| Param        | Type   | Notes                                                                     |
| ------------ | ------ | ------------------------------------------------------------------------- |
| `harnessBug` | string | `"true"` filters to harness-bug errors only; any other value returns all. |
| `since`      | number | `ts >=` (wall-clock ms).                                                  |

* **Request body**: none.

The read is fixed to `kinds: ['error']`, `limit: 500`, newest-first **by `ts`**. Wall-clock order is what a display feed wants, and it is the order the only index over `kind` can serve, so the cost stays proportional to the 500 rows returned rather than to every error ever recorded. The `harnessBug` filter is applied after projection.

### Responses

**`200 OK`**: the error rows plus the unfiltered harness-bug count. The whole payload is run through `redactObject`:

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  errors: Array<{
    seq: number
    ts: number
    taskId: string | null
    agentId: string | null
    runtime: string | null
    errorClass: string // from event data; defaults to 'Unknown'
    harnessBug: boolean // from event data
    message: string // from event data; defaults to ''
  }>
  harnessBugCount: number // count across ALL errors in the window, before the filter
}
```

`harnessBugCount` always reflects the full window (harness bugs found regardless of the `harnessBug` filter), so a UI can badge the alert count even while showing the unfiltered feed.

**`500 Internal Server Error`**:

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
# Harness-bug alerts in the last hour
curl "http://localhost:18790/api/obs/errors?harnessBug=true&since=$(($(date +%s%3N) - 3600000))"
```

***

## `GET /api/obs/health`

Fleet-health triage, a per-agent state folded from the event log, time-sensitive (computed against `Date.now()`). An agent with an open execution is `working` if a recent event landed, `stalled` if quiet past 5 minutes, and `zombie` if quiet past 30 minutes (the process is almost certainly dead, what orphan reconciliation reaps); an agent with no open execution is `idle`.

* **Path params**: none.
* **Query params**: `teamId` (string), scope to one team.
* **Request body**: none.

The read is capped at the **most recent 5000 events** for the requested scope. `orchestration_events` is append-only and is never pruned, so the triage folds a trailing window rather than the whole history: rows are selected newest-first, then folded in causal order. An agent whose activity has scrolled out of that window stops appearing.

### Responses

**`200 OK`**: one entry per agent that appears in the window:

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  agents: Array<{
    agentId: string
    status: 'working' | 'idle' | 'stalled' | 'zombie'
    lastEventTs: number
    activeTaskId: string | null
    openExecutions: number
    costUsd: number
  }>
}
```

**`500 Internal Server Error`**:

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
curl "http://localhost:18790/api/obs/health?teamId=<team-uuid>"
```

***

## `GET /api/obs/graph`

Folds the ordered event stream into a delegation/status/cost graph projection. The team graph is a pure projection of the log (replaying the same log always reproduces the same graph), so the surface cannot drift from reality. Two views fold from one ordered list: the task-delegation graph and the derived agent-to-agent delegation graph.

* **Path params**: none.
* **Query params**: `teamId` (string), scope to one team.
* **Request body**: none.

The read is capped at the **most recent 5000 events** for the requested scope. `orchestration_events` is append-only and is never pruned, so the projection folds a trailing window rather than the whole history: rows are selected newest-first, then folded in causal order. A task whose events have all scrolled out of that window stops appearing, and one only partly inside it projects from the events that remain.

### Responses

**`200 OK`**: the projected graph (this response is the projection output verbatim, not wrapped or redacted; it carries no raw event `data`):

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  tasks: Array<{
    id: string
    title: string | null
    status: string
    assigneeAgentId: string | null
    parentTaskId: string | null
    runtime: string | null
    teamId: string | null
    costUsd: number
  }>
  taskEdges: Array<{
    id: string
    source: string
    target: string
    kind: 'delegation' | 'dependency'
  }>
  agents: Array<{
    id: string
    costUsd: number
    taskIds: string[]
  }>
  agentEdges: Array<{
    id: string
    source: string
    target: string
    kind: 'delegation' | 'dependency' // agent→agent edges are always 'delegation'
  }>
}
```

**`500 Internal Server Error`**:

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
curl "http://localhost:18790/api/obs/graph?teamId=<team-uuid>"
```

***

## `GET /api/obs/stream`

Server-Sent Events live-tail of the event log, scoped by team/task/agent. The handler opens an `text/event-stream` response and polls the log every 750 ms on the monotonic `seq` cursor, pushing any new rows. It is cross-process correct (it catches writes from the standalone MCP stdio bins, not just the in-process server) and indexed. Resume from a known position via the standard `EventSource` `Last-Event-ID` header or the `?since=<seq>` query param.

* **Path params**: none.
* **Query params**:

| Param     | Type   | Notes                                                                                                       |
| --------- | ------ | ----------------------------------------------------------------------------------------------------------- |
| `teamId`  | string | Scope the tail to one team.                                                                                 |
| `taskId`  | string | Scope the tail to one board task.                                                                           |
| `agentId` | string | Scope the tail to one agent.                                                                                |
| `since`   | number | Resume cursor (`seq`). Negative/non-finite → `0`. The `Last-Event-ID` header takes precedence when present. |

* **Request body**: none.

<Note>
  This is an SSE route, not request/response. There is no JSON response body; the catalog below describes the wire frames. Each polled batch reads up to 500 rows past the cursor (`order: 'asc'`). The cursor only advances forward, so resuming with a stale `since`/`Last-Event-ID` replays from there. The stream is cleaned up on `req`/`res` close.
</Note>

### Connection

On open, the handler writes `HTTP/1.1 200` with:

```http theme={"theme":{"light":"github-dark","dark":"github-dark"}}
Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no
```

then emits a `: connected` comment frame and flushes any rows already past the cursor.

### Event catalog

| Frame         | When                                | Shape                                                                              |
| ------------- | ----------------------------------- | ---------------------------------------------------------------------------------- |
| `: connected` | Immediately on open                 | SSE comment (ignored by `EventSource`)                                             |
| event data    | Per new log row (poll every 750 ms) | `id: <seq>` line + `data: <json>` line (the full event row, `data` field redacted) |
| `: keepalive` | Every 20 s while open               | SSE comment (keeps the connection warm)                                            |

The `data:` payload is one stored event row, identical in shape to a `GET /api/obs/events` row, with its inner JSON `data` field masked (`••••`). The `id:` value is the row's `seq`; the browser sends it back as `Last-Event-ID` on auto-reconnect, so the tail resumes without gaps.

Example frames:

```text theme={"theme":{"light":"github-dark","dark":"github-dark"}}
: connected

id: 4187
data: {"seq":4187,"id":"...","ts":1718450000000,"kind":"tool_call","taskId":"...","agentId":"...","runtime":"claude-code","traceId":"...","data":"{\"toolName\":\"read_file\"}","tenantId":null,"createdAt":1718450000000}

: keepalive
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
curl -N "http://localhost:18790/api/obs/stream?taskId=<task-uuid>"

# Resume from a known seq
curl -N "http://localhost:18790/api/obs/stream?taskId=<task-uuid>&since=4187"
```

***

## `POST /api/obs/ingest`

Mirrors client-observed runtime events into the durable log. The OpenClaw runtime is observed in the browser (the server never sees those frames), so the SPA forwards them here to keep the activity terminal uniform across runtimes. Ingestion is **whitelisted** to the three per-tool kinds the browser legitimately observes: `tool_call`, `tool_result`, `error`. Board lifecycle events (`task_created`, `status_changed`, `execution_*`, …) are emitted server-side by the board REST handlers and are never accepted here. Each event is best-effort: a malformed row is skipped, never failing the batch.

* **Path/query params**: none.
* **Request body**:

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  events?: Array<{
    kind: 'tool_call' | 'tool_result' | 'error'  // any other kind is dropped
    ts?: number              // must be within [now - 24h, now + 60s], else server time is used
    teamId?: string | null
    taskId?: string | null
    agentId?: string | null
    runtime?: string | null   // defaults to 'openclaw' when omitted
    data?: Record<string, unknown>  // non-object → {}
  }>
}
```

A non-array `events` is treated as empty. At most 200 events are accepted per call (the rest are sliced off). Any event whose `kind` is missing or not in the whitelist is skipped.

A supplied `ts` is clamped to a band around server time (60 s ahead for clock skew, 24 h behind) and replaced with server time when it falls outside. `ts` is not just metadata: fleet health derives staleness from it, so a future timestamp would pin an agent at `working` and mask a genuine `zombie`. A mirror reports what a browser just observed, so a timestamp far from now is wrong regardless of intent.

### Responses

**`200 OK`**: the count of events actually appended (after whitelist + slice):

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "ok": true, "count": 3 }
```

**`400 Bad Request`**: an unexpected throw while ingesting (the error string is redacted):

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
curl -X POST http://localhost:18790/api/obs/ingest \
  -H 'Content-Type: application/json' \
  -d '{"events":[{"kind":"tool_call","taskId":"<task-uuid>","agentId":"<agent-id>","data":{"toolName":"web_search"}}]}'
```

***

## `POST /api/eval/smoke`

Runs the deterministic eval smoke suite (`SMOKE_TASKS`), the exact subset CI runs, and returns the real `SuiteReport`. It uses no live model, no provider keys, no executor/RuntimeAdapter, and no network; each trial gets its own temp-dir SQLite board (disjoint from the real `clawboo.db`), and the contexts are cleaned up after the run. The ablation self-test (variants over the harness's own subsystem flags) runs only from the manual `evals.yml` workflow; it is explained in the UI but never driven from this route, and no live-model judge is wired into it yet.

* **Path/query params**: none.
* **Request body**:

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  trials?: number  // clamped to [1, 3]; non-finite/missing → 1
  k?: number       // clamped to [1, 3]; non-finite/missing → trials
}
```

Both inputs are floored and clamped so the route can never be turned into a load generator.

### Responses

**`200 OK`**: the suite report:

```ts theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{
  tasks: Array<{
    taskId: string
    suite: 'capability' | 'regression'
    kind: 'coding' | 'research' | 'coordination'
    trials: Array<{
      taskId: string
      passed: boolean
      score: number
      graders: Array<{/* GraderResult */}>
    }>
    passAt1: number // per-trial success rate = empirical pass@1
    passPowK: number // probability all k trials pass = passAt1^k
    meanScore: number // mean partial-credit score across trials
  }>
  passAt1: number // macro-averaged pass@1 across tasks
  passPowK: number // macro-averaged pass^k across tasks
  k: number
}
```

**`500 Internal Server Error`**: the run threw (contexts are still cleaned up in `finally`):

```json theme={"theme":{"light":"github-dark","dark":"github-dark"}}
{ "error": "<message>" }
```

### Example

```bash theme={"theme":{"light":"github-dark","dark":"github-dark"}}
curl -X POST http://localhost:18790/api/eval/smoke \
  -H 'Content-Type: application/json' \
  -d '{"trials":3,"k":2}'
```

***

## Error envelope

Every error response on these routes is the standard envelope `{ error: string }`. On the obs routes the error string is run through the display redactor before it is sent. The two success-shaped exceptions are `/api/obs/ingest` (`{ ok: true, count }`) and `/api/eval/smoke` (the bare `SuiteReport`).

## See also

* [Observability concepts](/concepts/observability), the event log, traces, the Ghost-Graph projection, and fleet-health triage
* [Events & errors reference](/reference/events-and-errors), the orchestration event kinds and the error taxonomy
* [Observability dashboard](/using/observability-dashboard), the UI over these endpoints (traces, errors, fleet health, the eval scorecard)
* [Board API](/reference/rest-api/board), the board lifecycle events this log records
* [REST API overview](/reference/rest-api/index)