@clawboo/evals

Version 0.1.0
Purity server-only (private, node:fs + node:os for throwaway sqlite boards via @clawboo/db; private: true, never imported from the SPA bundle)
Purpose The eval harness turned inward on clawboo’s OWN orchestration: a task drives the orchestration against a clean throwaway board, then GRADES the final board state + event log (the outcome, not the narration), reporting pass@1 / pass^k and the marginal-contribution ablation scorecard.
Workspace deps @clawboo/db, @clawboo/executor, @clawboo/governance, @clawboo/obs
External deps zod ^3.25.0

Each trial runs against its own temp-dir sqlite board (isolation; leftover state causes correlated failures). pass@1 = ≥1 of k trials succeeds; pass^k = all k succeed; pass^k is the production-readiness bar. The package exposes a single . entry point (no subpath exports); the barrel re-exports six modules: ./types, ./env, ./runner, ./ablation, ./graders, ./tasks.

Scorecards + ablation results are written outside the repo. Graders come in three families (code / model / human); only code and model graders are implemented here. The model (LLM-as-judge) grader is non-deterministic + priced and is not yet wired into any task (a deferred live path), so nothing runs it today; never the PR smoke subset.

Public API

Functions

env (./env)

Signature	Contract
`makeBoardContext(flags?: EvalFlags): EvalContext`	Build a CLEAN context backed by a throwaway sqlite board in a fresh `mkdtemp` dir. Call once per trial. Defaults to `DEFAULT_FLAGS` (both subsystems on).
`cleanupEvalContexts(): void`	Remove every temp board created since the last cleanup (call in `afterAll` / a route `finally`).

runner (./runner)

Signature	Contract
`runTask(task, makeCtx, opts?): Promise<TaskReport>`	Run one task K times, each against a fresh ctx (`makeCtx` MUST build a clean board). Grades the outcome; reports `passAt1` = passes/K and `passPowK = passAt1^k`. Default K=1; `binary` scoring needs all graders to pass, `weighted` needs mean ≥ threshold (default 0.7). A thrown `run` becomes a single failing `run-error` grader.
`runSuite(tasks, makeCtx, opts?): Promise<SuiteReport>`	Run every task; macro-averages `passAt1` and `passPowK` across tasks. `k` defaults to `opts.k ?? opts.trials ?? 1`.

graders/code (./graders → ./code); fast, objective, deterministic; inspect the board + event log, not the transcript.

Signature	Contract
`boardStateGrader(task, expectStatus, label?): Grader`	Pass when the board task’s status is one of `expectStatus` (`missing` when absent). `task` is a static id or a resolver `(outcome) => id`; `label` names the result. Wired into the regression + capability tasks.
`logParseGrader(kind, minCount?, filter?): Grader`	Pass when an `OrchestrationEventKind` was recorded ≥ `minCount` times (default 1), optionally filtered by `taskId` (via `listEvents`). Reads the obs event log, which the board-CRUD tasks don’t emit, so it’s a seam for the deferred event-emitting path.
`readyGrader(task, shouldBeReady, teamId?, label?): Grader`	Pass when the task’s presence in `getReadyTasks` matches `shouldBeReady`, the dep-gate check. `task` is a static id or an outcome resolver. Wired into `reg-dep-gate`.
`outcomeGrader(name, predicate): Grader`	Free-form predicate over `(outcome, ctx)`; a `number` return is clamped to `[0,1]` (partial credit), a `boolean` maps to 1/0. Passes at score ≥ 1.
`eventBudgetGrader(maxEvents, filter?): Grader`	Pass when the recorded event count ≤ `maxEvents` (a transcript-cost bound). Like `logParseGrader`, a seam for the deferred event-emitting path.

graders/model (./graders → ./model), LLM-as-judge for subjective dimensions; defined but not yet wired into any task (the deferred live path).

Signature	Contract
`llmJudgeGrader(opts: LlmJudgeOptions): Grader`	One isolated judge per dimension. Builds a judge prompt (`buildJudgePrompt`), drives the caller-supplied `RuntimeAdapter` reusing `@clawboo/obs` `driveStructuredJudge`, and parses `{ score, reason? }` against `z.number().min(0).max(1)`. Passes at `score ≥ threshold` (default 0.6); an unparsed verdict scores 0.

ablation (./ablation)

Signature	Contract
`runAblation(opts: RunAblationOptions): Promise<AblationScorecard>`	Hold the harness fixed, run the 4 variants (`full` / `-verifier` / `-structured` / `none`); the capability tasks read the flags, so each “marginal contribution” is the harness’s scripted response to the flag, a self-test of the ablation wiring, not a live-orchestrator measurement. Default 3 trials.

Types & interfaces

types (./types)

Name	Shape / contract
`EvalSuite`	`'capability' \| 'regression'`.
`EvalKind`	`'coding' \| 'research' \| 'coordination'`.
`EvalFlags`	`{ verify: boolean; structuredState: boolean }`, the two harness subsystems the ablation toggles.
`EvalContext`	`{ db: ClawbooDb; flags: EvalFlags }`, a clean throwaway board + the active flags.
`TrialOutcome`	`{ summary?: string; data?: Record<string, unknown> }`, the final environment state (outcome, not claim).
`GraderResult`	`{ name; passed: boolean; score: 0..1; detail? }`, one grader’s verdict, with partial credit.
`Grader`	`(ctx, outcome) => GraderResult \| Promise<GraderResult>`.
`EvalTask`	`{ id; suite; kind; description; run(ctx) => Promise<TrialOutcome>; graders: Grader[]; referenceNote?; scoring?: 'binary' \| 'weighted'; threshold?; smoke?; tags? }`, `run` builds the clean env + drives the orchestration.
`Trial`	`{ taskId; passed; score; graders: GraderResult[] }`, one attempt.
`TaskReport`	`{ taskId; suite; kind; trials; passAt1; passPowK; meanScore }`, `passAt1` empirical, `passPowK = passAt1^k`.
`SuiteReport`	`{ tasks: TaskReport[]; passAt1; passPowK; k }`, macro-averaged across tasks.

runner (./runner)

Name	Shape / contract
`RunOptions`	`{ trials?: number; k?: number }`, K trials per task (default 1); `k` exponent defaults to `trials`.

graders/model (./graders → ./model)

Name	Shape / contract
`LlmJudgeOptions`	`{ name; dimension; rubric; makeAdapter(): RuntimeAdapter; model?; threshold? }`, one judge scores one dimension; the caller owns the reviewer adapter.

ablation (./ablation)

Name	Shape / contract
`AblationCell`	`{ variant; flags: EvalFlags; passAt1; passPowK }`, one variant’s result.
`AblationContribution`	`{ subsystem: 'verifier' \| 'structured-state'; deltaPassAt1 }`, mean pass@1 drop when removed.
`AblationScorecard`	`{ baselinePassAt1; cells: AblationCell[]; contributions; trials }`, `baselinePassAt1` is the full-harness pass@1.
`RunAblationOptions`	`{ tasks: EvalTask[]; makeCtx(flags): Promise<EvalContext>; trials? }`, `makeCtx` builds a clean ctx carrying the variant’s flags.

Constants

Name	Value / contract
`DEFAULT_FLAGS`	`{ verify: true, structuredState: true }`, the full-harness default (`./env`).
`ALL_TASKS`	`[...REGRESSION_TASKS, ...CAPABILITY_TASKS]`, every task (`./tasks`).
`SMOKE_TASKS`	`ALL_TASKS.filter((t) => t.smoke)`, the cheap, deterministic, no-live-model PR subset.
`ABLATION_TASKS`	`CAPABILITY_TASKS`, the ablation-sensitive set (success depends on a toggled subsystem).
`REGRESSION_TASKS`	4 load-bearing-guarantee snapshots: `reg-claim-409-no-retry`, `reg-dep-gate`, `reg-report-up`, `reg-state-machine`. All smoke; target pass ≈100%.
`CAPABILITY_TASKS`	3 capability evals: `cap-cross-runtime-resume` (reads the structured-state flag), `cap-verification-catches-bug` (reads the verifier flag), `cap-delegation-fanout`. All smoke + code-graded.

Classes

None; this package exports only functions, types, and constants.

Used by

apps/web (server), api/evalSmoke.ts (POST /api/eval/smoke) imports SMOKE_TASKS, makeBoardContext, runSuite, cleanupEvalContexts to run the deterministic suite on ephemeral boards on demand from the Observability dashboard. The SPA never imports the package directly; apps/web/src/lib/evalsClient.ts mirrors the SuiteReport / TaskReport shapes over REST.
CI, .github/workflows/evals.yml (workflow_dispatch) builds @clawboo/evals + its deps and runs the deterministic suite + the ablation self-test; no provider keys are used (the live-model grader is not yet wired into any task).

Source

Barrel: packages/evals/src/index.ts (re-exports ./types, ./env, ./runner, ./ablation, ./graders, ./tasks).

Reference

REST API

Packages

Public API

Functions

Types & interfaces

Constants

Classes

Used by

Source

See also

​Public API

​Functions

​Types & interfaces

​Constants

​Classes

​Used by

​Source

​See also

Public API

Functions

Types & interfaces

Constants

Classes

Used by

Source

See also