Testing strategy

Clawboo is a team-first orchestrator: many agents write one SQLite file, five runtimes execute heterogeneous work, and the whole thing ships as a single bundled CLI that has to boot on a stranger’s machine. Each of those facts has a matching test layer. This page explains the layers, why each exists, and the invariants the standing guard tests freeze in place; so you can extend the suite without re-learning the same lessons the suite was written to encode. The full gate is six commands, all green: pnpm lint, pnpm typecheck, pnpm test, pnpm e2e, pnpm verify:catalog, and pnpm assemble && pnpm test:clean-install. Every one of them runs as a parallel CI job (pnpm build is a seventh job); the last is the clean-install simulation, run on Ubuntu, Windows, and macOS. The strategy described below is deliberately phrased in terms of suites and intent, not exact test counts; counts change every session, and a doc that pins them goes stale on the next commit.

​

The model

Each layer answers a question the others can’t:

• Vitest (two projects): does the logic hold, and do the server’s stateful subsystems behave against real git and real SQLite?
• Playwright e2e: does the assembled product render and round-trip in a browser, without ever touching the developer’s real state?
• The clean-install smoke: does a packed-and-installed npx clawboo actually reach a working dashboard, including the stdio MCP bins and a real agent run?
• The eval harness: does the orchestrator itself still satisfy its load-bearing guarantees, measured as an outcome rather than a narration?
• The standing guards: do the repo’s structural invariants (schema source-of-truth, publish posture, source hygiene) still hold?

​

Vitest: two projects, one run

apps/web/vitest.config.ts defines two Vitest projects (a Vitest 3.x feature). One vitest run runs both; splitting them keeps the heavier component transforms from flipping the node-environment server and store suites. The node project runs every src/**/*.test.ts and server/**/*.test.ts in a Node environment. This is where the bulk of Clawboo’s logic lives: pure functions, Zustand stores, policy reducers, the board state machine, and where the server’s stateful subsystems get integration-tested against real infrastructure rather than mocks. The worktree orchestrator test drives the full claim → provision → handoff → complete lifecycle against a real temporary git repository and a real SQLite board, with $HOME overridden to a throwaway sandbox so it never touches the developer’s ~/.openclaw or ~/.clawboo. The all-on executor-runner integration test exercises board, executors, worktrees, verification, governance, and observability together in one flow, the cross-subsystem interactions (a verify gate fed by a runner-provisioned worktree; a budget recorded alongside an obs trace; a governance halt correctly skipping verify) that no per-subsystem test reaches on its own. Because these run real git and real SQLite, they get a 30-second testTimeout / hookTimeout, headroom for tests that take a few seconds each in isolation and could be starved past the 5-second default when the jsdom project runs concurrently in the same vitest run. The jsdom project runs every src/**/*.test.tsx in a jsdom environment with @vitejs/plugin-react. These are React Testing Library component tests over the dashboard panels; each one asserts render, a nav-gate or interaction, and (via msw) the right network behavior. Its setup file (src/__vitest__/setup.ts) does four things: registers @testing-library/jest-dom matchers, registers the jest-axe toHaveNoViolations accessibility matcher, wires the shared msw request-mock server, and shims the jsdom gaps the panels touch on render (matchMedia, ResizeObserver). Its testTimeout is widened to 15 seconds because the jest-axe accessibility sweeps are CPU-heavy and can be starved under concurrent load. The load-bearing detail is in the msw wiring. The shared server (src/__vitest__/mswServer.ts) is started with onUnhandledRequest: 'error', so any /api/* call a component makes without a matching handler fails the test loudly. That turns “this panel makes zero fetches on render” into a guarantee the component test itself encodes, not something only the e2e can prove. One default handler covers the single cross-origin call the panels make on mount, the GitHub star count; so that one request never reaches the real network and the strict same-origin policy can stay strict.

​

Playwright e2e: the mock gateway, and never your real data

playwright.config.ts drives the assembled product. Its webServer.command builds the UI bundle and starts the Express server in production mode (build:ui && start), pinned to port 19999, well outside the regular auto-fallback window (18790–18809) so a developer’s running pnpm dev instance never collides with a test run. The build timeout is bumped to 180 seconds because a cold vite build takes 80–120 seconds on the macOS dev box. The specs use a mock WebSocket gateway (tests/e2e/helpers/mockGateway.ts) instead of a real OpenClaw Gateway. It’s a small ws server that answers the handshake (connect, agents.list, agents.files.read) and, crucially for the board round-trip spec, pushes synthetic chat event frames after a chat.send. The synthesized reply is deterministic and role-aware: the leader given a normal user message replies with a structured <delegate> block (which derives a board task), the leader given a [Task Update] reflection replies with a plain synthesis (no delegate, so it can’t loop), and the specialist replies with a report-up summary (which drives the task to done). That lets a single Playwright spec drive a real chat → board → chat round-trip end-to-end over the mock.

​

Sandboxed HOME isolation

The most important thing the e2e config does is never touch the developer’s real state. The server’s SQLite path is derived from os.homedir(), and the fixtures run a DELETE-loop over /api/teams to clean stale state between runs. Without isolation, that loop would hit the developer’s actual ~/.openclaw/clawboo/clawboo.db, so the e2e suite is sandboxed in three layers:

1. mkdtemp at config load. playwright.config.ts creates a per-run sandbox directory under the OS temp root and overrides three environment variables on the spawned server via webServer.env: HOME, OPENCLAW_STATE_DIR (OpenClaw interop reads), and CLAWBOO_HOME (Clawboo’s own SQLite DB, settings, secrets vault, and worktrees). The server’s state therefore lands entirely inside the sandbox.
2. globalTeardown cleanup. After the run finishes or fails, tests/e2e/globalTeardown.ts removes the sandbox, but only after asserting the path lives under the OS temp dir, silently no-op’ing otherwise rather than risking an rm -rf of real data.
3. assertSandboxed guard rail. Before any destructive helper runs, assertSandboxed (in helpers/fixtures.ts) makes two checks: that the test-runner env’s sandbox markers point under the temp root, and that a live GET /api/system/status reports a stateDir under the temp root. The second check catches the nasty case where Playwright reused a stale, unsandboxed server that a developer started manually via pnpm dev or pnpm start; the guard refuses to proceed and tells the developer exactly how to recover. Every helper that touches a destructive endpoint calls it; new destructive helpers must too.

The specs cover the surfaces a smoke pass cares about: connection and auto-connect, the fleet list and agent detail view, the Ghost Graph canvas, team navigation, the group-chat onboarding gate and two-row layout, the chat → board → chat round-trip, a mid-session Gateway drop and self-healing reconnect (driven through the real /api/gateway/ws proxy), the chat transcript’s bounded render window and its “Load earlier messages” scroll-anchor restore, the eval-run-from-UI button, and the native-first and coding-agent onboarding flows. The group-chat helper can pre-mark the “Know Your Team” onboarding flags complete so most specs skip the gate, while the gate-specific test exercises it directly.

​

The clean-install smoke: does npx clawboo actually work

pnpm test:clean-install runs scripts/test-clean-install.mjs, which simulates npx clawboo on a stranger’s machine and asserts the published artifact reaches a working dashboard. It exists because Clawboo shipped two releases broken in ways every other test passed: a release where the bundled server returned Cannot GET / because the Express 5 SPA catch-all pattern didn’t match the bare /, and a release where the CLI’s port discovery did a TCP-only probe and mistook a foreign listener on an adjacent port for Clawboo, routing the browser to an “Unauthorized” page. A third bug of the same family is covered by a unit test rather than this script, because the script structurally cannot see it: res.sendFile() was called with an absolute path and no root, so send split the whole path into segments and 404’d on any segment beginning with a dot. npx installs under ~/.npm/_npx/…, so the .npm segment broke every deep route for real users while / still worked. CI never reproduces it, since runners check out to a dot-free path, so apps/web/server/lib/__tests__/serveSpa.test.ts runs the real mountSpa against both a plain and a dot-containing install path.

​

It runs against a real install, not the repo build

The script starts by pnpm packing apps/cli and npm installing the resulting tarball into a throwaway directory under the OS temp dir. Everything after that runs against that install. This is the load-bearing detail: running the repo’s apps/cli/dist/index.js in place lets Node’s module resolution walk up into the monorepo’s node_modules, so a module the bundle still loads at runtime resolves in the repo and is ERR_MODULE_NOT_FOUND on a user’s machine. Installing a packed tarball outside the workspace removes that escape hatch, so the published files whitelist and the published dependencies closure are what get exercised. pnpm pack (not npm pack) because pnpm changeset publish is what publishes: pnpm rewrites workspace: specifiers into concrete versions in the packed manifest, so the tarball under test is the one users download. npm install (not pnpm add) because npm’s hoisted layout is what an npx clawboo user resolves through. Before packing, the script refuses to run if anything already answers with the Clawboo signature in the 18790-18809 discovery window — the CLI would (correctly) attach to that dashboard and every assertion below would silently test the wrong server. After boot it re-proves the point structurally: the server writes its port to <clawboo home>/api-port.txt, and the run’s isolated $HOME must be the home that file lives under.

​

What it asserts

The script reproduces the exact condition the port-collision bug shipped under. It binds port 18791 with a fake service that returns 401 Unauthorized (mimicking the OpenClaw Gateway’s auxiliary-port behavior), spawns the installed CLI in an isolated state dir with an isolated $HOME and no env-var pins, and then asserts:

• the manifest still declares all five commands by name (clawboo plus the four clawboo-mcp-*), each one’s target exists in the tarball and got a node_modules/.bin shim, and the UI + third-party notices shipped;
• clawboo --version runs through that npm shim and reports the packed version — the same indirection npx clawboo uses, so a shim that exists but cannot execute (lost exec bit, a broken .cmd, a mangled shebang) fails here rather than on a user’s first run;
• every module the installed bundles still load from node_modules is a declared dependency, a Node builtin, or a documented optional external (see below);
• the CLI announces a dashboard URL that is not :18791; its HTTP-signature probe must reject the fake listener and let Clawboo’s own server pick 18790;
• GET / returns the SPA HTML (<div id="root"></div>);
• a deep SPA route (/some/spa/route) falls through to the same SPA HTML (the catch-all works);
• GET /api/settings returns Clawboo-shaped JSON (gatewayUrl string + hasToken boolean);
• GET /api/system/status returns the expected shape;
• loading / in headless Chromium mounts React ([data-testid="team-sidebar"]) and renders the fresh-install onboarding surface ([data-testid="onboarding-wizard"]), with no uncaught exception and no first-party console error;
• an installed stdio MCP bin (dist/bin/tasks.js) can be spawned and driven through a raw JSON-RPC handshake (initialize → notifications/initialized → tools/list), and its tool list includes list_tasks;
• an agent run can start: the script creates a native agent and a board task, then drives a real POST /api/runtimes/clawboo-native/run to a terminal done, and checks the report-up summary carries the provider’s reply and the task moved to done on the board;
• a second launch reuses the running dashboard: with a Clawboo-shaped listener already answering and its port recorded in api-port.txt, a second spawn of the installed CLI must attach to it — the announced port is the running one, the HTTP signature probe actually ran, and none of the fork-path banners (Starting Clawboo, Dashboard started) appear. The fake binds an ephemeral port, deliberately outside the 18790–18809 discovery window, so the ascending scan cannot reach it and a pass proves the runtime-port-file path ran rather than the scan.

The browser assertion exists because every HTTP check above it passes against a bundle that serves the right HTML and then dies on execution: matching <div id="root"></div> proves the shell was served, not that it booted. To keep it hermetic the run aborts every non-first-party request (the Google Fonts stylesheet in index.html, the api.github.com star count that mounts under the wizard) and counts console errors only when they originate first-party, so a rate-limited or unreachable CDN can never fail a release. It needs the browser binary: pnpm exec playwright install chromium once per clone, which ci.yml and publish.yml both do as an explicit step since Playwright has no install script of its own. The MCP assertion is what proves an external runtime can spawn a packaged MCP bin straight from a clean install. Note the split of responsibility on the command interface: clawboo --version is the one thing executed through the npm shim (it is the only side-effect-free subcommand — no server, no state dir, no port), while the dashboard boot and the MCP handshake run the entry files directly with node. Going through a shim for those would put cmd.exe between the harness and the child’s stdio on Windows, which is exactly the pipe the URL parsing and the JSON-RPC handshake depend on. The dispatch assertion covers the product’s main path — assign a task and it runs — so it can never be a publish-time unknown. It needs no API key and no network: the native runtime’s keyless ollama provider rides the shared OpenAI-compatible client with a base-URL override, so the script points OLLAMA_BASE_URL at a local stub that streams one canned reply. kind: 'research' keeps isolation at none, so no git repo or worktree is involved.

​

The externals-vs-dependencies check

scripts/check-bundle-externals.mjs (also runnable on its own as pnpm test:bundle-externals) statically extracts every require(...) / import(...) left in dist/index.js, dist/server.js, and dist/bin/*.js, and asserts the set is a subset of the package’s dependencies + Node builtins + a small documented-optional allowlist. It catches what booting can’t: a lazy import only fails when that feature is first used, which on a fresh install is long after CI went green. The extractor is a small hand-rolled JS scanner rather than a regex, because a regex produces false positives — the inlined ajv source contains the literal string 'require("ajv/dist/runtime/equal").default' (its standalone-codegen template) and esbuild writes module paths into comments. The scanner tracks strings, template literals with ${} nesting, comments, and regex literals so it only ever matches a call in code position, and it runs its own fixtures (including those adversarial cases) at the start of every check, since scripts/ has no CI-wired Vitest project. Its output was validated against esbuild’s own metafile for all six bundles. The allowlist lives in scripts/lib/bundle-externals.mjs and is a product decision, not a build detail: an entry means “a clean install cannot do X until the user installs this themselves”. Today it holds @opentelemetry/* (lazy, degrades to event-log-only) and @anthropic-ai/claude-agent-sdk (declaring it would add ~210 MB of platform binary to every install). Because the smoke test packs the assembled artifact, it depends on pnpm assemble having run first, which is exactly what the prepublish:check alias (pnpm verify:catalog && pnpm assemble && pnpm test:clean-install) and the CI smoke-test-bundle job do. The CI job runs it on a matrix of Ubuntu, Windows, and macOS: the Windows leg is the regression gate for the Windows-compat fixes (.cmd shim resolution, which→where, netstat-based process discovery) that a Unix-only run would never exercise, and macOS is a primary user OS.

​

The eval harness: grading the orchestrator’s own guarantees

@clawboo/evals is a private, server-only package that evaluates Clawboo’s own orchestration, not a runtime’s model output, but whether the board, the verifier, and the structured-state machinery still satisfy the guarantees they were built to satisfy. It is the layer that catches a regression unit tests would miss because the regression is behavioral across subsystems. The runner reports two numbers per task. pass@1 is “at least one of K trials succeeded”; pass^k is “all K succeeded.” pass@1 rises with K and pass^k falls, so pass^k is the production-readiness bar. Each trial runs against a clean environment; makeBoardContext mkdtemps a throwaway SQLite board per trial, because leftover state causes correlated failures, an eval cardinal sin. A thrown task body is recorded as a failed trial (run-error), never an unhandled rejection. Graders come in two flavors, and the split is deliberate:

• Code graders (graders/code.ts) inspect the outcome, the board’s final state and the orchestration event log, rather than the transcript’s claim of success. They’re fast, cheap, objective, and reproducible: a board-state assertion, an event-count assertion, a dependency-readiness check, a free-form outcome predicate with optional partial credit. These are preferred wherever the success criterion is mechanical.
• Model graders (graders/model.ts) are LLM-as-judge, for the subjective dimensions code can’t grade: coordination and handoff quality, groundedness. They reuse the shared structured-output judge from @clawboo/obs (the same one the read-only verification critic uses), score one isolated dimension per judge, and give the judge a way out. Because they need a real runtime adapter and provider keys, they are not yet wired into any task, so nothing runs them today; a live drift-canary that activates them is deferred. They never run on a PR.

The task set splits the same way: regression tasks are snapshots of load-bearing guarantees sourced from real failures the codebase hardened against (the atomic-claim-409-no-retry rule, the dependency gate, the report-up path, the state machine), all deterministic and code-graded; capability tasks are the ones whose success depends on a toggled subsystem flag. Every task today is deterministic and tagged smoke, so the same set runs in pnpm test (the PR gate) and in the manual evals.yml workflow; the ablation self-test is the extra read there. (A live-model task set is the deferred canary.) The ablation scorecard is a harness self-test, not a measurement of the live orchestrator. It holds the harness fixed and runs the capability tasks across four variants, ±verifier × ±structured-state; those tasks read the subsystem flags and behave accordingly, so each marginal contribution is the harness’s own scripted response to the flag, which confirms the ablation wiring is sound. Measuring the REAL subsystems’ contribution needs the live executor: the deterministic executorRunner integration test (apps/web/server/lib/__tests__/executorRunner.integration.test.ts) drives the real board, verifier, and worktree today, and a live-model version is the deferred canary. The ablation runs via the manual-only evals.yml workflow (workflow_dispatch), with no network and no provider keys; no live-model graders are wired into it yet.

​

The standing guards

Six guard areas in the suite aren’t testing features; they’re freezing structural invariants of the repo itself (seven test files, since repo hygiene splits tracked from untracked). They run inside pnpm test like any other test, and they fail the build the moment an invariant drifts. Schema source-of-truth (packages/db/src/__tests__/schemaSource.test.ts). Clawboo has no migration ladder; the CREATE TABLE IF NOT EXISTS DDL in ensureSchema (packages/db/src/schemaBootstrap.ts) is the sole schema-creation source. The Drizzle schema.ts is the type layer over the same tables, used for typed queries but never to apply migrations. Nothing keeps the two descriptions in sync automatically, so this test does: it builds a DB via the real createDb() and asserts every schema.ts table and its column set matches the live DDL, and vice-versa (the FTS5 virtual table and its shadow tables are excluded; they’re raw DDL not modellable in schema.ts). It also pins the posture decision: the package.json files array must not ship migration metadata, the db:migrate and db:generate scripts must not exist, and no drizzle/ migration directory may sit on disk. Schema reconcile (packages/db/src/__tests__/schemaReconcile.test.ts + apps/web/server/lib/__tests__/schemaUpgrade.test.ts). CREATE TABLE IF NOT EXISTS skips the whole statement when the table exists, so a new column would be a silent no-op on every database created before it, surfacing as a runtime SQL error rather than a boot failure. The package-level suite pins the three properties that make the fix trustworthy: the DDL parser sees exactly the columns SQLite creates from the same DDL (which is what makes parsing the schema, rather than hand-listing migrations, safe); a populated database missing a column is brought up to date without losing its rows, including when an index depends on that column; and a column that cannot be added fails loudly with an actionable message. The apps/web half proves the wiring, that the server’s own getDb() boot path applies it against a real file and that the boot probe agrees afterward. Both synthesise the “older database” by seeding rows and then removing what a later release would have added, because SQLite’s ADD COLUMN guards are row-dependent: an empty fixture passes while a real user’s populated file fails. A third assertion is the build-time gate: packages/db/src/__tests__/schemaBaseline.ts records the column set as of the last review, and the suite fails if any column added since could not be ALTERed into a database that already exists. That one exists because the runtime gate only fires against a database that is missing the column, and no fresh-install test database ever is, so without it shipping foo TEXT NOT NULL on an existing table passes CI and breaks every upgrade. The baseline needs no upkeep: a column added after the snapshot reads as new and is held to the addability rule, which is the rule regardless. Publish posture (apps/web/server/lib/__tests__/packagePosture.test.ts). Clawboo ships as a single npx clawboo CLI that inlines its workspace packages into one bundle; nothing else is released to npm. This guard asserts two invariants: the only non-private workspace package is clawboo (apps/cli); every other @clawboo/* package is private: true; and no non-private package has a runtime dependency on a private one (which would publish a manifest pointing at packages that never get published, so npm install would 404). The first assertion is exact: it expects the non-private set to equal ['clawboo (apps/cli)'], so adding a publishable package or accidentally un-privatizing a library trips it immediately. Layer boundaries (apps/web/server/lib/__tests__/importBoundary.test.ts). apps/web/server (Express, tsup-bundled for Node) and apps/web/src (the Vite browser SPA) are separate build targets, and packages are the shared layer that must never depend on apps. eslint.config.mjs enforces all three directions with no-restricted-imports, plus a no-restricted-syntax selector for dynamic import() (which no-restricted-imports does not visit). This guard runs the real config through ESLint’s programmatic API against synthetic sources, asserting each violation shape is flagged and legitimate imports are not, so a bare specifier like react-dom/server never trips the SPA rule. Its load-bearing assertion is the last one: apps/web’s lint script must still cover server/. That tree was historically unlinted ("lint": "eslint src/"), so narrowing the script back would silently kill the server-side rule while every other test still passed. The single sanctioned crossing, the prompt drift-guard at apps/web/src/features/teams/__tests__/nativeTeamPrompts.parity.test.ts, carries an inline eslint-disable-next-line so the exemption is visible at the site rather than buried in config. Repo hygiene (repoHygiene.test.ts + repoHygieneTracked.test.ts). The product source must stand on its own, free of non-product scaffolding strings (internal shorthand, external tooling paths). The pair is a belt-and-suspenders: one walks apps/ + packages/ + scripts/ on the filesystem (covering brand-new, not-yet-committed files, and home to a comment-scoped build-phase check), the other shells git grep --untracked over the whole committed-plus-untracked tree (respecting .gitignore, so dist/ and node_modules/ are skipped). Both assert zero matches against a shared pattern set, with carefully case-tuned rules so legitimate domain code, a lowercase session-1 sessionKey fixture, a marketplace persona that mentions project phases, is never falsely flagged. Both files self-exclude, since they carry the patterns literally. Browser-bundle purity (apps/web/src/__tests__/browserBundlePurity.test.ts). The lint-enforced layer boundaries above block relative imports across src/ ↔ server/, but say nothing about which bare @clawboo/* specifiers are safe for the browser — and @clawboo/db’s barrel pulls better-sqlite3, drizzle-orm, and node:* into anything that value-imports it. This guard scans every file under apps/web/src/** and fails on a value import of @clawboo/db (type-only imports are fine; esbuild drops an explicit import type outright, and isolatedModules is what forces type-only imports to be spelled that way), with a self-test asserting the scanner actually catches a static import, a re-export, and a dynamic import(). Its second half checks the substitute the board UI relies on: it resolves @clawboo/board-core’s built output and asserts the artifact declares zero bare import specifiers in either module format — a stronger check than a node:/better-sqlite3 denylist, because esbuild rewrites node:fs to fs in the CJS build, which a node:-prefix denylist would miss. Shared connection (apps/web/server/lib/__tests__/sharedConnection.test.ts). The server opens one SQLite connection per process and bootstraps the schema once, however many requests it serves; before this guard existed, every request handler opened its own connection and re-ran the whole DDL block. The runtime half drives real board handlers against a sandboxed SQLite file and asserts that a burst of 60 reads (and a burst of writes) moves the dbOpenStats() counters by exactly zero, plus that tearing down an SSE stream does not close the shared handle. There is no timing assertion: CI shares runners, so a millisecond threshold would flake and get raised until it asserted nothing; the connection count is the property, not a proxy for it. The source half walks apps/web/server and asserts zero createDb( calls, since the runtime half can only cover the handlers it calls and the background tickers, reaper, supervisor and worktree GC were migrated too. The counters live in the product (packages/db/src/openStats.ts) rather than in test scaffolding because no portable alternative exists: /proc/self/fd is Linux-only and a better-sqlite3 handle never appears in process.getActiveResourcesInfo().

​

How CI wires it together

The CI workflow (.github/workflows/ci.yml) runs lint, typecheck, test, build, and verify-catalog as independent parallel jobs, each on Node 22 with a frozen lockfile, plus the smoke-test-bundle job that runs pnpm assemble && pnpm test:clean-install on the Ubuntu + Windows + macOS matrix. The test job carries one step no other job has: after pnpm test:coverage it builds @clawboo/db and then runs packages/db/src/board/__tests__/board.contention.test.ts with CLAWBOO_CONCURRENCY_TEST=1. That suite spawns 20 real OS worker threads against a single SQLite file, so it needs the package’s own built dist (a worker gets no TS transform, and turbo test builds a package’s dependencies but not the package itself), and a describe.skipIf gates it out of the default pnpm test. It is the only guard on the guarded-create transaction, createCappedSubtask counting siblings and inserting inside one BEGIN IMMEDIATE, so run it locally with pnpm --filter @clawboo/db build && CLAWBOO_CONCURRENCY_TEST=1 pnpm --filter @clawboo/db test before touching that path. It is a test gate, not one of the removed CLAWBOO_* feature flags; an ungated companion test in the same file asserts the CI step still exists. The build job also runs node scripts/check-entry-chunk.mjs, a post-build assertion that the ~4.4 MB marketplace catalog is still a deferred chunk and not preloaded by the SPA entry; its source-side twin is the entryImportGraph unit guard, which walks the static import graph from main.tsx and prints the offending import chain. The live marketplace re-derive (pnpm verify:ingest) is its own workflow (verify-ingest.yml) on a weekly cron plus ingest-path PRs, deliberately off the release path. The Turbo task graph makes test, lint, and typecheck depend on ^build so every package’s workspace dependencies are built first. smoke-test-bundle also downloads Chromium (playwright install --with-deps chromium) before it assembles, for the browser assertion above; Playwright has no postinstall hook, so pnpm install alone never fetches the binary. The seventh job is e2e. It is Ubuntu-only, because the webServer command is POSIX shell syntax that cmd.exe cannot parse, and it does two things the other jobs don’t: it runs a full pnpm build first, and it uploads playwright-report/ plus test-results/ as an artifact so a failure is readable without reproducing it. The build step is not an optimization: every @clawboo/* package resolves through its gitignored dist/, nothing builds those during install, and both halves of the webServer command import them, so the suite would die before its first test without it. A green run is under 4 minutes on a hosted runner; the job’s timeout is 15, leaving headroom for a cold cache and the two CI retries. publish.yml re-runs the same gate inline rather than depending on this workflow: verify:catalog → build → lint → typecheck → test → assemble-cli.sh → test:clean-install, all before changeset publish. That makes the release path a strict superset of the pull-request path, which matters most for typecheck, since pnpm build never invokes tsc and a type error would otherwise sail through the build straight to npm. Two more workflows sit alongside it. codeql.yml runs GitHub code scanning over the TypeScript sources and over the workflow files themselves, on pull requests, on pushes to main, and weekly; its .github/codeql/codeql-config.yml excludes only the codegen’d marketplace catalog, which is ~41% of the repo’s TypeScript by volume and contains no executable logic. The manual eval workflow (evals.yml) is workflow_dispatch-only. The release pipeline that consumes these gates, Changesets, the publish workflow, and the clean-install gate’s role on a Version-PR merge, is documented separately.

​

Design rationale and trade-offs

The shape of this suite follows directly from what Clawboo is. Because the product is a bundled CLI that boots on a stranger’s machine, a unit suite isn’t enough; the clean-install smoke installs and boots the real tarball, on Windows and macOS too, and that’s where the worst regressions hid. Because the orchestrator coordinates real concurrent writers against one SQLite file, the server integration tests use real git and real SQLite rather than mocks; a mock can’t surface a lock convoy or a stale-claim race. Because the orchestrator’s guarantees are behavioral and cross-subsystem, the eval harness grades the outcome on the board, not the transcript’s claim of success, and the deterministic executorRunner integration test exercises the real verifier and structured-state across subsystems (the package-level ablation over its own flags is a harness self-test). And because a stray pnpm e2e run must never be able to touch real state, the e2e isolation is three independent layers, with assertSandboxed as the one that catches the case the other two can’t. The cost is real: the server integration and e2e tests are slow (hence the widened timeouts and the 180-second build window), the eval harness is a whole extra package, and the standing guards add maintenance friction whenever a structural choice genuinely changes. The trade is deliberate; the guards are cheap insurance against exactly the class of regression that ships green and breaks users.

​

Boundaries and non-goals

• No coverage threshold is enforced. The suite is intent-driven (does this guarantee hold?), not line-coverage-driven. There is no --coverage gate.
• Whether Playwright blocks a merge is a ruleset setting, not a workflow property. The e2e job runs on every pull request and every push to main, so a break is visible immediately either way; whether a red run also blocks the merge button is decided by the main branch ruleset’s required-checks list in the repository settings, not by anything in ci.yml. It shipped non-required so a flake could not wedge merges before the job had a run history. The clean-install smoke remains the gate on “the assembled artifact works”.
• Live-model evals never gate a PR. Only the deterministic smoke subset runs in pnpm test. The manual evals.yml workflow (workflow_dispatch) re-runs that same deterministic suite plus the ablation self-test; the live-model graders are defined but not yet wired into any task, so no provider-key path is active today.
• The schema parity guard is name-level, not shape-level. It catches added/removed tables and columns, not type or constraint drift, by design, until a real schema change warrants the deeper check.

​

See also

• Release process: Changesets, the publish workflow, and the clean-install gate on release
• Monorepo and build: Turbo, pnpm, build order, and the commands these tests depend on
• The board: the durable substrate the server integration and eval suites drive
• Executor runner: the claim→run→verify→handoff flow the all-on integration test exercises
• Codegen and ingestion: the offline verify:catalog gate that runs beside these suites, and the live verify:ingest check that runs weekly
• Database schema: the 28 tables the schema-source guard pins
• Glossary: canonical term definitions