Codegen and ingestion

• pnpm verify:catalog checks the committed files against a committed integrity manifest (scripts/ingest-manifest.json). It is offline, so it runs on every PR and on the release path.
• pnpm verify:ingest re-runs the whole render pipeline in memory against the live pinned upstream commits and diffs the result. It runs weekly, on demand, and on any PR touching the ingest scripts, deliberately off the release critical path.

​

What it is, and what it isn’t

• Runtime fetch would break offline-first installs and make the catalog non-reproducible; a deploy would depend on a live GitHub and on whatever the upstream HEAD happened to be.
• Hand-writing ~300-line entries across hundreds of agents is error-prone, unreviewable at scale, and would drift the moment an upstream source changed.

​

The model

1. The input is pinned. The two upstream commit SHAs are constants in scripts/lib/ingest-helpers.ts (AGENCY_AGENTS_SHA = '64eee9f8…', AWESOME_OPENCLAW_SHA = '659895e5…'). Both the ingest and verify scripts import the same constants, and every GitHub call, the recursive git-tree fetch and each raw-file fetch, embeds the SHA in its URL. Nothing reads a branch HEAD. Re-running today and re-running in a year against the same pins produce byte-identical output.
2. The transform is order-stable. Every domain’s agents are sorted by id before rendering, the awesome-openclaw entries are sorted by id, and the per-usecase named-agent extraction de-dupes deterministically. There is no Date.now(), no Math.random(), no filesystem-order dependence in the rendered content.

​

How the ingest pipeline works

​

Fetch (pinned, concurrency-limited)

​

Parse (markdown → typed entry)

• derives a stable id (agency-<slug(filename)>, with the sub-folder prepended for game-development files to avoid collisions),
• pulls a 1–2 sentence description from the YAML frontmatter (with a body-line fallback),
• distills a soulTemplate by collecting up to three sections whose headings match a small SOUL_KEYWORDS list (tolerant of leading emoji and possessive prefixes), falling back to the first 400 characters,
• matches skillIds against an inline SKILL_MATCH_CATALOG by word-boundary tag matching, and
• sets identityTemplate to the file’s content verbatim (see the zero-loss invariant).

​

Render and format (the Prettier step)

​

Team generation

• teams/agency-workflows.ts: five hand-curated workflows (WORKFLOW_TEAM_CONFIGS maps each example filename to a list of catalog agent IDs), with hub-and-spoke routing generated by buildHubSpokeRouting (first agent is the leader; everyone else routes to @<Leader>) and the full example .md body stored as workflowNarrative.
• teams/awesome-openclaw.ts: one team per usecase, members grouped by usecase slug.
• teams/synthetic.ts, the 30 “Excellence Teams” that partition every agency agent not already covered by a workflow team into per-domain clusters, so every agent appears in at least one team. The exclusion set comes from workflowAgentIds().

​

The zero-loss identityTemplate invariant

it('identityTemplate.length > 500 (zero-loss — full content preserved)', () => {
  for (const e of AGENT_CATALOG) {
    expect(e.identityTemplate.length).toBeGreaterThan(500)
  }
})

​

Design rationale and trade-offs

​

The integrity manifest

{
  "$comment": "AUTO-GENERATED — do not edit manually. Regenerate: pnpm ingest:marketplace…",
  "version": 1,
  "sources": {
    "agency-agents": { "repo": "msitarzewski/agency-agents", "sha": "64eee9f8…" },
    "awesome-openclaw": { "repo": "hesamsheikh/awesome-openclaw-usecases", "sha": "659895e5…" }
  },
  "files": {
    "apps/web/src/features/marketplace/agents/agency/academic.ts": "111bba55…"
  }
}

1. Hashes are over the canonical form, not the raw bytes. See the <Info> above: raw-byte hashes would go stale the moment the pre-commit hook restyled a generated file. Hashing the canonical form makes them invariant to style, to .prettierrc, and to CRLF. The one thing they are sensitive to is a Prettier major bump moving that canonical form, which is what pnpm ingest:manifest exists to re-bless.
2. The file set is derived, not hand-listed. Both the manifest and verify:catalog enumerate via catalogFilePaths() in scripts/lib/ingest-helpers.ts, composed from the same path helpers the generator writes through, so the list cannot drift from what is actually generated. verify:ingest additionally cross-checks its own file list against that enumeration, so the two verifiers police each other.
3. Manifest keys are repo-relative and POSIX-separated, so a regeneration on Windows does not rewrite all 20 keys with backslashes.

​

The verify gates

​

Refreshing the catalog

1. Pick the new upstream commit. Read the diff between the current pin and the candidate first; a re-ingest rewrites every generated file, so a large upstream change is a large PR.
2. Edit the SHA constant in scripts/lib/ingest-helpers.ts (AGENCY_AGENTS_SHA or AWESOME_OPENCLAW_SHA). The SHA is embedded in the 14 per-source data files’ headers and in every entry’s sourceUrl, so one constant moves all of them. (The six barrel and team files carry no SHA header — they re-export rather than restate the data.)
3. Regenerate with pnpm ingest:marketplace. This needs network. Locally the GitHub API allows 60 requests/hour unauthenticated and the run makes ~180, so export a GITHUB_TOKEN (or GH_TOKEN) first to get the 5000/hour ceiling.
4. Expect 20 regenerated .ts files plus an updated scripts/ingest-manifest.json.
5. Verify both gates: pnpm verify:catalog (offline) then pnpm verify:ingest (live).
6. Run the tests: pnpm test. The catalog suites assert count lower bounds, ID uniqueness, and the zero-loss identityTemplate floor — the checks that catch an upstream restructure quietly dropping content.
7. Commit. The pre-commit hook restyles the generated files into the repo’s Prettier style. That is expected and does not change the manifest hashes.
8. Open the PR. The ingest-paths trigger runs the live check automatically, so the PR gets both gates.

​

Boundaries and non-goals

• Not a live marketplace. There is no runtime fetch, no remote catalog API, no per-install update channel. The catalog is whatever was committed at build time. A “fetch from ClawHub” model is a hypothetical future, not a shipped feature.
• Not the source of truth for the hand-written slice. The 15 clawboo built-in agents and 5 built-in teams are authored by hand and live outside the pipeline. verify:ingest neither generates nor checks them; their correctness rests on ordinary unit tests, not on the drift gate.
• Counts are not test assertions. The catalog ships 304 agents and 82 teams, but the tests assert lower bounds (≥ 270 agents, ≥ 160 agency, ≥ 40 awesome, ≥ 15 clawboo) so a future re-ingest can grow the catalog without breaking them. Treat the zero-loss identityTemplate floor and the verify gate as the invariants, not the exact counts.

​

See also

• Marketplace catalog reference, the AgentCatalogEntry / TeamTemplate schemas, ID conventions, and per-source counts this pipeline produces
• The agent model, what a deployed catalog agent becomes
• Release process, Changesets, publish.yml, and the clean-install gate this sits alongside
• Monorepo and build, the Turbo / pnpm build the catalog compiles into
• Testing; the unit / e2e / clean-install strategy that backs the catalog’s invariants
• Glossary, canonical term definitions