Agent Reliability & Evaluation
- ·Evaluation suites, golden tasks and a recorded baseline, so a quality drop is caught by a run rather than by a customer.
- ·The same discipline covers ordinary code: automated checks before a merge, and a deploy that either completes or reverts.
- ·The harness tools on this site are the receipts, and each one is published with the finding it produced rather than a score.
| Built with | TypeScript · Agent evals · Vitest · GitHub Actions · CI/CD · Vercel |
| Delivered by | eval-engineer · · · debugger |
| Skills used | eval-suite · debug-loop · quality-gate |
Receipts
What this involves
A feature that calls a model has no compiler. Nothing fails when a prompt edit makes the answer worse, when a tool description quietly changes which tool gets picked, or when a model version ships and the same input starts returning something else. The only thing that catches those is a task set you run every time, with the previous run's scores written down next to it.
The receipts here are tools built to find one specific failure and published with what they found. sluice measured 666 duplicate side effects with naive retry, 0 with sluice under fault injection, and refused one call in five as indeterminate rather than guessing. snapgauge grades a tool-description edit as risky rather than cosmetic, pinned by 37 golden and compat cases at 100% exact match. chaff reports 44 golden cases at 100% exact match, 0 enforced findings on its own CLAUDE.md, 0 measured rules shipped on its own author's context file, and says in every line of output that it ships zero measured rules, because the evals that would admit one need paid model calls.
Two of them are worth more for being unflattering. driftwatch's first real run across seven repositories reported ten failures and every one of them was a false positive, which is now the headline on its own card. tally's first real target was itself, and 2 false positives removed after the first real run, 7 planted defect fixtures. A tool whose first run was clean has usually not been pointed at anything real yet.
Underneath the model layer the ordinary discipline is unchanged and is shown rather than described. The suite guarding this site is tests on node:test with no framework to install, and the typo correction in this console was checked against an independent reference implementation over 20,000 string pairs with zero mismatches, re-running from a fixed seed in tests/suggest-fuzz.test.ts. One Nadela Ops carries green at every deploy, private repository green tests alongside versioned migrations, applied in order applied migrations.
On the deployment side the pipeline is expected to fail loudly and early: checks run before a merge can reach production, and the deploy either completes or reverts rather than half-landing. Which platform hosts it follows the project, and the ones with a receipt in this repo are Vercel and Cloudflare Workers.
Under fault injection, naive retry produced 666 duplicate side effects with naive retry, 0 with sluice, and the landing page serves those figures from the committed results file rather than from a slide.
How the work runs
1. Discovery
We agree what the project has to achieve, what it must not break, and how we will both know it worked, before any code is written.
2. Build
The work is built with an AI agent as the working partner and a human reviewing every change before it lands. You get commits you can read rather than a status update.
3. Review
You review running software at each milestone, not a screenshot of it, and the next milestone absorbs what you send back.
4. Launch
We deploy to production with checks in the pipeline and alerting that reaches a human, and you hold the repository and the infrastructure accounts.
Questions people actually ask
What is actually different about testing a feature that calls a model?
The output is not deterministic, so a single assertion is the wrong instrument. What works is a fixed set of tasks with agreed scoring, run on every change to the prompt, the tools or the model, with the previous scores kept. That turns an argument about whether the answers feel worse into a comparison of two numbers taken the same way.
How is an evaluation kept honest rather than tuned until it passes?
By planting failures it is supposed to catch and by reporting the runs where it did badly. The ClubScope prototype's eval set includes a deliberately fabricated figure so the verifier has something to catch, and the run reports catching it. driftwatch's card leads on its first run being ten false positives out of ten. An evaluation with no recorded bad run has usually not been pointed at anything real.
Does this replace ordinary testing and CI?
No, it sits on top of it, and the order matters. Make the build reproducible first, then get checks running on every change, then automate the deploy, then add the model-layer evaluation. Adding an eval suite to a project whose build is not reproducible measures the build, not the model.
What does the test suite on this site actually consist of?
It is tests written against Node's built-in node:test runner, which is why it needs no framework installed and runs anywhere Node 22.6 or later is available. The suite covers the console engine, the i18n catalog, the MCP server and the game logic, and `npm test` prints the count.
What is in a deployment setup worth paying for?
Automated checks on every change, a preview environment for review, a production deploy that either completes or reverts, secrets held outside the repository, and alerting that reaches a human. The last one is the most commonly skipped and the most expensive to skip.