01Verification rules in CLAUDE.md
adoptedrecently updatedupdated #
why
An agent does what its instructions make cheap. Without written rules it declares victory on a green build, retries a flaky test until it passes, or edits the test instead of the code. A few explicit lines in the global CLAUDE.md change the default for every session.
how
- Put the rules in the global
~/.claude/CLAUDE.md, so they apply to every repo, and keep project specifics in the repo's own CLAUDE.md. - Verify by running: the repo's
verify skill when there is one, otherwise drive the real app. - Three rounds, then stop: after three failed fix attempts the agent hands over the failing output and what it tried.
- Keep the checks out of reach: a change must never weaken the tests, lint config,
verify skill or REVIEW.md that judge it in the same commit.
Excerpt for your CLAUDE.md
markdown
## Verification
- After any non-trivial change: run build/typecheck and relevant tests before declaring it done
- If you can't verify, say so explicitly instead of assuming it works
- Don't edit tests just to make them pass: if a test fails, understand why before touching it
- Add/update tests when changing public behavior
- **Verify by running**: if the repo has a `.claude/skills/verify` skill, run it before saying "ready for testing". Otherwise drive the real app (a browser automation tool for anything web) when the change is user-visible. Always state how the change was verified
- **Three rounds, then stop**: if a check still fails after three fix attempts, stop and bring me in with the failing output and what you tried
- **Flaky tests**: never add retries, sleeps or skips to make them pass. Find the root cause; if you can't, propose deleting the test with the reason written down, and wait for my confirmation
- **Checks are out of reach**: a change must not weaken or rewrite, in the same commit, the existing tests, lint/typecheck config, `verify` skill or `REVIEW.md` that judge it (adding new tests for new behavior is fine). If an existing check genuinely needs changing, do it in a separate commit and flag it to me
## Commit Messages
- Every commit ends with a `Verified:` trailer stating how the change was checked: `verify` (verify skill or end-to-end run), `tests` (automated tests only), `manual` (I tested it), or `none (<reason>)`. It is a verification stamp, not an attribution line
The loop these rules describe. Markasso source
02A verify skill in every repo
adoptedadded #
why
"Run the app" means something different in every repo. A verify skill writes it down once: the cheap checks, how to boot the real thing, which flows to drive and what to expect, so the agent collects evidence instead of guessing.
how
- Copy the template to
<repo>/.claude/skills/verify/SKILL.md and replace every placeholder; delete the sections that do not apply. - Track it in git: ignore
.claude/* but not .claude/skills/. - Run it twice in a row on a clean main before trusting it: it is your smoke test harness, it must be fast and deterministic.
- Say what is out of scope (paid API calls, deploys, what the tools cannot see) and when to ask a human instead.
- Every new project starts with
.claude/skills/verify from the template, on day one. - In work or client repos you don't own, keep the skill local with
.git/info/exclude and never commit it.
Template
markdown
---
name: verify
description: Verify a change to <PROJECT> end to end before saying "ready for testing" or committing. Runs the cheap checks first, then drives the real <app|CLI|service> and reports the evidence plus the Verified trailer value.
---
<!--
Copy it to <repo>/.claude/skills/verify/SKILL.md, replace every <PLACEHOLDER>,
delete the sections that do not apply and this comment. Make sure .claude/skills/
is tracked by git (ignore .claude/* but not .claude/skills/).
Keep it fast and deterministic: run it twice in a row on a clean main before trusting it.
-->
# Verify <PROJECT>
Run the steps in order and stop at the first failure. Fix the code, not the check.
## Rules
- Never add retries, sleeps or skips, and never edit the existing tests, lint/typecheck config or this skill to make a step pass.
- If a step still fails after three fix attempts, stop and report the failing output and what was tried.
- Clean up what you started (servers, temp dirs) even when a step fails.
## 1. Cheap checks
<typecheck command>
<lint command, check-only, no auto-fix>
<test command>
## 2. Run it
<How to start the real thing: dev server, built CLI, local service. Where it listens, how to tell it is ready, how to stop it.>
Servers: pick a free port at runtime with `--strictPort` (or equivalent), fail the wait loop if your process dies, and check an identifying marker (page title, health payload) before testing. Other sessions and agents run their own servers: never reuse or stop one you did not start.
Smoke flows (always):
1. <Flow>: <steps>. Expected: <observable result>.
2. <Flow that must fail>: <steps>. Expected: <error, exit code>.
Flows for the changed area: <where the feature map lives, e.g. docs/MANUAL.md>. Pick the flows that cover the diff and check their expected results.
Out of scope: <commands this skill must not run, e.g. paid API calls, deploys>.
## 3. Report
Reply with:
- each command run with its exit code;
- the evidence for every flow (output excerpt, screenshot path);
- the `Verified:` trailer value: `verify` when steps 1 and 2 passed, `tests` when only step 1 ran, `none (<reason>)` otherwise.
Kickoff, and local-only skills for repos you don't own
bash
# New project: start with a verify skill from day one.
mkdir -p .claude/skills/verify
cp <path-to-your-template>/SKILL.md .claude/skills/verify/SKILL.md
# Track the skill, ignore the rest of .claude/ (in .gitignore):
# .claude/*
# !.claude/skills/
# Work or client repo you don't own: keep the skill local, never commit it.
# .git/info/exclude works like .gitignore but is never shared.
echo ".claude/skills/verify/" >> .git/info/exclude
git status --short .claude # must print nothing
Section for an end-to-end Playwright suite
markdown
## 2. End-to-end (Playwright)
Before running the suite, make sure the port in `playwright.config` is free. With `reuseExistingServer: true`, Playwright happily tests whatever already answers on that port: another project, or this project on another branch.
PORT=<port from playwright.config webServer.url>
if lsof -tiTCP:$PORT -sTCP:LISTEN >/dev/null; then
echo "port $PORT is busy: stop, do not test someone else's server"; exit 1
fi
BROWSER=none npx playwright test
- `BROWSER=none` keeps `vite --open` from opening browser windows during the run.
- `Executable doesn't exist at ...` means the browser build for this Playwright version is missing. It is setup, not a test failure: run `npx playwright install chromium`, then run the suite again. Do not count it as one of the three rounds.
- Use selectors that do not depend on the UI language (roles with stable names, `data-*` hooks), or set `use: { locale: 'en-US' }` in the config.
Real example: Markasso, a web app driven through Telemaco
markdown
---
name: verify
description: Verify a change to Markasso before saying "ready for testing" or committing. Runs typecheck, Biome (check only) and vitest, the MCP build when mcp/ changed, then boots the web app on a private dev server and checks it loads cleanly with Telemaco. Reports the evidence plus the Verified trailer value.
---
# Verify Markasso
Run the steps in order from the repo root and stop at the first failure. Fix the code, not the check.
## Rules
- Never add retries, sleeps or skips, and never edit the existing tests, `biome.json`, `tsconfig*.json`, `vitest.config.ts` or this skill to make a step pass.
- If a step still fails after three fix attempts, stop and report the failing output and what was tried.
- Use `pnpm lint:ci`, never `pnpm lint`: the latter rewrites files.
- Never reuse or stop a dev server you did not start: other sessions and agents run their own (5173 is usually the user's). Start your own on a free port picked at runtime and always stop it, even when a step fails.
## 1. Cheap checks
pnpm typecheck
pnpm lint:ci
pnpm test
`pnpm test` also runs `mcp/tests/`. If the diff touches `mcp/`, also build the server:
pnpm mcp:build
## 2. Boot smoke (Telemaco)
Start a private dev server on a free port and wait until it answers. The loop fails if the server dies, so a port taken in the meantime can never make you test someone else's server:
PORT=$(node -e "const s=require('net').createServer();s.listen(0,()=>{console.log(s.address().port);s.close()})")
pnpm dev --port $PORT --strictPort > <scratchpad>/markasso-dev.log 2>&1 &
PID=$!
until curl -fs -o /dev/null http://localhost:$PORT/; do
kill -0 $PID 2>/dev/null || { tail -20 <scratchpad>/markasso-dev.log; exit 1; }
sleep 0.5
done
echo $PORT
With the Telemaco MCP tools:
1. `browser_navigate` to `http://localhost:$PORT/` with `waitUntil: networkidle0`. Expected: title starts with `Markasso` (any other title means you reached a server that is not yours: stop).
2. `browser_console_messages`. Expected: no errors.
3. `browser_evaluate`:
({ canvas: !!document.getElementById('main'), toolbarButtons: document.querySelectorAll('#toolbar button').length, rectangle: !!document.querySelector('#toolbar button[title^="Rectangle"]') })
Expected: `canvas: true`, `toolbarButtons` greater than 0, `rectangle: true`.
4. `browser_close`, then stop the server:
kill $(lsof -tiTCP:$PORT -sTCP:LISTEN)
Out of scope: drawing or any interaction on the canvas. Telemaco has no layout engine (`getBoundingClientRect` returns placeholder values), so canvas hit-testing cannot be driven from it; canvas behavior is covered by the vitest suite. When the change is a visible canvas or UI behavior, say so in the report and ask the user for a manual check (`Verified: manual` once they confirm).
## 3. Report
Reply with:
- each command with its exit code, and the vitest totals;
- the Telemaco results for steps 2.1 to 2.3;
- the `Verified:` trailer value: `verify` when steps 1 and 2 passed, `tests` when only step 1 ran, `manual` when the user checked a visible change, `none (<reason>)` otherwise.
from Markasso ↗
Real example: Themis, a CLI driven in a temp project
markdown
---
name: verify
description: Verify a change to Themis end to end before saying "ready for testing" or committing. Runs npm run check, rebuilds dist/, then drives the real CLI (check, init, status) in a temp project and reports the evidence plus the Verified trailer value.
---
# Verify Themis
Run the steps in order from the repo root and stop at the first failure. Fix the code, not the check.
## Rules
- Never add retries, sleeps or skips, and never edit the existing tests, `biome.json`, `tsconfig*.json`, `vitest.config.ts` or this skill to make a step pass.
- If a step still fails after three fix attempts, stop and report the failing output and what was tried.
- Never delete `dist/`: `~/.local/bin/themis` links to `dist/cli/main.js`. Rebuilding in place is fine.
- Do not run commands that call agents (`plan`, `tests`, `run`, `retro`): they cost money and minutes. Their logic is covered by the vitest suite with `FakeAgentRunner`/`FakeExecutor`.
- Always remove the temp directory, even when a step fails.
## 1. Cheap checks
npm run check
(typecheck, Biome, vitest including `tests/e2e/`).
## 2. Build
npm run build
## 3. Drive the real CLI
Run in bash (the commands rely on bash arrays and `PIPESTATUS`):
bash -c '
set -u
C=(node "$PWD/dist/cli/main.js")
T=$(mktemp -d)
trap "rm -rf \"$T\"" EXIT
"${C[@]}" check examples/webhook-service/spec.md; echo "check-valid exit=$?"
printf -- "---\nthemis: 0.2\nstack: node-ts\n---\n# X\n" > "$T/bad.md"
"${C[@]}" check "$T/bad.md"; echo "check-invalid exit=$?"
cp examples/getting-started/spec.md "$T/"
cd "$T" && git init -q
"${C[@]}" init; echo "init exit=$?"
"${C[@]}" status --json; echo "status exit=$?"
ls .themis
'
Expected:
| Flow | Expected |
|---|---|
| check-valid | exit 0, `valid (6 acceptance criteria, ...)` |
| check-invalid | exit 1, errors `invalid-field`, `unsupported-version`, `no-acceptance-criteria` |
| init | exit 0, writes `.themis/agents/*.md`, `.gitignore`, installs dependencies (needs network for `npm install`), ends with `next: themis plan` |
| status | exit 0, JSON with `spec.ok: true`, `plan: "absent"`, `contract.state: "absent"` |
| .themis | contains `agents`, `guard.mjs`, `local.json`, `verify.sh` |
If the change touches a command not listed above (for example `new`, `approve`, a new flag or a new diagnostic), add a run of that command against the temp project, with its expected exit code and output, to the report.
## 4. Report
Reply with:
- each command with its exit code;
- the relevant output lines for every flow, and any deviation from the table;
- the `Verified:` trailer value: `verify` when steps 1 to 3 passed, `tests` when only step 1 ran, `none (<reason>)` otherwise.
from Themis ↗
04A Stop hook that watches before it blocks
in trialnewadded #
why
Rules say "verify before you are done", but nothing tells you how often a session ends without it. A Stop hook can measure that. Blocking straight away would punish every false positive, so the first version only watches and writes down what it saw.
how
- Run it at the end of every turn: it reads the session transcript and logs one verdict per edited repo,
verify, commands, docs-only or none. - Count only the edits of the current session, so other sessions on the same repo cannot raise false alarms.
- Never block, never print, always exit 0: a broken hook must not break the session.
- Read the report after a week or two, fix the false positives, and only then decide whether to block turns that end on
none.
Registration and report
bash
# A Stop hook that only watches: it runs at the end of every turn, never
# blocks, never prints, always exits 0. It reads the session transcript and
# appends one JSON line per repo the turn edited, with a verdict:
#
# verify the repo's `verify` skill ran after the last edit
# commands a test/build/lint/typecheck command ran after the last edit
# docs-only only documentation files were edited
# none code was edited and nothing checked it afterwards
#
# Only edits made by this session count (Edit, Write, NotebookEdit calls in
# its own transcript), so other sessions on the same repo raise no false alarms.
# Merge this into the "hooks" object of ~/.claude/settings.json, next to your
# other hooks: do not overwrite the file. The guard matters: a Stop hook that
# exits 2 sends the agent back to work, and python3 exits 2 when the script is
# missing, so a hook meant to only watch would block every turn.
{
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "f=\"$HOME/.claude/hooks/verify-gate.py\"; [ -f \"$f\" ] && python3 \"$f\"; exit 0",
"timeout": 10
}
]
}
]
}
# After a week or two, read the verdicts per repo before deciding to enforce:
python3 ~/.claude/hooks/verify-gate.py --report
in trialrecently updatedupdated #
why
A verify skill is only as good as the browser that drives it. We ran one real test on Markasso's canvas: draw a rectangle, then check that it landed in localStorage. The results decided which tool does what.
how
- Telemaco: FAIL. No layout engine, and
browser_evaluate does not await Promises. - chrome-devtools-mcp: PASS only with synthetic events dispatched from a script, since it has no drag at coordinates; about 36 s.
- Playwright MCP: PASS with real mouse input (
browser_mouse_drag_xy), about 24 s. On the first try it even hit a panel lying over the canvas, exactly as a real user would. - Decision, now written in the global CLAUDE.md: Telemaco stays for reading the web, search and stealth fetches; driving and verifying local apps moves to Playwright MCP, pinned, headless and isolated. Moving the existing verify skills is in progress.
Playwright MCP for local apps
bash
# Playwright MCP to drive local apps with real mouse and keyboard input.
# Pin the version: the server is fetched by npx on every start.
# Pick the current one with `npm view @playwright/mcp version`.
# --scope user: one setup for every project, not just the current one.
# --headless: no window stealing focus while you work.
# --isolated: in-memory profile, so parallel sessions don't fight over one profile.
# It also means first-run dialogs show up on every launch.
# --caps vision: coordinate tools such as browser_mouse_drag_xy, needed for canvas work.
# --output-dir: screenshots and traces go to a cache folder, not into the repo.
PLAYWRIGHT_MCP_VERSION=0.0.85
claude mcp add --scope user playwright -- npx -y "@playwright/mcp@$PLAYWRIGHT_MCP_VERSION" \
--browser chrome --headless --isolated --caps vision \
--viewport-size 1280x800 --output-dir "$HOME/.cache/playwright-mcp"
# The browser follows the system locale, so the app's UI may be translated:
# select by ids, classes, roles or shortcuts, never by visible text.
# If the app has no stable hooks yet, force a locale with a config file:
# { "browser": { "contextOptions": { "locale": "en-US" } } }
# and add --config "$PWD/playwright-mcp.json" to the command above.
from Playwright MCP ↗
06A second opinion with Codex or opencode
in trialrecently updatedupdated #
why
An agent reviewing its own diff shares its own blind spots. A second model with a fresh prompt finds a different set of bugs, as long as you give it the code, not your conclusion, and treat its output as input, not truth.
how
- Codex is the long-term reviewer for deep and critical reviews, one call per review because its quota is limited; opencode's free models take smaller diffs, control runs and scoped tasks.
- Size the model to the task: light for a quick question, standard for a small or medium diff, deep for a branch, a plan or a debug session, where two free models run in parallel and you cross-check them.
- Ask before running it, and say what you send and why.
- When the repo has a
REVIEW.md, send it read from the base branch and ask the reviewer to apply the rules whose trigger matches the diff. - Read-only by default: opencode's
plan agent or Codex's read-only sandbox. Write access only inside a separate git worktree. - A subagent that works on another repo creates its own worktree there with
git worktree add: the agent tool's worktree isolation only covers the session's repo. - Use only models whose provider keeps no data and does not train on it, and never send secrets or client code you may not share.
- Verify every finding against the code, fix what is real, and write down what you rejected and why.
Commands
bash
# Two reviewers with different roles:
# Codex the long-term reviewer for deep and critical reviews (a branch, a plan, security);
# one call per review, its subscription quota is limited.
# opencode free models for smaller diffs, control runs and scoped tasks.
# Size the model to the task, and use only providers that keep no data and do not train on it:
# light explain a snippet, a small question, a trivial edit one small free model
# standard review of a small or medium diff, a scoped task one stronger free model
# deep branch review, debug after failed hypotheses, a plan Codex, or two free models
# in parallel, cross-checked
# Review a diff, read-only: opencode's built-in `plan` agent cannot edit files.
# If the repo has a REVIEW.md, send it from the base branch, never from the working tree.
# `git diff main` covers commits and uncommitted work; an empty diff is no review.
git diff main > /tmp/change.diff
test -s /tmp/change.diff || echo "empty diff: nothing to review"
git show main:REVIEW.md > /tmp/review-rules.md 2>/dev/null
opencode run --agent plan -m <model> \
"Review the diff in /tmp/change.diff for correctness bugs, unescaped input and missing tests. \
If /tmp/review-rules.md is not empty, apply the rules whose trigger matches a path in the diff. \
Give each finding a file:line, a severity (must-fix / should-fix / nit) and a concrete failure scenario. \
Do not edit files." > review.md
# Delegate a scoped task with write access, only inside a separate git worktree.
opencode run -w <worktree-name> -m <model> "<task, files, what is already ruled out>"
# Codex for deep or critical reviews: read-only sandbox, findings written to a file.
codex exec -s read-only -C . -o codex.md "<same review prompt>"
# `codex review --base main` is quicker but takes no rules file: prefer `codex exec`
# with the same prompt when the repo has a REVIEW.md.
07A REVIEW.md the change cannot rewrite
in trialnewadded #
why
A generic review prompt finds generic issues. A REVIEW.md tells the reviewer what breaks in this code, and because it is always read from the base branch, a change cannot soften the rules that judge it.
how
- Give the reviewer the diff plus
git show <base>:REVIEW.md, never the copy in the branch under review. - Every rule comes from a real bug, citing the fix commits, or from a stated project constraint. No generic advice.
- Make each rule checkable from a diff: a trigger on paths, a concrete check, and why it exists.
- Keep it under about 15 rules: add or sharpen one when a bug slips through review, delete the ones that never trigger or only produce noise.
- Change
REVIEW.md in its own commit, never together with the code it judges.
Template
markdown
<!--
Copy it to <repo>/REVIEW.md, replace the placeholders, delete this comment.
How it is used: the reviewer (Codex, opencode, /code-review) gets the diff plus
this file read from the base branch (`git show <base>:REVIEW.md`), never from
the branch under review, so a change cannot rewrite the rules that judge it.
Change REVIEW.md in its own commit, never together with the code it judges.
Writing rules:
- Every rule comes from a real bug (cite the fix commits) or a stated project
constraint (cite CLAUDE.md, an ADR, a spec). No generic advice.
- A rule must be checkable from a diff: concrete trigger paths, concrete check.
- Keep it under about 15 rules. When a bug slips through review, add or sharpen
a rule; when a rule never triggers or only produces noise, delete it.
-->
# Review guide: <PROJECT>
Check only the rules whose trigger matches a path in the diff. For each finding, name the rule (R1, R2, ...) or say "outside REVIEW.md" when it is a bug no rule covers. Report real defects with a concrete failure scenario; skip anything listed under "Do not flag".
## Rules
### R1: <short title>
Trigger: <paths or globs>
Check: <what must hold in the diff>
Why: <the failure it prevents> (fixes: <sha>, <sha> | source: <CLAUDE.md, ADR, spec section>)
## Do not flag
- Formatting, import order and lint-level style: <linter> enforces them in CI.
- <Deliberate convention that looks wrong but is intended, with its source.>
08Themis: gates and a locked contract
adoptedadded #
why
For larger work, rules are not enough: the agent must not be able to move the goalposts. Themis turns a spec into acceptance tests, locks them by digest, and only then lets agents write code that a separate verifier judges.
how
- Write the spec: goal, verify steps, limits, acceptance criteria.
- Approve the plan (gate 1) and the tests (gate 2): approving the tests records the digests of tests, spec, verifier and config in
.themis/lock.json, and a guard rejects any change to them. - Run: each task works in its own worktree; the verifier's exit code says retry or blocked, and
max_iterations caps the attempts. - Close with a retro (gate 3): misses become changes to the agents' instructions.
The gated flow
bash
themis new # spec skeleton: goal, verify steps, limits, acceptance criteria
themis check # every problem with line, column and field
themis init # generates the judge: verifier + guard, no AI involved
themis plan # planner agent proposes the task graph
themis approve plan # gate 1: does the plan make sense?
themis tests # test-author agent writes tests/acceptance/
git add -A && git commit -m "themis contract"
themis approve tests # gate 2: digests of tests, spec, verifier and config go into .themis/lock.json
git add -A && git commit -m "lock contract"
themis run # each task in its own worktree, verifier exit 1 = retry, 2 = blocked
themis status # gates, tasks, iterations against max_iterations, cost
themis retro # retro agent turns misses into instruction changes
themis approve retro # gate 3: apply them, then commit .themis/agents/
from Themis ↗
The Themis gated pipeline: the checks are locked before the worker starts. Markasso source