One rule your coding agent can't ignore: you're not done until the tests pass. This is the exact setup from my Reel: Codex in Cursor, a Stop hook, and acceptance tests that belong to the harness, not to the agent.
Inspired by Jediah Katz's talk "Is the Agent Harness Dead?" (AI Engineer): models keep learning the clever tricks, but trust, infrastructure and efficiency stay the harness's job.
#What it looks like
Codex wrote a first version and wanted to stop. The hook ran the tests and sent it back:
Codex fixed the code by itself. Only then the hook let it finish:
#1. My Stop hook + hooks.json
Put this in your repo as .codex/hooks.json:
{
"hooks": {
"Stop": [
{
"hooks": [
{ "type": "command", "command": "node \"$(git rev-parse --show-toplevel)/.codex/hooks/stop_require_tests.mjs\"", "timeout": 120 }
]
}
]
}
}And this as .codex/hooks/stop_require_tests.mjs. When tests fail, it returns decision: "block", and Codex continues with the reason as its next prompt. After 3 failed tries it stops and hands over to you, so it can't loop forever.
#!/usr/bin/env node
// Stop hook: the agent is not done until the harness acceptance tests pass.
import { execFileSync } from "node:child_process";
import { readFileSync, existsSync, writeFileSync } from "node:fs";
const input = JSON.parse(readFileSync(0, "utf8") || "{}");
const root = input.cwd || process.cwd();
const counter = `${root}/.codex/.stop-attempts`;
const attempts = (existsSync(counter) ? Number(readFileSync(counter, "utf8")) : 0) + 1;
let report;
try {
execFileSync("npx", ["vitest", "run", ".codex/acceptance", "--reporter=json", "--outputFile=.codex/.report.json"], { cwd: root, stdio: "ignore" });
} catch { /* failing tests exit non-zero; the report still gets written */ }
try { report = JSON.parse(readFileSync(`${root}/.codex/.report.json`, "utf8")); } catch { report = null; }
const total = report?.numTotalTests ?? 0;
const failed = report ? report.testResults.flatMap((f) => f.assertionResults.filter((a) => a.status !== "passed").map((a) => a.title)) : [];
const broken = !report || total === 0 || report.testResults.some((f) => f.status === "failed" && f.assertionResults.length === 0);
if (!broken && failed.length === 0) {
writeFileSync(counter, "0");
console.log(JSON.stringify({ systemMessage: `Harness check passed: ${total}/${total} acceptance tests green.` }));
} else if (attempts > 3) {
writeFileSync(counter, "0");
console.log(JSON.stringify({ systemMessage: `Harness check still failing after 3 tries (${failed.length} failing). Stopping for a human.` }));
} else {
writeFileSync(counter, String(attempts));
const list = broken ? "- the acceptance suite could not run" : failed.map((t) => `- ${t}`).join("\n");
const head = broken ? "Harness check failed." : `Harness check failed: ${failed.length} of ${total} acceptance tests fail.`;
console.log(JSON.stringify({ decision: "block", reason: `${head}\n${list}\nYou're not done yet. Fix the code in src/ (don't edit .codex/), then finish again.` }));
}Add .codex/.report.json and .codex/.stop-attempts to your .gitignore.
#2. Acceptance-test template
The trick: the tests live in .codex/acceptance/, owned by the harness. They encode your decisions (here: who counts as Pro), so the agent can't quietly pick the easy answer. Uses Vitest (npm i -D vitest).
// .codex/acceptance/billing.test.ts
import { describe, expect, it } from "vitest";
import { isPro } from "../../src/lib/billing";
const now = new Date("2026-10-21T12:00:00Z");
const inDays = (d: number) => new Date(now.getTime() + d * 86_400_000);
const user = (o: Record<string, unknown>) => ({ plan: "free", subscriptionStatus: null, currentPeriodEnd: null, ...o }) as any;
describe("who counts as Pro", () => {
it("free user without a subscription is not Pro", () => {
expect(isPro(user({}), now)).toBe(false);
});
it("14-day trial is Pro", () => {
expect(isPro(user({ subscriptionStatus: "trialing", currentPeriodEnd: inDays(9) }), now)).toBe(true);
});
it("past_due keeps Pro while Stripe retries the card", () => {
expect(isPro(user({ subscriptionStatus: "past_due", currentPeriodEnd: inDays(-2) }), now)).toBe(true);
});
it("cancelled keeps Pro until the paid period ends", () => {
expect(isPro(user({ subscriptionStatus: "canceled", currentPeriodEnd: inDays(5) }), now)).toBe(true);
});
});Replace these with the rules of your own feature. One test per decision you'd hate the agent to get wrong.
#3. How to trust hooks in Codex
- Start Codex in the repo:
codex - Codex shows "Hooks need review: 1 hook is new or changed". Hooks can run outside the sandbox, so Codex asks you first.
- Pick Review hooks, read the command, then trust it.
- Codex remembers the hook by its hash. If you change the file, it asks again.
Tip: Codex's own docs list this exact use case: "run a custom validation check when a chat turn stops". See developers.openai.com/codex/hooks.
#4. The same hook for Claude Code
Claude Code speaks the same language for Stop hooks (decision: "block" + reason). Add this to .claude/settings.json and reuse the script:
{
"hooks": {
"Stop": [
{
"hooks": [
{ "type": "command", "command": "node \"$CLAUDE_PROJECT_DIR/.codex/hooks/stop_require_tests.mjs\"" }
]
}
]
}
}#Why this matters
- The model writes the code. The harness makes sure it's true.
- Clever prompt tricks get learned by the next model. A rule that runs your tests doesn't.
- Keep the acceptance tests outside the agent's reach and make the hook stop after a few tries, so a human decides when it's stuck.
Made by @sangeeths.ai: real AI engineering, recorded live. Follow on Instagram for more. Share this kit: sangeeths.ai/harness · All free kits: sangeeths.ai
