Eval platform governance for AI coding teams

If your eval dashboard is green but you cannot say what the green run was allowed to do, you do not have governance yet, you have a comforting number. Eval platform governance is the set of receipts, scopes, and owners that keep an agent's quality score tied to what actually ran. The goal is simple: any green score should be reconstructable by a teammate who was not in the room.

This matters because of one old idea. When a measure becomes a target, it stops being a good measure. The moment an eval score decides what merges, people start writing to the score instead of to the work, and the score quietly drifts away from quality. Receipts are how you keep it honest.

See what the scored run was actually allowed to do

Most green-but-wrong runs start with fuzzy scope. The agent had room to touch things nobody intended, the eval graded the result anyway, and the receipt that would have caught it never existed. Claude's rule files (.mdc) read precise until two reviewers disagree about what a rule meant. The Claude agent docs explain the mechanism, but the boundary is yours to write.

The fix is a scope ledger: five plain lines per run that the platform can show next to the score. Goal, allowed paths, forbidden paths, verification command, merge owner. That is it.

goal: refactor checkout retry logic, no API changes
allowed: src/checkout/**, tests/checkout/**
forbidden: src/billing/**, infra/**, *.env
verify: pnpm test checkout && pnpm typecheck
owner: @dani

Once this sits beside the score, a green number stops being a mystery. You can read what the run was permitted to do before you trust what it did.

Write down the rules the score is grading

Claude Code reads a CLAUDE.md for its operating policy, and on a shared machine bash approvals turn into muscle memory fast. Sessions start inventing policy mid-run, and your eval ends up grading behavior that traces back to no written rule. The Claude Code getting started guide shows where that policy file lives.

Put a precedence clause at the top of the file so the graded behavior maps to something stated: which hooks win, which folders need human eyes, where temporary overrides go and when they expire. When a reviewer asks why the agent did X, the answer should be a line in the file, not a shrug.

Codex CLI, OpenAI's coding agent, has the mirror problem: a merged green where nobody saw the transcript is exactly as trustworthy as it sounds. The Codex quickstart covers setup, and an AGENTS.md is the right home for a small habit I call the replay sandwich. Intent line on top, command transcript in the middle, diff summary on the bottom, all in the PR. The number gets a story a human can follow.

Treat the harness itself as software with connectors

The eval harness is not neutral plumbing. It has MCP connectors of its own, and one of them will eventually reach data nobody drew on the diagram. MCP, the Model Context Protocol, is the standard agents use to talk to external tools and servers, which means every server is a door.

Give each server a connector card: allowed actions, forbidden actions, owner, rollback. One card per server, the harness included. The payoff is that incidents shrink, because when something misbehaves the operator already knows what "off" looks like and who owns the switch.

Here is the whole boundary on one page, a delegation snapshot you can paste into a repo and adapt:

---
description: Delegation boundary snapshot (adapt globs to your repo)
globs:
  - "**/*"
alwaysApply: false
---

- Claude: keep scopes explicit in `.mdc`; forbid undeclared MCP domains.
- Claude Code: cite `CLAUDE.md` precedence before expanding bash scope.
- Codex: ensure `AGENTS.md` carries replay-friendly verification notes for CLI runs.

Make a score earn its gating power

A score should not gate a merge until four questions have boring answers. Boring is the goal: if any answer is interesting, the run is not ready to be trusted on autopilot.

Before you let a run merge, walk this short strip:

• Primary-doc links were smoke-checked after publishing edits.
• MCP connectors mentioned (if any) list owners.
• Verification command output is pasted or linked.
• Forked agent work lists parent and child responsibilities.

A score is a thermometer, not a thermostat. It tells you the temperature of a run; it does not get to set the policy. Receipts are what let review do that part.

If a run touches anything risky, keep the OWASP Top 10 for LLM applications and the NIST AI Risk Management Framework next to the decision. They belong in the same conversation as the score. The broader argument lives under agentic coding governance, and the habits travel well to a team workshop where people practice writing these receipts by hand.

Common questions

• What does eval platform governance actually cover?

  It covers the receipts behind the score: a scope ledger for what the run was allowed to do, the replay transcript behind the green number, and connector cards for the harness itself. Without those, the platform grades runs that nobody can inspect. The score is real; the trust is not.

• Can eval scores replace code review?

  No. A score reports an outcome, while review explains a change, and those are different jobs. The minute a score gates merges it becomes a target, and Goodhart's law starts bending it away from quality. Keep scores as instruments, and keep receipts as the thing review actually reads.

• How do you stop Goodhart drift in agent evals?

  Attach receipts to every scored run: ledger, transcript, diff summary. When a score moves, someone should be able to open the runs behind it and explain why in a sentence. Drift survives in dashboards because nobody looks twice. It dies in transcripts that a reviewer actually opens.

• Where should I start if I have none of this?

  Write one connector card for the eval harness before your next scored run, then add the five-line scope ledger to the run after that. Two artifacts, fifteen minutes, and your next green number is already more honest than the one before it. Expand only once those two stick.

Start here

Pick one green-scored run from this week and try to rebuild it from receipts alone; whatever you reach for and cannot find is your first gap to close.