self-bench Turns Private PRs Into Evals

self-bench is an MIT-licensed open-source project from mupt-ai that turns completed GitHub pull requests and local coding sessions into private coding-agent benchmarks. It deals with a plain measurement problem: public coding evals can look impressive while saying little about whether an agent can work in your repository. The useful takeaway is simple: measure agentic coding on work your codebase has already accepted, then keep the tasks held out.

A private coding-agent eval is a benchmark built from your own repository history, with hidden tests and a known reference solution. self-bench matters because it tries to make that benchmark from normal engineering exhaust instead of from hand-written toy tasks.

Start with the thing self-bench actually builds

self-bench builds Harbor evals from a GitHub repo. It scans completed work, reconstructs each task from the commit before the change, creates hidden tests, stores the original implementation as the reference solution, and checks that the task fails before the solution and passes after it.

That last step is the important bit. A benchmark task that already passes at the starting commit is not a task. A task that cannot be validated after the original PR is applied is not useful evidence either.

As of August 14, 2026, the repository was mainly TypeScript, MIT licensed, and small enough to still feel like an experiment rather than an ecosystem. The README describes a local control plane using Bun, Docker Compose, Temporal workflow state, GitHub CLI access, and Modal sandboxes for disposable task generation and validation.

The trap is treating the generated tarball as objective truth. It is better than vibes, but it is still shaped by your merge history, your tests, your review habits, and whatever work the tool can successfully reconstruct.

Read the benchmark like a lab sample

The interesting claim is not that self-bench beats public evals. The interesting claim is that private repo history can become a repeatable lab sample.

SWE-bench-style tasks are attractive because they begin with a real broken-or-incomplete repository state and ask the model to produce a patch. self-bench borrows that shape, but points it at your own accepted changes rather than a public benchmark corpus.

That helps with saturation. A model can overfit public tasks through training data, benchmark-aware prompting, or plain ecosystem repetition. Your boring internal bugfix from three months ago is less likely to be memorized, and more likely to exercise the exact framework seams, naming conventions, and test patterns your agents will face tomorrow.

The trap is assuming privacy alone makes an eval good. A tiny repo with thin tests will produce thin signals. A codebase full of flaky integration tests will punish agents for test noise as much as reasoning failure.

Notice what it does not prove

self-bench can tell you whether an agent solves a reconstructed task under the runner's conditions. It cannot prove that the same agent will behave safely in a live branch with half-written requirements, messy chat context, and a reviewer who changes their mind.

That difference matters for Claude Code, Anthropic's coding agent, and for similar terminal-based coding agents. A benchmark run is a clean race. A real session is a conversation with file edits, shell commands, tool permissions, failed assumptions, and sometimes a human steering halfway through.

Use the result as a model comparison signal, not a moral ranking. If Model A solves 18 of 30 private tasks and Model B solves 11, that is useful. It still does not say which one should touch migrations, secrets, billing code, or release scripts without extra review.

The trap is collapsing eval score into permission level. Evals should inform where you ask the agent to work. They should not replace code review guardrails.

Run a tiny internal version before believing a big one

You do not need a grand measurement program to learn from the self-bench idea. Pick three merged PRs from one repository: one bugfix, one refactor, and one test-backed feature. Recreate the pre-change commit, ask your coding agent to solve the issue from the PR description, and compare the result to the merged diff.

For Claude Code users, keep the harness dull. Put task instructions in a short slash command, run from a clean worktree, and require the agent to stop after tests and a diff summary. A concise CLAUDE.md can help only if it contains durable repo facts, such as test commands and architecture boundaries, not task spoilers.

Here is a small command file shape that works well as a smoke test:

# .claude/commands/private-eval-task.md

You are solving a held-out repository task.

Start by reading the issue summary below and inspecting the current tree.
Do not inspect later commits, merged PRs, or branch history for this task.
Make the smallest patch that satisfies the task.
Run the relevant tests.
Stop with:
- files changed
- tests run
- remaining uncertainty

Task summary:
$ARGUMENTS

This is not self-bench. It is the hand-rolled version you can run in under an hour to see whether the idea fits your repo. If it feels noisy at three tasks, it will probably feel noisy at thirty.

For a neighboring terminal-agent measurement story, Mole Puts a Budget on Terminal Research is a useful contrast: it focuses on constraining research cost, while self-bench focuses on turning past code changes into eval tasks.

Try a one-hour private eval smoke test

Use this when you want a fast read on whether your repo history can produce meaningful coding-agent evals. It is deliberately small.

If this smoke test produces clean tasks, self-bench is worth a closer look. If it produces ambiguous tasks, fix the task selection process before blaming the model.

Common questions

• Can self-bench use private repos?

  Yes, self-bench is designed around private repository history and completed work. The README describes using an authenticated GitHub CLI with read access to the repo, then exporting a private .tar.gz benchmark for Harbor rather than publishing the tasks.

• Is this just SWE-bench for my codebase?

  Roughly, yes, but that shorthand hides the hard parts. self-bench uses a SWE-bench-style shape by reconstructing tasks from earlier commits, adding hidden tests, and validating against the original implementation, but the quality depends on your repo history and test coverage.

• Will this tell me which coding agent to buy?

  No, it will give you one better comparison signal, not a full purchasing answer. A private eval can show which models solve representative repo tasks, but you still need to compare latency, cost, tool permissions, review flow, and how the agent behaves during real development.

• How does this fit with Claude Code?

  It fits best as a measurement harness beside normal Claude Code work, not inside every chat. Use Claude Code for controlled task attempts, keep instructions stable with a slash command, and preserve the benchmark boundary by not exposing the merged PR or later commits.

• What makes a bad private eval task?

  A bad task is one where the expected behavior is unclear, tests are flaky, or the solution depends on hidden human context from the original review. Private evals are strongest when the pre-change state fails for a concrete reason and the accepted patch makes that reason disappear.

Best ways to use this research

• Best for: deciding whether repo-specific evals are worth building before you compare coding agents on real work.
• Best first artifact: a three-PR smoke test with one consistent Claude Code slash command, clean branches, and a solved or failed score for each task.
• Best comparison angle: public benchmark score versus private repo task performance, especially when public evals look saturated.
• Best caution: do not let eval scores silently become production permission levels. Measurement should feed review, not replace it.

If you are collecting examples for an AI coding workshop, this belongs in the measurement corner of agentic coding governance: small, repeatable, and tied to code that actually shipped.

Further reading

• self-bench — source
• Claude Code — overview
• SWE-bench
• Model Context Protocol — specification
• GitHub — openai/codex

Take the small measurement win

Start with three old PRs and one clean agent command. If the tasks are clear, self-bench gives you a path from anecdote to evidence without pretending the evidence is perfect.

One methodology lens

One useful way to read this through our methodology is the Plan step: delegate first-pass decomposition and dependency mapping, review the sequencing and assumptions, and keep ownership of scope and priorities. If that split is still fuzzy, the workflow usually is too.