Stop 6 of 7
How the harness is checked
The gates decide whether your work may proceed. Something has to decide whether the gates still work. That is a separate question, and it needs a separate instrument — one that runs the guards against inputs designed to defeat them and counts what holds.
Two instruments, two questions
They are not interchangeable, and neither one substitutes for the other. The difference is whether a model has to be in the loop.
| Behavioral eval | Hillclimb suite | |
|---|---|---|
| Asks | Did this intervention change what the agent actually did? | Does the plugin's machinery behave as specified? |
| Lives in | evals/*.md |
evals/hillclimb/ |
| Needs a model | Yes — fresh sessions, transcripts read and graded | No — offline, hermetic, no network |
| Graded by | A person, against binary facts stated in advance | bash autoresearch.sh, in about 200 seconds |
| Result | Keep, simplify, or remove one mechanism | harness_score, 0–100 |
The behavioral method is deliberately small — two conditions, three runs each,
one decision. It is described in full in evals/README.md, and its
honesty rule is worth repeating: an intervention can be available but
never retrieved, or retrieved but not relevant. Only all three
together justify keeping a mechanism.
What the hillclimb suite measures
Six suites, each a set of boolean checks, weighted into one number. Every check
runs the real shipped file — no mocks, no re-implementations — against a fixture
built fresh in a temp directory with its own HOME, no git config, a
fixed timezone and a fixed locale.
| Suite | Weight | What it runs |
|---|---|---|
| behavior | 0.25 | session-start, session-end, statusline and dashboard-log against ~49 adversarial fixtures |
| gates | 0.20 | enforce-scope, commit-gate, verify-task-quality, verify-git-identity, doctor, and the state lock, under hostile input |
| regression | 0.20 | test/run-tests.sh in full, folded in as an aggregate |
| contracts | 0.15 | the features.json validator, its agreement with the schema, stamp.sh's write promises, init.sh's skip protocol |
| static | 0.10 | manifests, frontmatter, file pointers, links, and the constants the prose states versus the ones the code enforces |
| determinism | 0.10 | same input, same output across repeats, timezones, working directories, and both state-module code paths |
Folding the existing test suite in at 0.20 is the load-bearing choice. Without it, the fastest way to raise the score would be to trade an old guarantee for a new one. With it, that trade is visible and costly.
What it found
The suite was written against the plugin as it stood, not to confirm it. Three of the defects it surfaced were silent — the guard returned success while doing nothing.
| Input | Expected | Actual |
|---|---|---|
git -c user.email=x@y.z push |
Blocked — identity does not match | Allowed. The matcher searched for git push as text, and a global option between the binary and the subcommand hid it |
echo x >| .harness/features.json |
Denied — lead-owned state | Allowed. >| is one redirect token; the splitter read its | as a pipe and lost the target |
A commit flipping features to passing with a long id list, suite failing |
Denied | Allowed. The deny payload outgrew the argument limit, the exec failed, and the hook exited 0 with no output |
A corrupt features.json at task completion |
Accept or block | Neither. The gate exited 1 — read as "not a block" — and stopped enforcing |
A corrupt harness.json at init.sh |
Fall back to stack detection | Silent death under set -e, which the task gate then reported as a compilation failure |
Testing the tester
A suite that passes proves nothing until you know it can fail. So the checks were mutation-tested: 66 deliberate breakages introduced into shipped files one at a time, each reverted afterwards, asking whether any check noticed.
Sixty-three were caught. Four were proven harmless by running the mutated hook directly. Eighteen initially slipped through — and every one of those was a hole in the suite, not in the plugin.
They shared a single mistake, and it is the most useful thing on this page: asserting shape instead of content. "The report is well-formed." "The status line is one line." "A gap file exists." "The exit code is 0 or 2." Every one of those still passes after the thing being checked has been removed entirely. A check has to assert the fact the code exists to produce.
Running it
From the repository root:
bash autoresearch.sh
It prints one METRIC harness_score=… line, a per-suite breakdown, and
one line per failing check naming what failed and why. No network, no API key, no
configuration. Two concurrent runs produce byte-identical output, so a change in
the score is a change in the plugin.
The checks are additive by policy: one may be added, but not removed, weakened, or made conditional to raise the score, and the total may never decrease. When a mutation leaves every check passing, that is a gap in the suite — not evidence the code is safe.