BreakWater

TL;DR

1. Breakwater independently rediscovered known weaknesses from HumanEval, from behaviour alone.

2. Found issues in live Terminal-Bench tasks and submitted upstream fixes

3. Survived worker crashes across 96 Docker trials with 0 lost results and 0 duplicate result rows

AI benchmarks give us clean numbers: pass, fail, 72%, SOTA.

But every one of those numbers eventually comes from a grader.

So I started with a different question:

What if the grader is wrong?

Breakwater is a distributed evaluation and falsification system I built to attack that assumption. It stress-tests benchmark verifiers with adversarial solutions, program mutations, replayed outputs, environment changes, and multi-stage probes, then runs those experiments on fault-tolerant infrastructure designed to survive crashes without losing or double-counting results.

The goal isn't to make another benchmark.

It's to test whether the benchmarks we already trust deserve that trust.

A PASS is a claim

Suppose an agent solves a coding task and the benchmark returns PASS.

That could mean the solution is correct.

It could also mean:

• the tests missed an edge case,

• the verifier accepted the wrong type,

• state leaked between cases,

• expected answers were accidentally exposed,

• environment randomness changed the verdict,

• or the evaluation system itself retried, duplicated, or lost work.

A benchmark score compresses all of that into one bit.

Breakwater expands it back out.

Instead of asking “did the benchmark pass this?”, Breakwater asks:

“Can I construct evidence that this PASS should not have happened?”

Treating the verifier as the system under test

The core tester attacks graders from several directions.

Simple probes come first: no-op implementations, constants, exceptions, echoes, and deliberately malformed solutions. A verifier accepting one of these is immediately suspicious.

Then Breakwater starts making small mutations to a correct implementation.

An AST mutation might change:

into:

If the mutated program is behaviorally different but still passes every benchmark test, Breakwater searches for a concrete input where the two implementations diverge.

That gives me something much stronger than “coverage seems weak.”

It gives me the exact missing test case.

Every finding needs a receipt

A benchmark-auditing tool can be wrong too.

So Breakwater doesn't treat its own detector as ground truth.

Every serious finding is packaged as a self-contained falsification receipt containing the task, mutation or probe, inputs, observed behavior, verifier result, and enough information to replay the claim in a fresh container.

python -m projectx.tester replay receipts/<finding>

The bundled validation suite contains intentionally broken and sound graders.

Breakwater identified 7/7 planted verifier flaws with the intended failure class, produced 0 CRITICAL/HIGH false positives on the 10 sound tasks, and independently replay-confirmed 9/9 generated receipts.

That constraint changed how I built the system: a finding isn't useful because an analyzer says it's interesting. It's useful when the evidence survives reproduction.

Sometimes the bug is the sequence

Some evaluation failures aren't visible from a single run.

Breakwater can re-verify the same recorded output multiple times, separating verifier nondeterminism from execution nondeterminism.

It can pin variables such as PYTHONHASHSEED, timezone, and locale to determine whether instability follows the environment.

It can verify the same case alone, inside a batch, and in reversed order to expose hidden state or order dependence.

And it separates solution execution from verification so the solution environment never needs access to the verifier or expected outputs.

That last one came from Breakwater finding a flaw in its own first harness: expected answers were reachable from the solution container.

I kept the broken version as a planted regression test.

Then I tried to break the machine running the experiments

A perfect falsification method is useless if its distributed runner silently loses results.

So Breakwater's second half is a fault-tolerant execution platform built around a simple guarantee:

at-least-once execution, exactly-once result effect.

Workers claim jobs through Postgres using:

Each job carries a lease and heartbeat. If a worker crashes or stalls, a reaper lets the lease expire and puts the trial back into the queue.

Results go through a separate ledger:

That means a trial can physically execute twice after a crash, while only one result is ever accepted.

I didn't just unit-test that behavior.

I killed the workers.

96 trials. Kill one worker. Freeze another.

The chaos run launched 96 Docker-isolated trials across four workers with three concurrent executions each.

Mid-run, Breakwater sent SIGKILL to one worker and SIGSTOP to another long enough for its leases to expire.

The system started 102 executions for 96 logical trials.

6 expired trials were requeued and recovered by another worker.

When the paused worker came back from the dead and tried to submit stale results, the ledger suppressed 3 duplicate submissions.

Final database invariants:

96 / 96 results accepted

0 lost results

0 duplicate result rows

The important part isn't merely that the demo completed. The final report queries the actual database state to prove those invariants instead of assuming the orchestration code worked.

Evidence survives the worker too

Each execution attempt stores its own stdout, result, and metadata artifacts through an S3-compatible object store.

That gives Breakwater provenance for retries as well as accepted runs.

If attempt 1 dies, attempt 2 succeeds, and a zombie attempt later submits a duplicate, those are three distinct pieces of execution history even though the result ledger still contains exactly one accepted outcome.

I also exposed experiment provenance as an SSE stream so execution state can be observed while trials are running rather than reconstructed afterward.

Then I pointed it at HumanEval

The adapter layer lets the falsification engine operate on external benchmarks without rewriting its core.

For HumanEval, Breakwater successfully converted 146 of 164 problems into its internal representation. Unsupported conversions were counted rather than silently ignored.

The audit reproduced five concrete shallow-test weaknesses.

Two were missing behavioral boundaries:

• HumanEval/0: a < → <= mutant could survive the original tests.

• HumanEval/35: an implementation indexing l[1] passed despite failing on a singleton list.

Three more — HumanEval/52, /56, and /61 — exposed a subtler contract issue. Their checks used bare truthiness assertions, allowing logically equivalent implementations returning values such as "yes" and "" instead of actual Booleans.

Breakwater discovered all 3/3 of those truthiness cases through behavioral testing and produced replayable witnesses.

These aren't claims that modern HumanEval variants remain broken: stronger suites such as HumanEval+ already close these gaps.

For me, that was actually a useful validation result.

Breakwater independently rediscovered known weaknesses from behavior alone.

Hard benchmarks should sometimes come back clean

I also built an adapter for Terminal-Bench / Harbor tasks, where the output isn't necessarily a Python value — it can be the entire resulting container state.

Breakwater solves the task in one container, commits the resulting filesystem state, and lets the verifier judge that exact state separately. That makes operations like verify_only, case isolation, and environment perturbation possible even for terminal-based agent tasks.

And importantly, the first real audits did not produce a wall of supposed vulnerabilities.

Ten completed Terminal-Bench tasks came back clean under the applicable soundness and robustness checks.

That mattered.

An auditor that always finds something isn't an auditor I would trust.

The audit escaped the repo

The real payoff came when the investigation started producing fixes upstream.

While auditing Terminal-Bench's risk-scorer-replay task, I reproduced an intermittent failure using the unmodified reference solution.

One of three runs could fail because the verifier enumerated a temporary SQLite journal file while another process was rebuilding the database. By the time the verifier tried to read the journal, SQLite had already deleted it.

That isn't model failure.

That's benchmark infrastructure manufacturing a failure.

I opened issue #1963 with the reproduction and submitted PR #1964 — Fix risk-scorer-replay transient file race, updating the verifier to safely handle the transient FileNotFoundError.

I also submitted PR #1965 — Fix biped jump transition margin for biped-contact-dynamics.

The task specification defined a 5-sample transition margin, and the run-phase verifier followed it. The jump-phase check used 3, meaning the benchmark was grading samples the task itself said should be excluded.

The fix makes the verifier match its own contract.

Those PRs were the point where Breakwater stopped being only a portfolio experiment and started feeding reliability work back into the benchmark it was auditing.

One engine, very different benchmarks

The adapter architecture ended up being important.

HumanEval represents value semantics: run a function and compare behavior.

Terminal-Bench represents state semantics: let an agent modify a machine and inspect what exists afterward.

Breakwater converts both into a common trial model while keeping benchmark-specific execution behind adapters.

Workers don't need to know which benchmark produced the task.

Receipts don't depend on the adapter being present later.

The falsification engine stays the same.

That made the project feel less like a collection of scripts and more like an actual evaluation platform.

What Breakwater is really testing

I started the project thinking mostly about distributed reliability.

It became a more interesting problem.

There are two separate trust boundaries in an AI evaluation:

Can I trust the execution system to faithfully run the experiment?

and

Can I trust the evaluator to correctly interpret the result?

Breakwater attacks both.

It kills workers, replays attempts, verifies SQL invariants, preserves provenance, mutates correct programs, searches for counterexamples, perturbs environments, cross-examines graders, and packages failures into reproducible evidence.

Because if we're going to use benchmark numbers to decide which agents are improving, which models are safer, and which systems are ready to deploy—

the thing producing the number should be tested too.