Oct 2026 · Grasphouse

Running an AI-Built Project

Contracts, rulings, planted bugs and gates: notes on knowing that code written by AI agents is right.

When AI agents write most of the code, the hard part of the project is no longer writing it. It is knowing that what got written is right, and keeping a long project from drifting while nobody holds all of it in their head.

These are notes from running Nablaframe, a small GPT trained from scratch so that its training and forward pass can be replayed and inspected. It is now in its seventh phase. Almost every line of its code was written by AI implementers, against specs and tests written by an AI spec partner, with me deciding what counts.

The one design idea underneath everything: no one grades their own work, and every claim traces back to something written down. Each mechanism below is a hand-off between roles, and each hand-off has a check that catches one kind of failure.

The cast #

Four roles, kept apart on purpose. The separation matters more than any tool.

The spec partner and the implementers are the same kind of model, but they never share a context. An implementer sees only the repo: the contracts, the spec and the decision log. That is what makes their agreement mean something.

Back to contents ↑

Contracts first: the delta #

The unit of work is a delta: one slice of the spec turned into contracts and failing tests, applied to main before anyone implements it.

A delta ships:

• Contracts as docstrings. Each module’s docstring says exactly what it does, refuses and returns. Docstrings live next to the code, so an implementer can’t miss them and a reviewer can diff them.
• Stubs that raise NotImplementedError, so the code imports and the tests run.
• Tests that fail on the stubs. The delta doc lists exactly which ones: “18 failed, 363 passed” is a precise to-do list, and done means those 18 pass with no skips.
• Exact edits to existing files, made by a script that replaces one quoted passage at a time and stops if a passage is missing or repeated. A delta never rewrites a file wholesale, so main never holds a half-changed contract.
• A doc of what was settled. Every reading the spec left open is written down with its reason, so the reviewer rules against text, not memory.

The point is that the implementer’s job is small and checkable. It doesn’t decide what to build; it makes named tests pass without touching the contract.

Back to contents ↑

Testing the tests #

A test suite written by the same model that will be judged by it is easy to fool. So every delta is proven twice before it ships.

• A private reference implementation. The spec partner implements every contract itself, privately, and never ships it. If the reference can’t pass the tests, the contract is wrong or impossible. If it passes, the contract can be met.
• Planted bugs. The reference is then broken on purpose, one plausible bug at a time: an off-by-one, a swapped argument, a strict inequality where the spec says “at most”. Every planted bug must make a test fail. The last delta planted 90; each escape became a new test case before shipping.
• Bars that can fail. Some checks are designed to be unable to fail, by accident. One measure capped its count at 8 while the bar asked for “at most 8”; the gate now counts without the cap. A wrong function that only relabelled the right one could never score badly, so it was dropped by a ruling.

The reference has a second use: when an implementer’s PR arrives, both versions run on real trained models, and their outputs have to match value for value.

Back to contents ↑

Rulings #

Every choice that could reasonably go either way becomes a numbered entry in a decision log: its date, the decision, and the evidence it rests on. The project has more than 60. Committing an entry is the ruling, so only the governor does it.

Two rules make the log work:

• Check before you assume. Every role reads the log before treating a choice as open. A settled question isn’t argued again in the next PR.
• Stop and report. When a spec contradicts the code or a ruling, the agent that notices it stops and reports the conflict instead of choosing. The repo’s instructions say so in one line, and the agents follow it.

Most new rulings arrive mid-build, not in planning. One came from a wrong function that could never fail. Another came from tracing what a code change would do to records written a phase earlier. The most recent came from a gate that couldn’t run as specified (below). Each took the governor a minute to decide, because the options and the evidence arrived written down.

Back to contents ↑

Implementers, PRs and review #

Each implementer is a thread with a fresh context and one package: named files, named tests, and an objective “done when”. Threads that don’t depend on each other run in parallel, each on its own cloud machine; those that do run in waves. A thread opens a draft PR and lists every reading it had to make where the docstring left a choice open.

Review is where the independence pays off. For each PR the spec partner:

1. Applies the diff to a fresh clone of main, and checks that no docstring, test or out-of-scope file moved.
2. Runs the thread’s named tests, the full lane, and the whole wave applied together.
3. Runs the code on real data, where it exists, against the private reference.
4. Posts the review as a PR comment, verdict first, with a ruling on each listed reading.
5. Merges only when every condition holds: the reviewed commit, only the allowed files, no decision left open, a passing lane, and the governor hasn’t asked to hold it.

Watching is event-driven. A small script polls GitHub once a minute and wakes the reviewer when a PR opens or gets new commits, with a scheduled check every two hours in case the session that runs it was reclaimed. A fixed half-hourly timer was the first version; reviews started up to 30 minutes late.

One caution from the last wave: “these threads share no files” is not the same as “these threads are independent”. One thread’s test needed another thread’s code, so the merge order mattered. The thread noticed and said so in its PR.

Back to contents ↑

Gates and the machine of record #

Tests say the code does what the contract says. Gates say whether the project’s claims hold. A gate is a script that runs on the reference machine, measures something against a bar set from earlier experiments, and writes one report.

• Reports are generated, never edited. A validator rejects any report whose pass or fail disagrees with its own numbers, so a report can’t say pass while its measurements say fail.
• Gates chain. Each refuses to start until the reports it builds on exist and pass.
• Only one machine counts. The cloud machines the agents use are for development. Every number in a report comes from the Mac.
• Results are rebuildable. Records are keyed by their inputs and the code that made them, and the gates recompute and compare hashes.

A case study: eight probe pairs #

The newest gate traces a small modular-addition model on a fixed probe of 8 input pairs. Before tracing, the engine runs an oracle that requires the model’s plain pass and its recording pass to produce bit-identical logits. That policy was ruled long ago and had held on every model since.

While exercising the gate on a cloud machine, the implementer found the two passes 1 float32 rounding step apart on about a third of the logits. The first suspect was the machine: a different CPU, more threads. A second machine with another CPU gave exactly the same counts, so it wasn’t the hardware. Then the Mac failed too.

The cause was size. Each pair is only 3 tokens, so 8 pairs make 24 rows, and at that size the compiler builds the two passes differently. From 48 pairs the passes agree bit for bit; the earlier models’ probes had never been small enough to notice. A 64-pair probe passed the oracle at all 45 trained checkpoints and an 8-pair probe at none. The ruling: trace 64 pairs and keep the oracle strict. From first report to ruling took an afternoon, and no gate result was ever at risk, because the gate would have failed loudly instead of passing quietly.

Back to contents ↑

Where it breaks #

The process has three weak points, and each has shown up in practice.

• One author for the contracts and the review. A blind spot in the spec is likely to be in the tests and the reference too, and independent implementers only partly offset that. In one delta the spec partner declared three threads independent when one depended on another, and shipped a bug in one of its own tests. Both were caught, one by a thread and one by a rerun. The real backstop is the gates, which nobody’s tests can talk past.
• Effects across deltas show up only when someone traces them. Records are keyed partly by the code that built them. A small change to a shared module quietly gave every record from the previous phase a new key, and a planned rerun of that phase’s final gate would have refused to start. Nobody had checked for that until it was traced, and it took a ruling to route around it.
• The governor is the bottleneck, by design. Rulings and phase sign-offs wait for me, so the loop moves at the pace of my attention, not the machines’. That is deliberate: deciding what counts as done is a judgement, not an optimization. It does mean the work should arrive as a decision with its evidence attached, never as an open question.

Back to contents ↑

Takeaways #

If you run a project with AI agents doing the building, these are the habits that paid for themselves:

1. Keep the roles apart: whoever specifies doesn’t implement, whoever implements doesn’t review, and one human decides.
2. Write contracts where the code lives, and ship them with failing tests before anyone implements.
3. Prove each contract with a private implementation, then break it on purpose and make sure the tests notice.
4. Ask of every bar whether it can fail. If it can’t, it isn’t a bar.
5. Keep a decision log, and make “stop and report” the default answer to a contradiction.
6. Give implementers small, named jobs and let them list their own readings. Those lists are where the spec’s gaps surface.
7. Review on real data, not only on tests, and merge only under written conditions.
8. Pick one machine of record, and let only its generated, validated reports state results.
9. Make results rebuildable: key everything by its inputs and code, and recompute.
10. Expect the process’s own blind spots. Trace each change’s effect on what already exists before it ships.

Back to contents ↑

Part of Grasphouse.