A check that already passes cannot measure the work

Two weeks ago, the spec agent in efferent proposed an acceptance check named no-ts-project-yet.

Read the name twice. The check asserted the current state of the workspace — that the TypeScript project it was about to set up didn’t exist yet. Green before a single line of work. A check that starts green stays green through a run that changes nothing, and — since the checks are how the run is judged — counts toward accepting it. The model that authored it didn’t flag it. I read the draft and didn’t either. The first thing that caught it was a warning at the mouth of the forge run, after the spec was already locked.

The fix that came out of the incident: a proposed check now runs at the moment it’s proposed, against the untouched workspace — and if it passes, the proposal is rejected. This post is about that mechanism and its edge cases: vacuous greens, reds that are red for the wrong reason, multi-line commands, and the no-op hole all of it exists to close.

What this post won’t do is sell you red-first. “Watch the test fail before you make it pass” is old discipline and you already know the argument. The point is that it’s a discipline humans skip when tired and agents skip statistically — so the question isn’t whether, it’s where a machine enforces it, and what the enforcement can actually see.

Sixty seconds of pipeline#

Smith does not build from a chat message. A rough idea first becomes a SpecDoc, refined with the human by a refiner agent that never implements. The refiner’s toolkit is read-only exploration plus exactly one write tool — propose_spec, every call replacing the whole draft. When the draft holds, the human :locks it; only a locked spec forges; :ship turns an accepted run into a commit and a PR.

The spec is a Schema.Class, and its machine half is the point:

/** A machine-checkable acceptance criterion: a named shell command that must
 *  exit 0. Drivers append one rank-2 command gate per check (`accept-<name>`). */
export class SpecCheck extends Schema.Class<SpecCheck>('SpecCheck')({
  name: Schema.String.pipe(Schema.pattern(/^[a-z0-9][a-z0-9-]*$/)),
  command: Schema.NonEmptyString,
}) {}

At forge time each check joins the staged gate pipeline right next to the typecheck and test command — a command gate like any other, fail-closed. The pipeline is the entire verdict: the run is done when the gates say so, not when the model feels finished. (A workspace with no discoverable gates at all refuses to forge — a run with nothing to verify would accept anything.) A spec check is the definition of this change’s done-ness, committed alongside it at .efferent/specs/<slug>.md.

Which is exactly what makes a vacuous check poisonous. The gate declares victory. If the accept gates are green before the work, a run that edits nothing is an accepted run.

Enforced where the check is born#

The whole mechanism is a probe inside the propose_spec handler — the refiner’s one write, so there is no path around it:

// RED-FIRST, ENFORCED at the source (the doctrine: validation in
// deterministic code, never advisory): run each proposed check
// against the workspace NOW — the same probe forge uses — and
// bounce any that already pass. A green-before-the-work check is
// a precondition, not acceptance; it would accept a no-op run.
const vacuous = yield* vacuousAccepts(
  (params.checks ?? []).map((check) =>
    makeCommandGate({
      name: check.name,
      argv: ['bash', '-c', check.command],
      timeoutMs: RED_FIRST_TIMEOUT_MS, // 30s — a forge gate gets five minutes
    }),
  ),
  snapshotWorkspace(cwd), 
)
if (vacuous.length > 0) {
  return yield* Effect.fail({
    error: 'VacuousChecks', 
    message: `check(s) ${vacuous.join(', ')} already PASS on the untouched workspace — …`,
  })
}

vacuousAccepts builds, for each proposed check, the same command gate the forge will later run — bash -c, exit 0 means pass — and probes it against the workspace as it stands, untouched because nothing has happened yet. Any check that comes back green fails the whole proposal. And because propose_spec declares its failures as returned values, the bounce isn’t an exception ending the session — it’s a tool result the refiner reads and acts on in the same turn, with the draft still hot. The message it reads:

check(s) … already PASS on the untouched workspace — red-first: an acceptance check must FAIL before the work and pass only once the work is done. A check that describes the current state (“no X yet”) is a precondition, not acceptance: drop it or restate it as a constraint, then re-propose.

The sort in the middle is the real finding from the incident. no-ts-project-yet wasn’t nonsense — it was a true, even useful statement about the workspace. It was in the wrong slot. A statement about the current state is a precondition, and the spec has a place for those: constraints. Acceptance is the set of statements that are false now and true when the work is done. The bounce teaches that distinction at the exact moment it was confused — and the tool’s own description now carries the same warning in capitals: “ENFORCED: the checks run when you propose.”

Two smaller rules ride the same handler. A multi-line command bounces before anything runs — “make it a single line (join statements with ’;’, or move the logic into a test file and run that)” — because a check is exact shell, never normalized, re-run verbatim by every future gate. And the propose-time probe caps each check at thirty seconds where a forge gate gets five minutes: a check slower than that belongs in a test file the check then runs. Both rules push the same way — the spec holds one auditable line, and anything with real logic becomes code.

Why propose is the cheap moment#

Three moments could hold this enforcement; two are worse.

At forge — where the warning lived before the fix — the information arrives after the refine conversation is over and the human has locked. Acting on it means walking the pipeline backwards: abort the run, reopen the spec, re-refine. The feedback crosses two stages and lands on the human — the one participant who didn’t author the check.

At lock, it would still interrupt the human: a verdict about a command the refiner wrote, delivered to the person about to approve it.

At propose, the loop is one hop. The bounce lands on the agent that authored the check, in the turn it authored it, as a failure shape it already knows how to handle: re-propose. And the verdict is unambiguous in a way it never is again — before any work, “the workspace as it stands” and “the baseline the work will be measured against” are the same thing, so one execution fully decides vacuousness. No heuristic, no second model judging the check’s intent. A vacuous check is caught by running it, not by asking anyone.

Asking had already failed, which is the instructive part. Before the fix, the refiner’s system prompt said RED-FIRST in capitals — a check “must FAIL on the workspace as it is NOW.” The model wrote no-ts-project-yet anyway. The comment on the enforcement names the house doctrine — validation in deterministic code, never advisory — and the prompt keeps its red-first paragraph only so the bounce is comprehensible when it lands. The instruction explains; the probe enforces.

Red for the wrong reason#

Green-before-the-work is one degenerate check. There’s a red that’s just as useless, and it wears the costume of compliance: exit 127.

An earlier forge run against a zig codebase hit it in full. The host had no zig toolchain, so every accept check was red — properly red, by red-first’s lights. The coder provisioned its own toolchain into the workspace, and the gates still exited 127, because the gate environment had diverged from the coder’s. Three attempts burned against findings no code edit could move.

Two changes came out of that run. Gates and coder now resolve the same PATH, including <workspace>/.local/bin — in the comment’s words, the oracle and the worker must share reality. And a 127/126 exit with a not-found signature became its own species of finding:

// Exit 127/126 with a not-found/not-executable signature is an ENVIRONMENT
// failure, not a code failure — the old fixHint ("make it pass") sent the
// coder chasing an unfixable finding.
if ((run.exitCode === 127 || run.exitCode === 126) && MISSING_TOOL.test(run.output)) {
  return [
    new Finding({
      rule: RuleId.make(`env/${options.name}`), 
      severity: 'error',
      message: `ENVIRONMENT: \`${command}\` exited ${run.exitCode} — its tool is missing from PATH: …`,
      fixHint: Option.some(
        'editing code cannot fix this — provision the missing tool into ' +
          '<workspace>/.local/bin (your shell AND the gates see it there) ' +
          'or have it installed on the host',
      ),
    }),
  ]
}

The env/ prefix is what the classification keys on. Before attempt 1 spends anything, the forge re-probes the spec’s accept gates and surfaces both degenerate kinds:

// RED-FIRST, before attempt 1 spends anything: an accept check that is
// already green on the untouched workspace cannot measure the work, and
// one that is red because its TOOL IS MISSING (exit 127) is red for a
// reason no code edit moves.
const probe = yield* probeAccepts(acceptGates, snapshotWorkspace(run.cwd))
yield* probe.vacuous.length > 0
  ? publish({ type: 'vacuous_checks', names: probe.vacuous })
  : Effect.void
yield* probe.missingTools.length > 0
  ? publish({ type: 'missing_tools', names: probe.missingTools })
  : Effect.void

Warn, not bounce — deliberately, on both. A vacuous check at forge means either the workspace drifted after lock or the check never went through propose at all (the spec file is plain markdown, and humans can edit it directly); the human can Esc. A missing-tool red is satisfiable — the coder can provision the tool into .local/bin, which now counts for the gates too — so refusing to run would be wrong in the other direction.

Note the asymmetry with propose: a missing-tool check sails through the propose-time probe untouched. It is red, and red is all propose demands. The classification waits for forge, because forge is where attempts get burned against it.

What running the check cannot catch#

The ledger, honestly.

Red is necessary, not sufficient. test -f out.txt fails on the untouched workspace and passes after touch out.txt. The probe eliminates checks that cannot measure the work; it is silent about checks that measure it badly. The distance between “fails before” and “passes exactly when the work is genuinely done” is still spec-writing skill — the refiner’s first, and mine at lock.

Only one species of wrong-red is classified. A check that is red because it is broken — a typo in a grep pattern, a path that will never exist — reads as properly red at propose and at forge, and can never go green. It will spend every attempt. One execution can prove a check can fail; no number of executions on the untouched workspace can prove its pass condition is reachable.

After lock, enforcement decays to advisory. The bounce exists only at propose. If the workspace moves between lock and forge — half the feature lands in someone else’s commit — an honestly-red check goes vacuous, and the forge warns and proceeds. The strict version would refuse to forge a drifted spec; today that judgment is the human’s Esc.

Enforcement-by-execution means execution. The refiner is otherwise bashless — read-only tools plus its one write. The probe runs model-authored one-liners on the real workspace, before any human has read them. A locked spec’s checks run at forge regardless, but by then the draft has been reviewed; at propose, nothing has. The thirty-second cap bounds how long a check runs, not what it does.

Where the discipline lives#

Red-first survives contact with an agent only as structure. The model had the instruction in capitals and wrote the vacuous check anyway; I had the draft in front of me and missed it; the component that caught it was the one that ran the command. So now the command runs at the moment it’s born — the one moment the verdict is unambiguous, the authoring agent is still on the line, and the fix is a re-propose instead of an unlock or a burned attempt. The spec gets refined in conversation, but conversation never decides. A gate declares victory at the end, and a gate declines the check at the beginning.