Part 15 built the engine: a state machine, an append-only event log, a driver that steps missions forward, workers that get a fresh context per turn, a sentinel tool that reports status honestly, and brakes that stop a mission from looping forever. That engine can drive a mission through plan, execute, review, and done. What it can’t do on its own is tell you whether “done” is true.
That’s the gap this part closes. A worker calling mission_status{"status": "done"} is a claim, not evidence, and believing it is how you end up with a mission that reports success over a workspace with no file in it.
The rule: evidence flips the flag, never a claim
Every PlanUnit has a Passes boolean. Exactly one thing is allowed to set it true: the harness’s own check of the actual filesystem and the actual exit code of a command it ran. Not the worker’s mission_status call, not the reviewer’s approve verdict on its own. Both feed into the decision, but neither is the decision.
Two checks do the work, in a fixed order:
// internal/brain/missions/verify.go
func CheckArtifacts(workRoot string, artifacts []string) []string
func RunVerify(ctx context.Context, workRoot, verifyCmd string) (VerifyResult, error)
CheckArtifacts runs first, before any model-authored verify_cmd gets near a shell. For every path a plan unit declared, it checks: the path is relative (an absolute path or a .. climb is rejected outright, never resolved), it exists inside the workspace, it isn’t a directory, and it isn’t empty. Only once every declared artifact clears that gate does verify_cmd run at all.
Why that order, and why insist on both? A verify_cmd is one POSIX shell command a planner writes for a unit, something like grep -qi 'retry-after' summary.md. A model that wants to look done for free can make that string a tautology: echo ok, exit 0, “verified.” CheckArtifacts closes that door: a shell command’s exit code proves nothing about content the harness hasn’t already confirmed exists on disk. Conversely, an artifact existing proves nothing about whether it’s correct, which is what verify_cmd is for. Existence and content are different questions, and each needs its own harness-run answer.
VerifyResult records evidence, not a verdict rendered somewhere else:
type VerifyResult struct {
ExitCode int
OutputSHA256 string
Excerpt string
Passed bool
}
The sha256 of the full output plus a bounded trailing excerpt means a mission’s outcome is auditable from its event log alone, long after the workspace is gone. Passed is ExitCode == 0, nothing else.
Why command substitution is illegal in verify_cmd, at plan time
A planner is a model. Nothing stops it from writing test "$(cat out.md)" = ok as a verify_cmd, and early on, one did. The check itself isn’t the problem; where it collides with an unrelated rule is. The permission classifier treats command substitution ($(...), backticks) as opaque, since it can’t statically know what a substitution will run, so the safe default is to ask a human. On a schedule-fired mission there’s no human to ask (more below), so the ask resolves as an automatic denial and the mission parks on a verify step that should have been mechanical.
The fix rejects the plan at submission instead of letting it survive planning and fail later, deeper in the pipeline and further from the actual mistake:
// internal/brain/missions/runner.go
for _, u := range spec.Units {
if strings.Contains(u.VerifyCmd, "$(") || strings.Contains(u.VerifyCmd, "`") {
return Spec{}, fmt.Errorf("mission runner: verify_cmd must not use command substitution: write the direct command instead")
}
}
This plugs into the same one-retry recovery path the planner already uses for malformed JSON or an empty plan: the error names the exact problem, the planner gets one more turn, and it corrects itself immediately instead of the mission stalling for a full permission-ask timeout. Input redirection (<) stays legal; only substitution is banned, narrow enough not to break commands like test -f x && wc -l < x.
The same opaque-command rule applies to a worker writing its own output. A shell redirect (echo "x" > out.md) or a heredoc is destructive-classified for the same reason substitution is, so it parks. Workers instead get a dedicated write_file tool, scoped to the mission’s own workspace, as one of the turn-scoped ExtraTools that shadow the base tool registry by name for that turn only. The system prompt is explicit: create or update files only with write_file, use shell for reading and checking, never for writing. That isn’t a style preference, it’s what makes CheckArtifacts trustworthy: a file written through a tool with a fixed, auditable contract is a very different thing from a file that may or may not have landed after a shell pipeline nobody watched run.
The reviewer’s job is to find a reason to say no
Harness checks catch existence and mechanical correctness. They don’t catch “this diff doesn’t actually solve the stated problem” or “this summary makes a claim the source doesn’t support.” For coding missions, and for any unit without declared artifacts to check, a reviewer turn runs before a unit can close.
The reviewer runs on its own route (m.ReviewRoute), deliberately not the same model or chain the worker used: the point of a second opinion is that it isn’t the first opinion marking its own work. The one tool it can call, review_verdict, makes rejection the well-supported path and approval the fallthrough:
// internal/brain/missions/review.go
Description: "Report your review verdict. Call this exactly once. Look for reasons
to reject before approving, approve only when you cannot find a real gap."
A rework verdict requires a findings array: one entry per gap, each with a title, an optional file, and a detail explaining what’s wrong and what would fix it. Free-text disapproval isn’t enough; it has to be itemized so the worker’s next turn has something concrete to act on.
What the reviewer sees matters as much as its posture. ReviewPacket is built from the harness’s own reads, not the worker’s account: the mission’s goal and plan first (an earlier failure mode was reviewers rejecting for “missing original mission goal,” because they’d never been given it), the actual artifact bytes read from disk by ReadArtifacts, a flat listing of the whole workspace so the reviewer sees what exists even when a unit declared nothing, the git diff against the mission’s base commit for coding work, and the worker’s own evidence text last, ordered deliberately as the least trustworthy input.
Resuming the same reviewer, and dropping it when it’s stuck
A rework verdict isn’t the end of the review; it’s the start of a loop between one worker retry and the same reviewer session. The driver keeps a GatekeeperState per mission, the reviewer’s own message history, and passes it back in on the next round, so a delta recheck can look at what changed since the last verdict instead of re-reading and re-litigating the whole workspace.
But a resumed reviewer anchored to its own earlier judgment can get stuck defending it. The driver fingerprints each rejection’s findings (title and file, sorted, hashed, deliberately excluding the free-text detail so the same gap phrased differently still collides) and compares it to the previous round’s fingerprint. Two identical rejections in a row drop the gatekeeper state entirely, so the next round reads everything with genuinely fresh eyes instead of re-confirming what it already believes.
One more rule closes a side effect nobody asks for: a reviewer with shell access might run tests to check a claim, leaving the worktree dirtier than the worker left it. After every review round, win or lose, the driver rolls the worktree back unconditionally, so the reviewer’s own side effects never leak into the next round’s diff.
Skipping review when there’s nothing left for it to catch
Not every unit needs a second opinion. If a non-coding unit declared artifacts, and CheckArtifacts plus verify_cmd both already passed against harness evidence, an LLM review round adds tokens, latency, and the reviewer’s own failure modes on top of a question already answered mechanically:
// internal/brain/missions/driver.go
func (d *Driver) trySkipReview(ctx context.Context, m Mission) (StepInput, bool) {
if m.Kind == "coding" {
return StepInput{}, false
}
unit, idx := currentUnit(m.Spec)
if unit == nil || len(unit.Artifacts) == 0 {
return StepInput{}, false
}
...
}
Coding missions always review, because a diff can be wrong in ways an existence check never sees. Units with no declared artifacts always review too, because there’s no harness evidence to stand on. This isn’t a shortcut around scrutiny, it’s the same trust rule applied one level up: where deterministic evidence already settled the question, an LLM’s opinion is redundant, not additional safety.
The war story: two directories, one bug
The clearest illustration of “evidence, not claims” backfiring on itself came from a batch of missions given the same trivial research goal. Ten of twelve failed. The failures all traced to one bug: the worker’s shell tool executed in a single global workspace root, while verify_cmd and the reviewer both looked in the mission’s own per-mission directory. A worker would write its file, correctly, into the shared root; the harness would then check for that file in a different directory entirely and, correctly, find nothing there.
The failure mode inverted expectations: a worker whose verify_cmd actually checked file content (test -f out.md) failed reliably, while a worker whose verify_cmd was a tautology (echo ok) passed every time, the exact opposite of what verification should reward. The shared root was also quietly a dumping ground, with artifacts from unrelated missions colliding in the same directory.
The fix was the same turn-scoped ExtraTools mechanism described above, rooted correctly: missionTools builds the shell and write_file tools from m.WorkRoot(), the mission’s own worktree-or-workspace path, not a fixed global constant. Bundled into the same fix: reviewers had also been missing the mission goal and real artifact contents, and the harness’s own protocol tools were themselves getting stopped at the permission gate meant for model actions. All three were symptoms of one lesson: the state machine’s transition logic was sound, but the places where it touched the real filesystem and real model turns were where the bugs lived.
Fail fast when nobody is watching
A permission ask that can’t be answered isn’t neutral, it’s a stall. A schedule-fired mission hitting an opaque shell command, or a model calling a tool name that doesn’t exist, used to park on the same interactive prompt a live chat session uses, for the full deny timeout, with no one there to click anything.
Two changes closed that. First, a tool call is checked against the turn’s actual offered tool surface before it ever reaches the permission chain:
// internal/brain/loop/agent.go
if !toolNames[call.Name] {
return fmt.Sprintf("unknown tool %q, use one of the tools you were given", call.Name), "error"
}
A hallucinated tool name is now a same-turn error the model can correct, not a park. Second, for a turn marked Unattended (any mission with a schedule_id), a permission decision of “ask” resolves as an immediate, explicit denial instead of a park:
if unattended {
return fmt.Sprintf("permission denied automatically (unattended mission): %s. "+
"No human is available to approve. Rewrite the call to avoid the flagged "+
"pattern, create files with write_file instead of shell redirects, avoid "+
"command substitution, or use only tools the agent's allowlist grants.", res.Rationale), "denied"
}
Missions created interactively through the web UI keep the normal park-and-wait behavior; a human is actually there to answer. The distinction is who’s watching, not what kind of mission it is.
Why the docker socket moved out of the brain
Running model-authored shell commands anywhere real means running them somewhere. The first version ran them via exec.CommandContext inside the brain’s own process. Worker and reviewer shell calls and verify_cmd inherited brain’s full environment (database credentials, the master key, provider tokens) and whatever binaries happened to be in brain’s own container image, which is exactly why one real mission got stuck needing python3 that brain’s minimal image never shipped.
The fix moved execution into a per-mission Docker container with fixed resource caps (memory, CPU, process count) so a runaway command can’t starve the host. But that container has to be created and driven by something holding the Docker socket, and the Docker socket is root-equivalent on the host. Giving brain that socket meant a compromised brain process was a compromised host, not a compromised sandbox.
sandboxd exists to shrink that blast radius to exactly what a mission is already allowed to do. It’s a separate compose-internal service; it alone mounts /var/run/docker.sock, and it exposes four routes: exec (SSE-streamed output), remove, list, and health. Mission IDs are validated against a strict UUID pattern, and every container property (image, user, resource limits, mounts, network mode) is fixed server-side, never accepted from the caller. Brain talks to it through a small sandboxclient package with the same function signature the in-process executor had, so nothing above that boundary needed to change. The container itself runs read-only with every Linux capability dropped.
The honest framing: this doesn’t make a compromised brain harmless. It can still tell sandboxd to run arbitrary commands inside mission containers, because that’s precisely what missions are supposed to do. What it removes is the jump from a compromised brain to a compromised host, a very different and much worse thing.
Resolving symlinks before anything executes
A quieter gap in the same family: the permission chain’s path containment check was lexical only, inspecting only absolute tokens as strings. A symlink sitting inside a workspace and pointing outside it, or a relative path token, could route a shell command or a write_file call past containment undetected. WithinRoot already resolved symlinks correctly, it just had no caller where it mattered. The fix wired it in: CheckCommandPaths re-validates every path-like token in a shell command, symlinks resolved, immediately before it execs, and write_file checks its resolved target the same way before writing.
What’s still rough
Weak worker models occasionally invent artifact filenames that don’t match what the plan declared. The mitigation, rendering the exact declared paths in the worker’s own packet and listing what actually exists on an artifact-check failure, reduces the problem; it doesn’t eliminate it. And the reviewer is, by its nature, the least reliable link in this chain: an LLM judging LLM output, which is exactly why it never gets the final word alone. Harness evidence still has to agree.
Next: missions on a clock, schedules, notifications, spend, and a canary.