A real chat turn ran for roughly thirty minutes against a remote model on a CPU-only homelab box. When it ended, the session had nothing to show for it. No assistant message. No error event. No pending_state partial. The session_events log for that turn was empty, and turn_active had silently flipped to false. From the database’s point of view, the turn never happened.
This post is the story of the five commits that closed that hole, one layer at a time. Each fix revealed the next one underneath it. By the end the original symptom was impossible, but only because four different producers of the same event had been patched individually, and a fifth, structural backstop had been welded in underneath all of them.
The contract being defended
Everything in this story comes back to one comment in the gateway’s stream package:
// internal/gateway/stream/events.go
// Channel contract: a stream ends with exactly one terminal event:
// "done" (provider finished, possibly preceded by "incomplete" when
// the stream was cut off) or "error" (nothing more is coming), and
// the channel closes immediately after.
Every provider, no matter its wire format, translates into this normalized vocabulary. Every consumer downstream of it, the gateway’s own router, brain’s gateway client, the agent loop, the chat persistence path, the missions runner, the SSE handler that faces the browser, assumes that contract holds. A stream that ends with zero events, or two terminals, or a terminal that never gets delivered, is not a well-formed empty answer. It’s a broken turn.
D-044 is the durable-evidence half of the same promise: a turn that ran must leave either real content or a real failure event in session_events, the append-only log from Part 6. It must never vanish silently. The whole post is about defending that one line of comment, at every layer it got broken.
Chapter 1: the proxy that kills idle connections
The only externally caused bug in this chain was also the one that exposed the rest of them.
Part 21 detached a chat turn from its HTTP request. A turn now runs on its own context with a 30-minute ceiling, and closing the browser tab no longer kills the answer that was almost done. That was the right trade, but it had a consequence nobody had seen yet: a detached turn can run long enough to trip a completely different class of timeout.
Cloudflare’s 524, and similar proxy timeouts on other front-ends, do not fire on total request duration. They fire on a gap with no bytes flowing. A slow reasoning model on an overloaded CPU can go on the order of 100 seconds between tokens on a long turn. From the proxy’s vantage point, nothing has moved for a minute and a half. The connection looks dead. So the proxy kills it, mid-generation, and the turn that was about to produce its next chunk gets cut off for the sin of being slow.
The fix is a heartbeat. Every 20 seconds, the SSE handler writes a comment line that carries no event name at all:
// internal/brain/api/api.go
// streamHeartbeat keeps a chat SSE connection alive through proxies
// that kill an idle one (Cloudflare's ~100s 524 fires on a gap with no
// bytes flowing, not on total duration): a slow model can go well
// past that between tokens. A comment line carries no event, so it
// never reaches the client's SSE parser or reducer.
const streamHeartbeat = 20 * time.Second
The relay loop now selects between two cases, the events channel and the ticker:
// internal/brain/api/api.go
ticker := time.NewTicker(streamHeartbeat)
defer ticker.Stop()
for {
select {
case ev, ok := <-events:
if !ok {
send(m)
return
}
// ... forward ev to the client ...
case <-ticker.C:
_, _ = fmt.Fprintf(w, ": ping\n\n")
flusher.Flush()
}
}
A : ping line is an SSE comment. It has no event name, so the browser’s EventSource parser discards it, and Timothy’s web client reducer never sees it surface as an event. It exists purely to put bytes on the wire often enough that the proxy stops counting the gap. The same ticker got wired into the /live re-attach stream, because a client that reconnects mid-turn needs the same protection.
The second half of the fix is on the client. A dropped connection is no longer a hard failure. Because a Timothy turn runs detached from any single HTTP request (D-042, from Part 21), it is very likely still running server-side when the network blips or the laptop sleeps. So the web client reattaches:
// web/src/pages/Chat.tsx
// A transport-level failure (dropped connection, proxy timeout,
// laptop sleep) throws a plain error, not ChatError: the turn
// itself runs detached from this request (D-042), so it's very
// likely still alive server-side. Reattach via /live instead of
// surfacing a false "failed" state.
toast.info('Connection dropped - reattaching')
handedOffToLive = true
attachLive(sessionRef.current)
Only a plain TypeError (the transport got cut) triggers this path. A definite ChatError, a real server response with a status code, still falls back to a transcript refetch, because that’s a genuine failure, not a network event. attachLive itself caps retries at maxLiveReconnects = 5 so a turn that is genuinely gone doesn’t loop the reattach forever.
This was the only chapter where the enemy was outside the system. Once turns could survive 30 minutes, they exposed everything else.
Chapters 2 and 3: the provider and the gateway that forgot to tell the client
The heartbeat kept the proxy from killing the connection. But a slow model on a CPU-only box could still drop on its own, when the remote Ollama process hiccuped or the host ran out of memory. And when it did, the next two holes in the contract opened up, one at the provider layer and one at the gateway.
The provider layer has a small helper called emit that every HTTP driver uses to push normalized events onto its channel:
// internal/gateway/provider/httpx.go
// emit sends an event unless ctx is done; it reports whether the send
// happened. Streams must never block forever on an abandoned consumer.
func emit(ctx context.Context, ch chan<- stream.StreamEvent, ev stream.StreamEvent) bool {
select {
case ch <- ev:
return true
case <-ctx.Done():
return false
}
}
That ctx.Done() case is the right instinct for most events. A stream whose consumer has gone away should not block forever on a send nobody will read. But it is the wrong instinct for the terminal error that runStream emits when the upstream connection dies, because that terminal is the only evidence the attempt failed. When the upstream connection (remote Ollama) drops at nearly the same instant the parent context cancels, the terminal-error emit loses its race against ctx.Done() and returns false. The provider channel closes having delivered zero events. No chunk. No done. No error. A well-formed empty SSE response, from the gateway’s point of view.
The gateway’s streamAttempt then ranged over that channel with a for ev := range ch loop, and the loop body never ran once. The local result stayed all zero values. Nothing was sent to the client. The result was indistinguishable from a legitimate empty completion, so nothing failed over and nothing surfaced. Brain fell back to a generic “stream ended without a terminal event” message that told nobody anything.
The fix in #156 tracks whether any terminal was ever seen:
// internal/gateway/api/api.go
sawTerminal := false
for ev := range ch {
switch ev.Type {
case stream.EventError:
sawTerminal = true
// ...
case stream.EventIncomplete:
sawTerminal = true
// ...
case stream.EventDone:
sawTerminal = true
// ...
}
}
// A provider channel can close having delivered nothing at all (no
// chunk, no EventDone, no EventError) if runStream's own terminal
// emit lost its race against the parent context right as the
// upstream connection cut (httpx.go's emit silently no-ops when its
// ctx is done).
if !sawTerminal && !res.failed {
res.failed, res.reason = true, "provider stream closed without a terminal event"
res.entry.ErrorCode = "stream_cut"
}
With res.failed = true and nothing having streamed, the chain can fail over to the next provider. The ledger records stream_cut. The client gets a reason.
But #156 only mutated streamAttempt’s local result. That’s correct when no content reached the client, because failover handles it from there. The problem is what happens once content already streamed. Once bytes have crossed the wire to the client, failover is off the table (the gateway’s own bright line: errors after content are relayed honestly, never silently restarted elsewhere). In that case #156 recorded stream_cut in the ledger correctly, but the SSE stream just closed after the last chunk with no terminal frame sent to the client at all.
Brain’s persistTurn knows how to handle a real EventError frame. It just never got one. A mid-stream cut against a real 30-minute attempt left zero session_events for the turn: no assistant message, no error, turn_active silently flipped false. The turn ceased to exist in the durable record.
That was #157. The fix mirrors the existing EventError case’s streamed-vs-not-streamed split. If content already streamed, send an explicit terminal error frame:
// internal/gateway/api/api.go
if !sawTerminal && !res.failed {
res.failed, res.reason = true, "provider stream closed without a terminal event"
res.entry.ErrorCode = "stream_cut"
if res.streamed {
send(stream.StreamEvent{Type: stream.EventError, Err: &stream.StreamError{
Code: "stream_cut",
Message: res.reason,
}})
}
}
The gateway now tells the client when a stream cuts mid-flight, whether or not bytes have crossed. That left the brain-side hole.
Chapters 4 and 5: the brain-side race, and the root cause underneath
Brain’s gateway client, gwclient, had its own synthetic terminal for exactly this case. If the HTTP read failed without a real terminal having arrived, it would synthesize a gateway_stream_cut error event and push it onto the channel. The persistence path would then have something to record.
The guard on that synthesis was the problem:
// internal/brain/gwclient/gwclient.go (before #158)
if err != nil && !sawTerminal && ctx.Err() == nil {
The context passed into gwclient.Stream is turnCtx, brain’s own 30-minute turn ceiling. When a real long turn’s deadline expires at nearly the same instant as a genuine upstream failure, ctx.Err() is already non-nil by the time the read error surfaces. So the guard skipped the fallback entirely. persistTurn had nothing. The turn vanished.
This is the exact race the synthetic terminal existed to catch, and the guard was skipping it on the exact condition that made the race fire.
#158 dropped the ctx.Err() == nil condition, so the synthesis always attempts delivery. It still raced the send against ctx.Done() so it would never block forever on an abandoned consumer:
// internal/brain/gwclient/gwclient.go (#158)
if err != nil && !sawTerminal {
select {
case ch <- stream.StreamEvent{Type: stream.EventError, Err: &stream.StreamError{
Code: "gateway_stream_cut", Message: err.Error(), Retryable: true,
}}:
case <-ctx.Done():
}
}
Here is the honest sharp edge. That fix’s regression test failed once in CI despite passing locally. The failure was not a flake in the test. It was the test correctly reporting that the fix was nondeterministic. Once ctx is already done, the delivery select races ch <- against ctx.Done(). Both cases are ready. Go picks randomly. A single attempt can land either way depending on the scheduler.
The test was rewritten to run 40 independent attempts and require the terminal to win at least once:
// internal/brain/gwclient/gwclient_test.go (#158)
// The fix's delivery races ch<- against ctx.Done() once ctx is
// already done when the read error surfaces (never blocking
// forever on an abandoned consumer is the whole point of racing
// against ctx.Done() at all), so a single attempt can't
// distinguish "fixed, lost this particular race" from "the old,
// unconditionally-skipped bug". Run many independent attempts and
// require the terminal to win at least once.
const attempts = 40
delivered := 0
for i := 0; i < attempts; i++ {
// ... set up gateway stub, cancel ctx, close release, count deliveries ...
}
if delivered == 0 {
t.Fatalf("gateway_stream_cut never delivered across %d attempts, want at least one", attempts)
}
Confirmed against a real 30-minute turn: gateway logged completion with error_code=provider_error right after the gateway-side fixes landed, yet still produced no session event. The brain-side synthetic terminal was getting dropped on the floor. And the probabilistic test was the warning sign that #158 was not the real fix, just a closer approximation of one.
The real root cause was that four different producers treated end-of-turn delivery as a best-effort select racing ctx.Done(). When the 30-minute turn ceiling and a slow provider’s request_timeout (D-041) landed at the same instant, both select cases were ready, Go picked randomly, and a lost coin-flip dropped the only evidence a turn existed. #159 is the commit that stopped racing.
The same design, applied at every site:
// internal/brain/gwclient/gwclient.go (#159)
// Delivery deliberately does NOT race ctx.Done(): with both select
// cases ready (deadline fired, consumer mid-receive) Go picks
// randomly, and that coin flip has eaten the only evidence of a real
// ~30min turn. Every consumer of this channel drains until close, so
// the send succeeds immediately in practice; the timeout only bounds
// a hypothetically abandoned consumer.
if err != nil && !sawTerminal {
select {
case ch <- stream.StreamEvent{Type: stream.EventError, Err: &stream.StreamError{
Code: "gateway_stream_cut", Message: err.Error(), Retryable: true,
}}:
case <-time.After(5 * time.Second):
}
}
The 5-second bound replaces ctx.Done(). The justification is an invariant: every consumer of this channel drains until close, so the send succeeds immediately in practice. The timeout only exists to bound a hypothetically abandoned consumer that should not exist. That is a deliberate trade. A genuinely wedged consumer holds a goroutine for 5 seconds longer than the old ctx.Done() race would have. In return, a frequently-eaten terminal stops getting eaten. The test reverted to a single deterministic assertion, because the fix itself was deterministic again.
The agent loop got the same treatment. A new closure, emitFinal, delivers the usage-and-terminal pair without racing ctx.Done():
// internal/brain/loop/agent.go
emit := func(ev stream.StreamEvent) {
emitMu.Lock()
defer emitMu.Unlock()
select {
case out <- ev:
case <-ctx.Done(): // the race that ate terminals
}
}
// emitFinal delivers end-of-turn events (the usage/terminal pair)
// without racing ctx.Done(): when the turn deadline fires at the
// same instant the terminal is sent, both select cases in emit are
// ready and Go picks randomly; a lost flip drops the only evidence
// the turn ever existed.
emitFinal := func(ev stream.StreamEvent) {
emitMu.Lock()
defer emitMu.Unlock()
select {
case out <- ev:
case <-time.After(5 * time.Second):
}
}
All four terminal-producing exit points in the loop, gateway failure, normal completion, force-synthesis, and max-iterations, now route through emitFinal. The loop’s synthesized EventIncomplete can no longer be coin-flipped away.
And then the structural backstop, in persistTurn. If every upstream producer lost its terminal and there is no text and no failure, synthesize one:
// internal/brain/chat/chat.go
// No failure AND no text at all means every upstream producer lost
// its terminal on the way here (deadline racing the stream cut).
// Synthesize one rather than append nothing: a turn that ran must
// never leave zero events. A flushed partial (text non-empty,
// already checkpointed) keeps today's behavior: the pending state
// is the evidence.
if failure == nil && text == "" {
failure = &session.TurnFailed{
Code: "turn_aborted",
Message: "turn ended with no terminal event (deadline exceeded or stream cut)",
}
}
Zero-evidence turns are now structurally impossible. Even if every producer above this point loses its race, the persistence path itself refuses to append nothing.
The missions runner got the same sawTerminal discipline. A bare channel close is now an infra error instead of a clean empty verdict. Before this, a silent cut read as a missing sentinel, and the harness burned a recovery re-run plus a forced retry: two full model turns for one silent infrastructure failure. EventIncomplete is now an infra error too, not a short clean answer. And an EventError with a nil Err no longer panics (the runner used to dereference ev.Err.Message unconditionally; chat’s noteFailure already guarded this, the runner didn’t).
Two more fixes shipped in the same commit because they shared a failure mode with the terminal-loss class: a single bad event silently killing a turn.
The loop’s executeOne now recovers a panicking tool into an error result for the model, rather than letting it crash the process:
// internal/brain/loop/agent.go
// A panicking tool must become feedback the model can read (D-009),
// never a process crash: executeOne runs inside an errgroup worker,
// and an unrecovered panic there takes down the whole brain, every
// active turn and mission with it.
content, status := func() (content, status string) {
defer func() {
if p := recover(); p != nil {
a.logger.Error("tool panicked", "tool", call.Name, "panic", p, "stack", string(debug.Stack()))
content, status = fmt.Sprintf("tool %s failed with an internal error: %v", call.Name, p), "error"
}
}()
return a.resolveAndRun(ctx, exec, sessionID, call, toolNames, unattended, emit)
}()
That one is unrelated to streaming. It shares the shape: a single unexpected event takes down the whole turn. An errgroup worker panic does not just kill its own turn, it takes the whole brain with it.
And TURN_TIMEOUT is now an env-gated override on the compiled 30-minute ceiling, with a floor:
// cmd/brain/main.go
// TURN_TIMEOUT raises the detached-turn ceiling above the compiled
// 30m default, needed when a route serves a slow CPU-only backend
// whose provider request_timeout (D-041) would otherwise collide
// with the brain ceiling and manufacture a deadline-vs-terminal
// race at the exact same instant. Env-gated feature, default off.
if v := os.Getenv("TURN_TIMEOUT"); v != "" {
d, err := time.ParseDuration(v)
if err != nil {
app.Log.Warn("invalid TURN_TIMEOUT ignored", "value", v, "error", err)
} else if err := svc.SetTurnTimeout(d); err != nil {
app.Log.Warn("TURN_TIMEOUT rejected", "value", v, "error", err)
} else {
app.Log.Info("turn timeout overridden", "timeout", d)
}
}
The floor is enforced in SetTurnTimeout: values at or below 10 minutes are rejected, because 10 minutes is the permission-park timeout, the window a turn spends waiting on a user’s yes/no on a tool call. A ceiling at or below that floor means a parked ask could never be answered in time. This is opt-in, not on by default, because raising the global turn ceiling has a real cost: longer-lived detached goroutines and a larger memory hold per active turn. It exists for the slow CPU-only backend whose request_timeout keeps colliding with the turn ceiling, attacking the root cause directly instead of patching the race it produces.
The recurring shape
This is the same species of bug the series keeps running into, last seen in Part 22: a mechanism has more than one enforcement point, the layers live in different packages, and nothing forces them to be touched together.
The contract in events.go is one sentence. Defending it took five commits across four packages: the provider helper, the gateway stream attempt, the brain gateway client, the agent loop, the chat persistence path, the missions runner. Each layer could be closed individually, and each individual closure was correct, but the contract only held once every layer was closed. The gateway fix in #157 was correct on its own terms. It still left the brain-side hole in #158 wide open, because the gateway recording stream_cut in its ledger does not help a persistTurn that never receives an error frame.
The test harness itself has a small defense against this class regressing silently. The scriptedAgent fake in runner_test.go used to let any batch of events close the channel bare. It now auto-appends an EventDone to any batch that doesn’t already end on a terminal, because the real loop.Agent guarantees exactly one terminal before close. Only the terminal-loss regression tests opt out, with rawClose: true. Every other test gets the contract enforced by the fake.
What the chain cost, and what it gained
The honest accounting. The 5-second bounded send trades a rare stuck goroutine for a frequently-eaten terminal. A genuinely wedged consumer, which by the drain-until-close invariant should not exist, holds a goroutine 5 seconds longer than the old ctx.Done() race would have. That is the cost. It is deliberate.
The turn_aborted synthesis in persistTurn is a structural safety net. The user always sees something. But it is the last backstop in a chain of four producers, and if it fires at any frequency, an upstream terminal-delivery path has regressed. The code comment says so plainly: it exists because every upstream producer lost its terminal on the way here. It is not license to stop defending the producers above it.
TURN_TIMEOUT is opt-in, not a default, because raising the global turn ceiling has real cost in goroutine lifetime and memory hold.
And #158 was nondeterministic by design. Its probabilistic test was a warning sign that the root cause was still one commit away. #159 made the fix deterministic by making the code deterministic. The test reversion, from 40 attempts to a single assertion, is the proof that the underlying race was gone.
What the system gained is simple to say and took five commits to make true. A turn that ran must always leave a trace. The contract in events.go, exactly one terminal event per stream, is now defended at every layer it crosses. The original symptom, a 30-minute turn vanishing with zero session_events, is structurally impossible: every producer that could drop the terminal has been given a bounded delivery path, and the persistence path itself refuses to append nothing as a last resort.
The durable record was always the point. The session_events log from Part 6 is only worth anything if it is a complete record, and a complete record is one where a turn that ran is a turn that appears.
Landed across #155 through #159, 214c529 through 4cbd3eb on main.