Timothy Part 9: Memory That Belongs to Me, Staged Facts That Only Supersede

Phase 2 gave the model tools, permissions, and skills so it can act inside a bounded workspace while the user keeps the final say. The next obvious gap was memory: every conversation started from a blank slate for the model, and nothing survived provider boundaries or session restarts except what I manually pasted back in.

Vendor-managed “memory” features were off the table. I want memory that belongs to Timothy, not to any model vendor’s opaque store. It has to survive swapping providers, work across sessions, and give the owner full visibility and veto power.

Typed, staged, supersede-only (D-011)

Three memory types because they age differently and should be retrieved with different emphasis:

• episodic: something that happened (“met with Ada on 2026-07-09 about the invoice”)
• semantic: a durable fact or preference (“prefers aisle seat on long-haul flights”)
• procedural: a how-to (“when debugging the payment flow, always check the ledger row first”)

Retrieval weights them (semantic 1.0, procedural 0.9, episodic 0.7) and lifecycle rules differ.

Facts never land directly in active retrieval. Agent-proposed memories start as pending. They become visible only after explicit promotion (user confirmation or a narrow auto-promote policy). Statuses are pending → active / rejected; later active → archived. The owner confirms (or edits) via the confirmation surface.

The key invariant: content rows are immutable. A correction does not UPDATE the text. It inserts a fresh row and sets superseded_by on the old one. The chain is walked with a seen map to detect cycles. Search and retrieval only ever return heads of chains. This mirrors the append-only event log from sessions and makes audit and “what changed” trivial.

One Postgres instance holds everything: pgvector for cosine, a generated tsvector for full-text, and a plain entities table. No separate vector database, no graph store to keep in sync.

-- migrations/0010_memory.sql
CREATE TABLE IF NOT EXISTS memories (
    id                uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    type              text NOT NULL CHECK (type IN ('episodic', 'semantic', 'procedural')),
    content           text NOT NULL,
    embedding         vector(1536),
    entity_refs       uuid[] NOT NULL DEFAULT '{}',
    ...
    superseded_by     uuid REFERENCES memories (id),
    status            text NOT NULL CHECK (status IN ('pending', 'active', 'rejected', 'archived')),
    confidence        real,
    tsv               tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
);
CREATE INDEX ... ON memories USING hnsw (embedding vector_cosine_ops) WITH (m=20, ef_construction=200);
CREATE INDEX ... ON memories USING gin (tsv);

Later retrieval bookkeeping added last_retrieved_at (0011), deliberately distinct from last_confirmed_at, which only moves on promotion or re-confirmation.

Extraction: the model reads, code decides

Extraction fires in two places, both best-effort so a slow or failing call never touches the user turn:

1. After a chat turn completes (fire-and-forget goroutine in the brain).
2. Pre-compaction, on its own slice of the compaction budget, so facts in turns about to be summarized away are harvested first.

One mini call proposes candidates. The system prompt is strict:

Reply with ONLY a JSON array … each content is ONE fact, self-contained (absolute dates, full names, no “he”/“it”) … Anything phrased as a rule … is semantic, NEVER episodic.

The response is decoded with DisallowUnknownFields, enum validation, range checks on confidence, and a hard cap (20 facts). Garbage → one retry → drop. No partial facts leak through.

Then:

• Batch embed (or degrade gracefully if no embedding provider yet).
• Nearest-active check: cosine ≥ 0.99 drops as exact dup; ≥ 0.95 marks for later consolidation.
• Insert as pending (unless the caller was the explicit remember tool).

Promotion policy lives in Go, not the model. Only episodic facts with confidence ≥ 0.8 auto-promote. Semantic and procedural always queue: preferences and standing instructions belong to the owner. Anything containing directive language (“always”, “must”, “from now on”, “rule”, credential-looking strings) is forced into pending even if the LLM gave it a high score. A false positive here just means one more item in the queue; a false negative on a directive would be a trust violation.

// internal/memory/extract/extract.go (simplified)
if m.Type == store.TypeEpisodic &&
    m.Confidence >= autoPromoteConfidence &&
    !looksDirective(m.Content) {
    _ = st.Promote(ctx, id)
}

The remember tool is the explicit-user path:

// internal/brain/tools/builtin/remember.go
// Use ONLY when the user explicitly asks to remember something...
// actor decides the entry status

It calls the memory client with actor=user, the store inserts directly as active, and the result is visible immediately. The permission system exempts it (you asked the tool to do exactly what you just told it to do).

Owner in the loop

Pending facts are confirmed, rejected, or edited (edit performs a supersede) through the authenticated surface; active memories and their supersede chains are browsable.

What deliberately does not happen

• No in-place content UPDATEs anywhere.
• last_confirmed_at is never bumped by mere retrieval or use.
• Extraction and retrieval are on separate services and clocks; failures degrade to no-op or empty results.
• Until an embedding provider is wired, the vector leg and near-dup detection run dark; text + entity legs still deliver recall. That mode was designed and tested as the launch state.

War stories worth keeping

The model sometimes emits zero answer text on tool-using turns (certain providers). Extraction used to gate on assistant text and silently skipped those turns. Now user words alone on a completed turn are enough.

A padding question about HTTP caching once became an “episodic fact.” The 180-day unretrieved archival job is the designed pressure relief; real usage data will drive any prompt tightening.

The budgeter that counted only raw content plus a flat 3-token overhead quietly overran once the fence, preamble, and per-item framing were added. Single-sourcing the framing strings between the packer and the renderer fixed it, and the property test now costs the exact final block.

Next up: how those confirmed facts actually come back into context on every turn, three retrieval legs, one token budget, a fence that cannot be escaped, and the injection point that keeps provider caches happy.

---

All of the above is on main as of the long-term memory merge (3d27906). The live stack and the unit/integration tests are the source of truth.