In Part 1 I explained why I am building Timothy, my self-hosted personal AI assistant. Before writing any code, I wrote the architecture down and forced myself to defend every choice. This post is that design.
One rule shapes everything (D-001): deterministic code orchestrates; each LLM session does exactly one thing.
Why never let the LLM manage state or other LLMs?
Every AI system I have watched fail in production failed the same way: an LLM was asked to manage state, or manage other LLMs, and it lost coherence, defended its own mistakes, or confidently declared unfinished work done. Models are brilliant and unreliable. So in Timothy, Go code owns every loop, every state transition, and every safety decision. The model gets one well-framed job at a time.
Why Exactly Three Services?
Why three services instead of one monolith or a more fine-grained split?
I have seen enough microservice sprawl to be suspicious of it, and enough monoliths to know where the seams belong. Three backend services is the minimum that separates three concerns that genuinely must not mix (this is D-014):
The credential boundary is the important one. Only the gateway holds API keys. If the brain is compromised through a prompt-injected tool call, there is no key to steal: the worst it can do is send requests through the gateway, where every one of them lands in the ledger.
Browser
│
├── web :3300 (chat, dashboard, settings)
│
└── brain :8300 public API, agent orchestration, scheduler
├── gateway :8081 provider credentials, routing, cost ledger
└── memoryd :8082 long-term memory, retrieval
| Service | Owns | Never touches |
|---|---|---|
| brain | sessions, the agent loop, tools, permissions, scheduling | provider credentials |
| gateway | provider drivers, routing, streaming, the cost ledger | conversation content it doesn’t need |
| memoryd | typed memories, hybrid retrieval, compaction jobs | providers (it calls the gateway like everyone else) |
The credential boundary is the important one. Only the gateway holds API keys. If the brain is compromised through a prompt-injected tool call, there is no key to steal: the worst it can do is send requests through the gateway, where every one of them lands in the ledger.
Everything else I deliberately did not build: no message broker, no separate tool-runner service, no cache tier. A single-user workload does not need them, and every service I skip is one that cannot page me.
One Database, All State
Why one Postgres (with pgvector) instead of specialized stores?
Postgres with pgvector is the only datastore: sessions, memory, provider configuration, cost ledger, task state, all of it (D-015, D-011).
I was tempted by the fashionable stack: a vector database for embeddings, a graph database for entity relationships, an event store for sessions. Then I counted the synchronization bugs I would be signing up for. Postgres does all three jobs well enough at my scale: pgvector for semantic search, plain full-text search for keywords, and foreign keys for the entity graph. One backup target, one restore drill. The decision keeps the system simple and the data consistent.
How a Request Flows
A chat message makes this trip:
- web → brain: the message arrives over REST, the reply streams back over SSE
- brain → memoryd: fetch a token-budgeted block of relevant memories
- brain assembles the prompt: system prompt, skill index, memory block, conversation
- brain → gateway: request by task type (reasoning, coding, mini, summarize, realtime, plus image, video, and speech for media)
- gateway picks the provider from a routing table, streams the response, and writes a ledger row: tokens, latency, cost
- brain runs the tool loop if the model wants to act, persists the turn, streams everything to the browser
The task-type indirection in step 4 is what makes multi-provider life pleasant. Nothing in the brain knows model names. A routing table maps each task type to an ordered chain of provider + model; the table is edited via SQL (a panel later); the gateway hot-reloads it. Swap a model, reorder a fallback chain, disable a provider that is having a bad day, no deploys.
Providers Are Data, Not Code
The gateway supports three driver implementations behind one interface: a native Anthropic driver, one OpenAI-compatible driver that covers everything speaking that dialect (Z.AI, xAI, OpenRouter, local runtimes), and a Bedrock driver for first-party Nova and Titan models. Configured instances differ only by base URL (or region), model list, and headers.
CLI drivers (subprocess for subscriptions that have no API) are planned for the same interface: same routing table, same ledger. Secrets are always by reference.
Secrets never live in the database. A provider row stores a reference (the name of an environment variable locally, a Vault path in production) and the gateway resolves it at call time.
Sessions Are Event Logs
Every session is an append-only sequence of events: user messages, assistant turns, tool executions, compactions. Two different views project from the same log: a rich transcript for clients, and a lean context for the model. Nothing is ever updated in place, so a container restart mid-conversation loses nothing.
Long conversations stay affordable through two mechanisms. After each turn, raw tool traffic is distilled into a compact structured residue (what changed, what failed, what was learned) instead of dragging full traces through every subsequent request. And when the context approaches its budget, older turns are summarized automatically, with one critical detail: durable facts are extracted to memoryd before summarization, because summaries lose exactly the names, dates, and commitments that matter most.
Memory With a Confirmation Gate
Memories are typed (episodic events, semantic facts, procedural preferences) and nothing an agent writes becomes truth directly. Extracted facts land in a pending state; routine observations auto-promote, but anything touching preferences, identity, or standing instructions waits for owner confirmation. Facts are never edited in place, only superseded, so I keep a full audit trail of what Timothy believes and why.
Retrieval runs three searches in parallel (vector similarity, full-text, entity lookup), fuses the rankings, weights by recency and type, and returns a fixed token budget. Below a relevance threshold it returns nothing at all: an assistant with no memory of something beats an assistant confidently wrong about it.
Safety Lives in Code
Why put safety in Go middleware instead of the system prompt?
Timothy will run unattended tasks, which makes this the part I refuse to compromise on (D-009, D-010). Every tool call passes through middleware: workspace path allowlists, argument clamps, step ceilings. Permissions resolve through a fixed chain: hard policy guard, project allowlist, session grants, and finally an interactive prompt. A danger classifier forces confirmation for destructive shell commands no matter what any allowlist says.
None of this is in a prompt. A prompt can be persuaded; conditional logic cannot. The meta-principle (D-001) puts all orchestration and safety in deterministic code.
The Decisions, Written Down
I keep every binding choice as a numbered decision record in the repository, twenty-one so far, from “deterministic orchestration” to “curated content is a knowledge library, not memory.” Each one states its context and consequences, and when the implementation forces a deviation, the record gets amended in the same commit.
It is the same discipline I apply at work, and it earns its keep the first time future-me asks “why on earth did I do it this way?”
In the next post I start the build for real with the LLM gateway: normalized streaming, data-driven routing, and the cost ledger. The two posts after that cover the brain’s first authenticated chat API and the initial streaming surface that closes the loop for the foundation.