Timothy Part 3: The LLM Gateway, Providers as Data, Streaming, and a Cost Ledger

The first request that ever flowed through Timothy’s gateway cost me nothing, not because it was free, but because the ledger refused to guess.

Why a cost ledger that records NULL rather than guessing prices?

GLM-4.7 took 11 input tokens, answered with 204 output tokens in 11,988 ms, and the cost column stayed NULL: I had not configured prices yet, and a ledger that invents numbers is worse than no ledger (D-013). Once prices landed, the next short chat turn recorded $0.000873. That is the level of accounting I wanted from day one.

In Part 2 I laid out the architecture and the rule that shapes everything: deterministic code orchestrates; each LLM session does exactly one thing.

The first real code I wrote was the gateway. It is the only service that ever sees a credential. Everything else (the brain, memoryd, clients) talks to it over a clean internal interface and never learns an API key or how to call Anthropic versus xAI.

Providers Are Data, Not Code

Why store providers, models, and routing as data in Postgres instead of hard-coded in Go?

I did not want a new Go file every time I added a model or a provider (D-004). Instead, providers live in Postgres.

-- migrations/0002_gateway.sql
CREATE TABLE IF NOT EXISTS providers (
    id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    name           text UNIQUE NOT NULL,
    kind           text NOT NULL CHECK (kind IN ('api', 'cli')),
    driver         text NOT NULL,
    base_url       text NOT NULL DEFAULT '',
    ...
    models         jsonb NOT NULL DEFAULT '[]',
    credential_ref text NOT NULL DEFAULT '',
    enabled        boolean NOT NULL DEFAULT false,
    ...
);

A row for Anthropic looks like this at seed time:

• driver = “anthropic”
• credential_ref = “ANTHROPIC_API_KEY”
• models = array of objects with id, capabilities, and prices (USD per million tokens)

The same table holds Z.AI and xAI as “openaicompat” drivers with their base URLs and different price structures. CLI drivers (actual claude -p or similar) will land in the same table later.

At startup the gateway reads the enabled rows, resolves each credential_ref against the environment (or later Vault), and builds a live Registry. If the ref is empty, the provider is marked unhealthy and routing skips it automatically.

One Normalized Stream for Every Provider

Why normalize every provider into a single event vocabulary instead of leaking their native formats?

Different providers speak different protocols. Anthropic has its own SSE shape, tool-use blocks, and prompt-caching headers. OpenAI-compatible endpoints are close but never identical (D-013).

I refused to let that leak into the rest of the system.

// internal/gateway/stream/events.go
const (
    EventChunk          EventType = "chunk"
    EventReasoningChunk EventType = "reasoning_chunk"
    EventToolStart      EventType = "tool_start"
    EventToolEnd        EventType = "tool_end"
    EventUsage          EventType = "usage"
    EventRetry          EventType = "retry"
    EventIncomplete     EventType = "incomplete"
    EventDone           EventType = "done"
    EventError          EventType = "error"
)

type StreamEvent struct {
    Type     EventType
    Text     string
    ToolCall *ToolCallEvent
    Usage    *Usage
    Err      *StreamError
    Retry    *RetryInfo
    Meta     *Meta   // attached on the terminal "done"
}

The nine event types cover everything a consumer needs to render honestly:

Event	Meaning
chunk / reasoning_chunk	assistant text / thinking text
tool_start / tool_end	a tool call opened / its arguments completed
usage	provider-reported token accounting
retry	transient failure, driver retrying, shown, not hidden
incomplete	the stream was cut or truncated before finishing
done / error	exactly one of these terminates every stream

Every driver, anthropic or openaicompat, must turn its native output into this vocabulary and obey one strict contract:

The channel ends with exactly one terminal event (“done” or “error”) and then closes.

No second terminal. No “done” followed by more chunks. The brain and the web client can rely on it.

Inside the drivers I added real hardening:

• Hard per-request timeout that cancels the wire call.
• Retries on 429/5xx/timeout, at most three, with exponential backoff plus jitter, surfaced as retry events so the caller sees the turbulence.
• If the stream is cut after any content has been sent, the driver emits incomplete or error instead of silently retrying to another provider.
• Only failures before the first byte allow failover down the chain.

Data-Driven Routing and Failover

A request arrives with a task_category (reasoning, coding, mini, summarize, realtime, embedding) and an optional model_hint.

The gateway looks up the enabled chain for that category:

// example chain for "coding"
[
  {"provider_id": "...", "model": "claude-sonnet-5"},
  {"provider_id": "...", "model": "glm-4.7"}
]

It walks the chain in order. For each entry it checks:

• The provider is enabled and healthy (credential resolved and non-empty).
• The model declares the required capability (chat or embeddings).

The first usable attempt wins. If every candidate is skipped, the caller gets a structured NoRouteError that names every rejection reason.

A POST /internal/reload or a 30-second background poll rebuilds the snapshot atomically. The old snapshot keeps serving in-flight requests while the new one takes over.

Every Call Produces a Receipt

I wanted a real cost ledger, not “we’ll add metrics later.”

CREATE TABLE IF NOT EXISTS cost_ledger (
    id                 uuid PRIMARY KEY,
    ts                 timestamptz NOT NULL DEFAULT now(),
    provider           text NOT NULL,
    model              text NOT NULL,
    task_category      text NOT NULL,
    session_id         text,
    ...
    input_tokens       integer,
    output_tokens      integer,
    cache_read_tokens  integer,
    cache_write_tokens integer,
    latency_ms         integer NOT NULL,
    status             text NOT NULL,
    error_code         text,
    cost_usd           numeric(12, 6)
);

The gateway writes one row for every attempt, success or failure. Prices come from the models jsonb on the provider row. If a model has no prices, cost_usd is left null, never guessed. Cost is computed at request time and frozen into the row: when a provider changes prices later, history stays truthful.

Two hard-won details live in this table.

Cached tokens mean different things per provider. OpenAI-style APIs include cached tokens inside prompt_tokens; Anthropic reports cache reads separately from input_tokens. Add the two columns naively and you double-count every cache hit on OpenAI-compatible providers. The drivers normalize to the split representation before a row is ever written.

The zero-latency ledger. A refactor of the streaming path returned a result struct by value while a deferred function stamped the latency onto the dead local copy, the classic Go defer-after-copy footgun. Every streamed request recorded latency_ms = 0. My tests asserted tokens, status, and cost; none asserted latency, so everything stayed green. I only caught it by querying the actual rows and reading 0 where 6-second GLM responses should have been. The fix was a named return value and a test against a deliberately slow fake, and a lesson I keep relearning: verify the running system, not the code.

The ID can be pre-generated before the stream starts so the meta event that closes the SSE can include the ledger row identifier. Clients can show it immediately.

The Internal API Surface

The gateway exposes a narrow HTTP surface inside the compose network:

• POST /v1/stream: the main one. Body carries task_category, optional model_hint, system prompt, messages, tools. Returns SSE of StreamEvent JSON frames.
• POST /v1/embed
• GET /v1/providers: current config plus health
• POST /internal/reload

No auth on these routes. The credential boundary is the gateway itself.

What It Feels Like in Practice

With the stack running and one provider enabled:

# internal-only, from inside the compose network
curl -N -X POST http://gateway:8081/v1/stream \
  -H 'content-type: application/json' \
  -d '{
    "task_category": "mini",
    "messages": [{"role":"user","content":"Explain pgvector in one sentence."}]
  }'

You get a stream of frames:

data: {"type":"chunk","text":"pgvector turns Postgres into a vector database by adding..."}\n\n
data: {"type":"usage","usage":{"input_tokens":18,"output_tokens":42}}\n\n
data: {"type":"done","meta":{"provider":"anthropic","model":"claude-sonnet-5","ledger_id":"..."}}\n\n

A row appears in cost_ledger with the exact token counts the provider reported, the measured latency, and the computed cost based on the prices stored in the provider’s model list.

Disable the first provider in the chain via SQL, hit reload, and the next request automatically tries the fallback (or fails cleanly with the full list of skipped candidates).

Why This Shape

Treating providers as data means adding a new model or even a new cheap provider is a SQL insert plus a price update, no deploy. The normalized stream means the brain never contains “if anthropic” branches. The ledger means I have a receipt for every model call, which will matter the first time I look at the dashboard and think “wait, that much?”

The gateway is deliberately boring infrastructure. That is the point.

In the next post I wire the brain on top of it: bearer authentication for the public API, the pass-through chat that turns one user message into a gateway call, and the terminal meta event that tells the client which provider actually answered and how much it cost.