Timothy Part 11: First-Party Models on AWS, a Bedrock Driver and Capability-Aware Routing

With long-term memory in place, the vector leg was running dark until an embedding provider that produced exactly 1536 dimensions landed. Titan Embeddings v1 matches the schema perfectly, and Nova (via Bedrock) lets me spend AWS credits instead of third-party tokens. Third-party models on Bedrock were deliberately left out: the goal was first-party access.

The driver sits behind the same interface

Nothing outside internal/gateway/provider/ had to change. A new bedrock.go implements the Provider interface:

• Stream for chat (Converse API + tool calling + system prompts)
• Embed
• capability introspection

It produces the exact same event kinds (chunk, reasoning_chunk, tool_start/tool_end, usage, incomplete, done, error) as Anthropic and the OpenAI-compatible driver. Retries, truncation, and latency are normalized the same way.

The router and ledger continue to see rows in the providers table. Adding Bedrock was an INSERT (disabled by default) plus wiring the driver.

Contract parity details

A few sharp edges had to be handled to keep the “one stream contract” promise:

• Max-token truncation on Bedrock must surface as an incomplete event, never a clean done.
• The AWS event stream error must be checked after the loop; a dropped connection otherwise looks like success.
• Hard per-request timeout at the driver level.
• Prompt caching: Nova supports it (checkpoint after the system block; writes are free, reads discounted). Titan does not accept the cache block, so the driver gates the feature by model family.

Nova’s thinking appears inline as <thinking> spans when tools are in use. The driver splits those into proper reasoning_chunk events so reasoning is handled consistently across models.

Credentials without keys, model IDs that lie

No static access keys anywhere. The AWS SDK default chain is used:

• In deployment: IAM role.
• In dev: local SSO profile name stored in the same credential_ref column every other provider uses.

Compose mounts ~/.aws read-only into the gateway container for the dev case.

A gotcha worth a paragraph: the documented amazon.nova-pro-v1:0 model ID is rejected for on-demand use. Nova only works through cross-region inference profiles (us.amazon.nova-pro-v1:0). The seed data ships the correct profile IDs. Bare model IDs would have failed at runtime; the driver now documents the real ones.

Model-level capability routing

Before this change the router only looked at driver capabilities. Pointing a coding or chat route at an embeddings-only model would pass routing and then fail at the wire with a confusing error.

Now every chain entry is judged against the model’s declared capabilities (intersected with what the driver says it can do for that model). Incompatible entries are skipped before any provider call, and the no-route report names the exact skipped model. This is groundwork for future modality routing (image in, video out, etc.).

Seeding strategy: the Bedrock provider is appended as the last fallback of every chat route (inert until manually enabled via SQL) and gets the first entry on the embedding route. Existing order is untouched.

What the live-run fixes caught

After the initial driver landed, a follow-up pass fixed real-world issues:

1. Bedrock was marked unhealthy when its credential_ref didn’t resolve to an env var. The driver re-uses that column for an AWS profile name the SDK resolves itself. The health check now special-cases providers that bring their own resolution.
2. Nova thinking tags were leaking verbatim into answers. A stateful splitter (in thinktag.go, handles tags that straddle chunk boundaries) turns <thinking>…</thinking> spans into reasoning_chunk events for consistent handling.
3. SSO token cache writes. The AWS SDK rotates SSO tokens and must write to ~/.aws/sso/cache. The compose setup mounts the parent .aws read-only but opens only the cache subdirectory as writable (${HOME}/.aws/sso/cache:/...). Credentials/config stay read-only.

The race detector also earned its keep: one provider instance, concurrent streams, a lazily-built AWS client with no synchronization. sync.Once around the client construction closed it.

Why this matters for memory

Titan embeddings end the degraded mode for the memory vector leg. Near-dup detection at ingest, the consolidation merge step, and the primary recall leg now have real vectors. The schema was deliberately pinned at 1536 so this swap would be a pure addition, not a migration.

Everything else (hybrid legs, RRF, budgets, the fence) already worked; the embedding provider simply turns more lights on.

What ships

• Full Bedrock Converse driver (chat + tools + embeddings) behind the existing Provider interface.
• Model-level (not just driver-level) capability routing.
• Correct inference-profile model IDs, credential health that understands SSO profile refs (not just env keys), Nova thinking-tag normalization via stateful splitter, and compose volume for the writable SSO cache.
• Seeded (but disabled) as last-resort chat provider + first entry on the embedding route.
• No change to the public API surface or the cost ledger contract.

The gateway remains a table-driven multiplexer. Bedrock is just another row (and another driver implementation) that the rest of the system never has to know about specially.

Next: surfacing the ledger, a control plane for providers and routes, and the first tool that reaches the live web.

---

All of the above is on main as of the Bedrock provider merge (fc23183) and live-run fixes (fb951c4). The provider tests and the gateway’s normalized event contract are the source of truth.