Timothy Part 13: Connectors for Gmail, Calendar, and MCP Tools

Everything the model could act on until now lived inside the box: the shell tool ran in a sandboxed workspace, web_search hit an internal SearXNG instance, web_fetch pulled public pages. None of it touched anything that belonged to me personally. If I wanted the assistant to check my inbox or see what was on my calendar, someone had to build that, and it had to survive the same scrutiny as everything else: no raw tokens in the database, no silent capability creep, no tool the model can call without the user ever having agreed to it.

Connectors as data, one more time

The gateway already treats providers as rows, not code (providers and task_routes). Connectors follow the same pattern: a connectors table, admin CRUD with audit, and a manager that rebuilds live sources when a row changes, no restart needed.

-- migrations/0020_connectors.sql
CREATE TABLE IF NOT EXISTS connectors (
    id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    name           text UNIQUE NOT NULL,
    kind           text NOT NULL CHECK (kind IN ('mcp', 'google')),
    config         jsonb NOT NULL DEFAULT '{}',
    credential_ref text NOT NULL DEFAULT '',
    enabled        boolean NOT NULL DEFAULT false,
    created_at     timestamptz NOT NULL DEFAULT now(),
    updated_at     timestamptz NOT NULL DEFAULT now()
);

config holds kind-specific settings: an MCP endpoint and headers, or a Google client ID and scopes. credential_ref names a secret, never carries one. The OAuth tokens themselves never touch this table or this row; they live in the same encrypted secret store the gateway already uses for provider API keys, addressed by the ref name stored here. Brain resolves that ref through its own secret-store handle, same database and same master key the gateway uses, because the tools execute inside brain’s agent loop and proxying a resolved secret value over the compose network would put it on the wire for anything else listening there.

A Manager owns the built sources:

// internal/brain/connectors/manager.go
type Manager struct {
    store    *Store
    builders map[string]Builder
    mu       sync.RWMutex
    sources  map[string]Source
    ready    chan struct{}
}

Reload lists enabled rows, builds each through its kind’s Builder, and swaps the whole map atomically. A connector whose build fails (bad endpoint, expired token) is skipped with a log line; it does not take the others down, and the previous set stays live until the swap succeeds. Every tool a source exposes gets renamed to <connector-name>_<tool-name> before it reaches the registry, so two connectors can never collide on a tool name and connector tools can never be mistaken for a builtin.

Why MCP as the generic path

I did not want to hand-write a Go client for every third-party API I might eventually want. The Model Context Protocol already standardizes exactly this: a server describes its tools, the client calls them over JSON-RPC. Adding “GitHub” or “Grafana” as a connector becomes a config row pointing at an existing MCP server, not a new Go package.

The client only speaks streamable HTTP, not stdio. Stdio would mean brain spawning arbitrary admin-configured subprocesses inside its own container, a much bigger trust boundary than reaching out to a URL:

// internal/brain/connectors/mcp.go
type mcpConfig struct {
    Endpoint string            `json:"endpoint"`
    Headers  map[string]string `json:"headers"`
}

At build time the client does the MCP handshake (initialize, notifications/initialized, tools/list) and caches the tool list for the source’s lifetime; reloads rebuild from scratch rather than trying to diff a live session. Because a streamable-HTTP server is allowed to answer either with plain JSON or with an SSE stream carrying the one response message, rpc() has to handle both: it reads the Content-Type header and either decodes directly or scans the event stream for the envelope whose id matches the request.

A resolved credential ref becomes a bearer token; an unset one just means the request goes out unauthenticated and the server’s 401 surfaces naturally at initialize instead of the client guessing.

Why Google gets its own kind instead of staying generic

Gmail and Calendar are the two integrations I actually wanted on day one, and neither speaks MCP. Both need OAuth, which is a different shape of problem: a browser redirect dance, a refresh-token lifecycle, and per-scope tool gating that a generic bearer-token client cannot express.

The dance is entirely brain-owned:

• StartAuth builds Google’s consent URL with access_type=offline and prompt=consent. Google only issues a refresh token on a consented offline grant; skip either parameter and the connector works fine until the first access token expires, then dies silently with no way to get a new one short of deleting and re-adding it.
• A per-attempt state token (24 random bytes, base64) is stored in memory with a 10-minute expiry and swept opportunistically. The callback route is unauthenticated by necessity, since Google redirects the user’s own browser to it, not an authenticated API client, so the state token has to be the entire proof that this callback belongs to a dance brain actually started.
• HandleCallback exchanges the code, and if Google comes back without a refresh token the flow fails loudly rather than storing a bundle that will quietly stop working: “remove Timothy’s access at myaccount.google.com/permissions and reconnect.”
• The token bundle (access token, refresh token, expiry) is JSON-marshaled and written to the secret store at the connector’s credential_ref. token() reads it back, and refreshes 2 minutes ahead of expiry, re-storing the bundle. Google’s refresh grant omits the refresh token on the response; the code has to remember to keep the old one instead of overwriting it with an empty string.

TIMOTHY_PUBLIC_URL is what the redirect URI is built from. Without it, Google connectors can be configured in Settings but StartAuth refuses outright with a clear error, since there is no browser-facing address to send Google back to.

The tool surface is scope-gated and explicit about what it does: gmail_search, gmail_read, gmail_read_attachment, gmail_send, calendar_list_events, calendar_create_event. Nothing here is marked exempt from the permission chain the way current_time or web_search are. Sending an email or creating a calendar event is a real-world side effect with a real recipient, and the tool descriptions say so plainly: gmail_send’s description ends with “the recipient sees it immediately”; calendar_create_event’s says “attendees receive invitations immediately.” Reads still have to pass through the loop, but writes ride the same danger classification as a destructive shell command.

Why a Python sidecar instead of parsing in Go

gmail_read initially fell back to Gmail’s truncated snippet whenever a message had no text/plain part, which is most booking confirmations and receipts: they are HTML-only. Writing a good HTML-to-text extractor, and separately a PDF/Office-document reader, in Go was more surface area than I wanted to own, and Microsoft’s markitdown library already does both well. Rather than shell out to Python from inside brain, it runs as its own internal sidecar:

# markitdown-svc/main.py
@app.post("/convert")
async def convert(request: Request, x_filename=Header(None), x_mimetype=Header(None)):
    body = await request.body()
    stream_info = StreamInfo(filename=x_filename, mimetype=x_mimetype)
    result = converter.convert_stream(io.BytesIO(body), stream_info=stream_info)
    return JSONResponse({"markdown": result.markdown})

No auth, no published port: same trust boundary as SearXNG, reachable only from other services on the compose network. gmail_read posts the HTML body and gets structured markdown back instead of a stripped-down snippet. gmail_read_attachment came along with it, and had to solve a smaller, odder problem: real Gmail attachment ids are 300-400+ character opaque tokens, and models reliably mangle them when copying a value that long into a tool call. So the tool takes a human-readable filename instead and re-fetches the message itself to resolve the real attachment id, rather than trusting the model to reproduce a token that long correctly.

The same sidecar later took over web_fetch’s HTML and PDF handling too. The DOM text walk web_fetch used before flattened away headings, tables, and links, and outright rejected PDF responses, which was exactly the structure a research task most needed to keep. HTML degrades to the old extractor if the sidecar call fails; PDFs just error clearly if the sidecar isn’t configured, rather than pretending to read a format the fallback path never supported.

I trimmed the connector preset tiles in Settings down to Gmail, Calendar, and GitHub. Grafana and a raw custom-MCP tile were in the first cut of the UI and came out before anyone but me ever saw them: fewer presets to explain, and a plain custom MCP form still exists for anything not on the list.

Two bugs that only showed up under load

Allowlists silently dropped every connector tool. Agent configs (missions, for instance) can restrict a run to a named allowlist like gmail_search. But connector tools register namespaced, so the tool the loop actually offers is gmail_gmail_search. The allowlist filter did an exact string match, so an agent scoped to gmail_search was silently offered nothing from Gmail at all, no error, just an empty connector surface. The permission-grant layer already had a suffix matcher for exactly this kind of namespacing mismatch (ToolMatches, originally unexported as toolMatches); the fix was exporting it and reusing it in the allowlist filter too, so the two layers can never disagree about what a config-authored name refers to.

Turns started before connectors had loaded. Connector loading is asynchronous at boot, and nothing signaled when it was actually done. For roughly the first 60 seconds after a restart, chat turns ran with a builtins-only tool set, since a failed first reload (common while the database pool is still warming up) only retried on a full 60-second ticker. The fix adds a Manager.Ready() channel that closes after the first successful reload and its tool-rebuild hook, not just the database read; retries the initial load every 5 seconds instead of waiting a full minute; and has turns wait on readiness, bounded at 15 seconds and logging if it times out, before they snapshot their own tool surface. The ordering detail matters: ready closes strictly after the hook that rebuilds the agent’s live tool set, because a waiter that only checked “did Reload finish” could still race the rebuild and get a stale, connector-less snapshot anyway.

What’s still missing

Only Gmail and Calendar exist as first-party kinds; everything else third-party goes through the generic MCP path, which is the intended shape but means no Slack or Notion tool exists yet beyond whatever a public MCP server for them can offer. The OAuth dance only supports Google’s flow, not the generic OAuth2 that other providers would need for a first-party kind of their own. And connector credentials, like provider credentials, still require the master key to be configured at all or the whole connector surface stays unmounted, brain just runs without it, same as before.

Next: agents, not modes: agent profiles, scored routing, and one settings surface.