MCP servers, over streamable HTTP

A server is a row with a URL; its tools are discovered by a button and
cached, then offered beside the built-in ones. Written by hand rather than
taken from the reference SDK, because that SDK's transport does its own
connecting -- and the one thing that must not be bypassed is check_url on
every hop. Owning the transport is the point; the framing beside it is the
small part.

Sessions are per call: initialize, initialized, the call, a best-effort
DELETE. Caching one wants an owner, a TTL, eviction, a lock and a shutdown
hook, and the server may expire it under all of that anyway -- ToolContext
is a session-free snapshot precisely so nothing in a tool holds live state.

A server's names and descriptions reach the model as instructions and are
bounded before they do; what it returns is escaped preformatted text, never
markdown. Tools are namespaced per server, so two servers exposing "search"
do not collide and neither shadows a built-in.

Also: a round's calls now run together under a semaphore, results indexed
so each tool turn stays paired with its call, and generation.status names
what is running -- a remote tool is latency-bound, and a silent pause is
what a hang looks like.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -19,7 +19,7 @@ lembas info                         # paths + counts, useful when confused
 lembas secret-key                   # generate LEMBAS_SECRET_KEY
 lembas create-admin                 # create or promote an admin
 
-pytest                              # 590 tests, ~35s
+pytest                              # 698 tests, ~40s
                                     # PLAN.md tracks what is and is not built
 ruff check .                        # lint (line length 100)
 python scripts/build_artwork.py     # regenerate artwork (SVG + PWA icons;
@@ -82,6 +82,7 @@ src/lembas/
     admin_search.py  web search provider and credentials
     admin_prompts.py the prompt fragment editor and its preview
     admin_suggestions.py  the cards offered on the new-chat screen
+    admin_tools.py   custom HTTP tools and MCP servers
     audio.py         transcribe, speak, voice discovery
     library.py       knowledge, notes, skills pages; memory CRUD
     files.py         upload, serve, remove attachments
@@ -96,6 +97,7 @@ src/lembas/
     llm/openai_client.py   httpx streaming + model discovery
     search/          ddgs, SearXNG and Firecrawl behind one shape
     library/         documents, notes, memories, skills, FTS
+    mcp/             remote MCP servers: framing, transport, rows to tools
     audio.py         OpenAI-shaped /v1/audio/* client
     fetch.py         URL retrieval, HTML to text, the SSRF guard
     sharing.py       one visibility rule for every library store
@@ -106,6 +108,8 @@ src/lembas/
     suggestions.py   new-chat starting points, seeded once
     harness.py       the operational prompt built from what a model has
     tools.py         tool registry, schemas, streamed-call reassembly
+    custom_tools.py  the admin-defined HTTP tool runner
+    tool_access.py   who may be offered which admin-defined tool
     chat.py          request building, endpoint resolution, titles
     markdown.py      markdown-it + pygments + nh3
     crypto.py        Fernet encrypt/decrypt/mask
@@ -317,6 +321,77 @@ survives. Tools are only offered when search is enabled, the user has
 array to an endpoint without support fails the whole request, exactly as images
 do without `vision`.
 
+**The registry is resolved per request, not imported.** `REGISTRY` holds the
+built-ins; a custom tool or an MCP tool is a row. `tools.resolve_tools()` returns
+a `ToolSet` carrying the schemas *and* the runners, and the runners travel to the
+loop on `ToolContext.tools` — because a generation outlives the session that
+could look them up. `run_tool` consults that map, so what may be *run* is what
+was *offered*. `None` means nobody resolved a set and falls back to the built-ins;
+an empty dict is authoritative. Reaching for the global registry instead is how a
+model naming a tool its chat was gated out of used to get it run anyway.
+
+**A row-backed tool's family is `gate:slug`.** `custom:weather`, `mcp:github`.
+The capability flag and the permission are named after the *gate*
+(`tool_custom` / `tools.custom`), so a server advertising forty tools does not
+mean forty checkboxes on every model; the full family exists so each row can
+carry its own prompt fragment, gated to appear exactly when its tool is offered.
+`harness._families` and `admin_prompts` therefore take a `db`. Custom and MCP
+deliberately do **not** require `library.use`: an endpoint an administrator wrote
+has nothing to do with anyone's own notes.
+
+**An argument may fill a hole; it may never move the target.** A custom tool's
+URL is a template. The scheme and host must be literal — checked at save *and*
+again at call time, since a row can predate a check — values are escaped for
+where they land (`quote(safe="")` in a URL, JSON-escaped in a body, control
+characters stripped in a header), and the filled URL's origin is compared with
+the template's afterwards. An undeclared `{{name}}` becomes nothing rather than
+passing through, which is the opposite of `prompts.substitute` and deliberately
+so: a literal `{{x}}` in a URL is not a feature.
+
+**Three places now follow redirects by hand.** `fetch.fetch`,
+`custom_tools._send` and `mcp.client.Session._post`, each re-running
+`check_url` on every hop. `fetch()` itself is not reusable — GET-only,
+bodyless, and it *raises* on any content type that is not HTML or text, which is
+every JSON API there is. The duplication is deliberate; bending a page fetcher
+into a general HTTP client is not. A secret is dropped when a hop leaves the
+origin it was issued for.
+
+**MCP sessions are per call.** Initialize, `notifications/initialized`, the call,
+then a best-effort `DELETE`. Caching one would need an owner, a TTL, eviction, a
+lock (a round runs its tools concurrently) and a shutdown hook, and the server
+can expire it underneath all of that anyway — `ToolContext` is a session-free
+snapshot precisely so nothing in a tool holds live state. The cost is one POST in
+front of a call that is already a network round trip. `307`/`308` are followed;
+`301`/`302`/`303` turn a POST into a GET and are refused rather than guessed at.
+
+**A discovered MCP tool is JSON, not a row.** `McpServer.tools_json` caches
+`tools/list`. `Model` is a table because each row carries eight independent admin
+decisions; a discovered tool carries one (offered or not, in
+`tool_overrides_json`, where absent means on), credentials and guidance are per
+server, and the list is replaced wholesale on every refresh — a table would mean
+reconciling rows against a cache of somebody else's document.
+
+**An MCP tool has two names.** The server's own, which `tools/call` needs, and
+the offered one in the schema — `slug_tool`, lowercased into
+`[a-z0-9_-]{1,64}` because endpoints accept less than MCP does. Built-ins claim
+their names first and can never be shadowed; a custom tool whose slug collides is
+*refused at save*, an MCP tool is renamed silently, since the one that can adapt
+should be the one that has to. The rename never leaves `mcp/registry.py`.
+
+**A server's tool metadata is untrusted input that becomes instructions.**
+Names, descriptions and schemas from `tools/list` are bounded and sanitised in
+`mcp/protocol.clean_tool` before anything reaches a model. What a tool *returns*
+is untrusted too, and is rendered as escaped preformatted text — never through
+`services/markdown.py`, which is the one path allowed to emit HTML.
+
+**A round's tool calls run together.** `generation._run_calls` gathers them under
+a semaphore of four and keeps the results **indexed, not appended as they
+finish**: each tool turn must line up with the assistant turn's `tool_calls` or
+an endpoint matching on `tool_call_id` pairs the right id with the wrong content.
+Safe because `run_tool` never raises and every runner opens its own session.
+`generation.status` names what is running, because a remote tool taking seconds
+with nothing streaming is exactly what a hang looks like.
+
 **Tool-call arguments arrive in fragments.** `delta.tool_calls` carries an
 `index`, a name that appears once, and an `arguments` string split across
 chunks. `tools.ToolCallAccumulator` rejoins them keyed on `index` — not on
@@ -503,14 +578,18 @@ notes describe the machine.
 
 ## Not built yet
 
-Custom tools and MCP, agentic execution (local subprocess and SSH connection
-profiles), image generation. Nav entries mark where each one goes. The tool
-loop in `services/generation.py` is what they plug into — a new tool is a
-`ToolDef` in `services/tools.py:REGISTRY` plus a permission and a capability
-flag, not a new code path. Its guidance is the same shape: a
-`prompts.register_source` yielding one `Fragment` per tool row puts it in the
-harness, on the admin page and in the preview without touching the assembler,
-the save handler or a template.
+Agentic execution (local subprocess and SSH connection profiles), image
+generation. A nav entry marks where each one goes. The tool loop in
+`services/generation.py` is what they plug into — a new tool is a `ToolDef`
+reaching `tools.resolve_tools()` plus a permission and a capability flag, not a
+new code path. Its guidance is the same shape: a `prompts.register_source`
+yielding one `Fragment` per row puts it in the harness, on the admin page and in
+the preview without touching the assembler, the save handler or a template.
+Custom HTTP tools and MCP servers are built and are the worked example of both.
+
+**Local MCP is deliberately absent.** Only remote servers over streamable HTTP.
+Spawning `npx` is the agentic-execution feature, which wants a confirmation model
+before it does anything; a URL is a different act with a different blast radius.
 
 **Unknown is not zero.** `Model.context_length` of 0 means nobody has said how
 big the window is, which is different from "small". The context percentage is