Knowledge, notes, memory and skills, and a harness to make them used

Four places a model can reach for, differing in who writes a record and how it
gets in front of the model.

**Knowledge** is uploaded by a person and searched by the model. It goes through
`services/files.py:prepare` — the same pipeline as a chat attachment — so the
same PDF produces the same text whichever way it arrived, and `Document` carries
the same content columns as `Attachment` for the same reason.

**Notes** are written by the model and edited by you. Too long to inject, so
they are searched.

**Memory** is short facts, and every one of them goes into every request. That
single decision is where the rest of its design comes from: records are capped
short, the block has a budget, there is no search tool because the model is
already looking at them, and they are not shareable — a record about a person is
not content to hand round.

**Skills** are saved procedures. Only the name and description are injected; the
body is fetched when the model decides one applies, which is what makes a
hundred skills affordable. A model may write and revise its own — the safety
story is not a gate but a record: every revision is kept, attributed and
revertible. A model that has just read a hostile page can save a skill that
outlives the conversation, and the honest mitigation is that it is visible and
undoable rather than that it was prevented.

**The harness** is why any of it gets used. A model handed a tools array
ignores it and answers from recall, because nothing in the request suggests
otherwise. `services/harness.py` assembles a preamble from what this chat
actually has: when to reach for each tool, the memories, the skill index.

This is an exception to "system prompts are precedence, not concatenation", and
a deliberate one. That rule governs the three *authored* layers and is
untouched — exactly one still wins. The harness is a different axis: it
describes the machinery rather than the behaviour, nobody authored it, and there
is nothing for it to disagree with. It is prepended to whichever authored prompt
won, in one system message, since several endpoints reject a second.

Supporting changes:

- **Sharing**, in one helper. `visible_to()` is the only definition of who can
  see a library item and every listing and tool goes through it. Sharing grants
  *reading*; two people editing one note with no history and no merge is worse
  than copying it. **Administrators do not bypass this** — they bypass
  permissions elsewhere because an admin can grant themselves those anyway, but
  reading somebody's private notes is a different act.
- **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an
  external-content index needs. Idempotent, like the column sync beside it.
  Terms are ANDed and then ORed: the caller is usually a model writing a whole
  question, and requiring every word loses the match on one absent term.
- **The attach button is a menu** — file, image, a web page, or a document from
  the library. Attaching a document copies it, because history must not change
  when a document is edited later.
- **A URL fetcher with an SSRF guard.** This server can reach the router, the
  other services on the box and LLeMbas itself, and the address can come from a
  model. Private ranges are refused *after resolution* and redirects are followed
  by hand so every hop is checked. An admin can open it deliberately.
- **Model capabilities split** into protocol support and a toggle per built-in
  tool. Rows predating the split have no `tool_*` keys, and absent counts as on
  when `tools` is on — otherwise an upgrade silently takes web search away from
  every model already configured for it.

Also fixes the test fixture, which built the schema with `create_all` and so ran
against a database without the FTS tables production has; it now runs
`sync_schema`, the same path startup takes.

430 tests, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -19,7 +19,7 @@ lembas info                         # paths + counts, useful when confused
 lembas secret-key                   # generate LEMBAS_SECRET_KEY
 lembas create-admin                 # create or promote an admin
 
-pytest                              # 338 tests, ~16s
+pytest                              # 428 tests, ~26s
                                     # PLAN.md tracks what is and is not built
 ruff check .                        # lint (line length 100)
 python scripts/build_artwork.py     # regenerate artwork (SVG + PWA icons;
@@ -81,6 +81,7 @@ src/lembas/
     admin_audio.py   speech-to-text and text-to-speech endpoints
     admin_search.py  web search provider and credentials
     audio.py         transcribe, speak, voice discovery
+    library.py       knowledge, notes, skills pages; memory CRUD
     files.py         upload, serve, remove attachments
     preferences.py   per-user theme, default model, password, audio
   db/
@@ -92,7 +93,11 @@ src/lembas/
   services/
     llm/openai_client.py   httpx streaming + model discovery
     search/          ddgs, SearXNG and Firecrawl behind one shape
+    library/         documents, notes, memories, skills, FTS
     audio.py         OpenAI-shaped /v1/audio/* client
+    fetch.py         URL retrieval, HTML to text, the SSRF guard
+    sharing.py       one visibility rule for every library store
+    harness.py       the operational prompt built from what a model has
     tools.py         tool registry, schemas, streamed-call reassembly
     chat.py          request building, endpoint resolution, titles
     markdown.py      markdown-it + pygments + nh3
@@ -293,6 +298,57 @@ do without `vision`.
 chunks. `tools.ToolCallAccumulator` rejoins them keyed on `index` — not on
 name, which breaks the moment a model calls one tool twice in a turn.
 
+**Four stores, four different reasons.** `services/library/` — `documents`
+(uploaded by a person, searched by the model), `notes` (written by the model,
+searched), `memories` (short, and *injected whole* every turn), `skills` (index
+injected, body fetched by tool). The shape of each follows from how it reaches
+the model: a memory is capped short because it costs tokens on every request
+forever, a note is not injected because a dozen would fill the window.
+
+**Sharing goes through one helper, and admins do not bypass it.**
+`services/sharing.py:visible_to()` is the only definition of who can see a
+library item, and every listing and tool uses it. `permissions.resolve` gives an
+admin everything, deliberately — but that is about configuration, which an admin
+can grant themselves anyway. Reading someone's private notes is not the same
+act, so `sharing` has no admin branch. Sharing grants **reading only**.
+
+**FTS5 tables are outside the model-driven schema sync.** They are not
+SQLAlchemy models, so `sync_schema()` cannot diff them; `db/migrations.py:
+ensure_fts()` writes them out with `IF NOT EXISTS` and creates the triggers that
+keep an external-content index correct. It runs at every startup and converges,
+like the column sync beside it. `tests/conftest.py` calls `sync_schema` rather
+than `create_all` so tests run against the same schema.
+
+**A failed search rolls back.** One broken FTS statement otherwise leaves the
+session unusable and every later query in the request fails too, which looks
+nothing like a search problem.
+
+**Knowledge attachments are copies.** Attaching a library document to a message
+duplicates its text and its file (`files.copy_document`). Referencing it would
+mean a conversation changing when a document is edited or deleted later — the
+same reason PDF text is extracted once at upload.
+
+**The link fetcher is an SSRF hole unless guarded.** `services/fetch.py` refuses
+loopback, private and link-local addresses **after resolution** — a hostname
+pointing at 127.0.0.1 walks past any check that only reads the URL — and follows
+redirects by hand so every hop is checked. An admin can open it deliberately.
+The URL can come from a model, which can be talked into things by a page it just
+read.
+
+**The harness is an exception to the prompt-precedence rule, on purpose.**
+"System prompts are precedence, not concatenation" governs the three *authored*
+layers, and it stands: exactly one still wins, and `effective_system_prompt`
+still decides which. `services/harness.py` is a different axis — it describes
+the machinery rather than the behaviour, nobody authored it, and there is
+nothing for it to disagree with. It is prepended to whichever authored prompt
+won, in one system message (several endpoints reject a second one), and
+`build_request` is where the two meet.
+
+**A model's tool flags default to on when `tools` is on.** Rows configured
+before the per-tool split have no `tool_*` keys. Reading absent as off would
+silently take web search away from every model already set up for it, so
+`tools.enabled_tools` treats absent as inherited.
+
 **Tool results are not replayed.** Like reasoning, `Message.tool_calls_json` is
 stored and rendered but never fed back as context. The answer already contains
 what the model made of the results; replaying stale results and the schema into
@@ -368,8 +424,9 @@ notes describe the machine.
 
 Custom tools and MCP, agentic execution (local subprocess and SSH connection
 profiles), image generation. Nav entries mark where each one goes. The tool
-loop in `services/generation.py` is what they plug into — a second tool is a
-registry entry, not a new code path.
+loop in `services/generation.py` is what they plug into — a new tool is a
+`ToolDef` in `services/tools.py:REGISTRY` plus a permission and a capability
+flag, not a new code path.
 
 No OCR: a scanned PDF is stored with an explanatory `extraction_error` rather
 than silently contributing nothing.