Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Knowledge, notes, memory and skills, and a harness to make them used

Browse Source 
Four places a model can reach for, differing in who writes a record and how it
gets in front of the model.

**Knowledge** is uploaded by a person and searched by the model. It goes through
`services/files.py:prepare` — the same pipeline as a chat attachment — so the
same PDF produces the same text whichever way it arrived, and `Document` carries
the same content columns as `Attachment` for the same reason.

**Notes** are written by the model and edited by you. Too long to inject, so
they are searched.

**Memory** is short facts, and every one of them goes into every request. That
single decision is where the rest of its design comes from: records are capped
short, the block has a budget, there is no search tool because the model is
already looking at them, and they are not shareable — a record about a person is
not content to hand round.

**Skills** are saved procedures. Only the name and description are injected; the
body is fetched when the model decides one applies, which is what makes a
hundred skills affordable. A model may write and revise its own — the safety
story is not a gate but a record: every revision is kept, attributed and
revertible. A model that has just read a hostile page can save a skill that
outlives the conversation, and the honest mitigation is that it is visible and
undoable rather than that it was prevented.

**The harness** is why any of it gets used. A model handed a tools array
ignores it and answers from recall, because nothing in the request suggests
otherwise. `services/harness.py` assembles a preamble from what this chat
actually has: when to reach for each tool, the memories, the skill index.

This is an exception to "system prompts are precedence, not concatenation", and
a deliberate one. That rule governs the three *authored* layers and is
untouched — exactly one still wins. The harness is a different axis: it
describes the machinery rather than the behaviour, nobody authored it, and there
is nothing for it to disagree with. It is prepended to whichever authored prompt
won, in one system message, since several endpoints reject a second.

Supporting changes:

- **Sharing**, in one helper. `visible_to()` is the only definition of who can
  see a library item and every listing and tool goes through it. Sharing grants
  *reading*; two people editing one note with no history and no merge is worse
  than copying it. **Administrators do not bypass this** — they bypass
  permissions elsewhere because an admin can grant themselves those anyway, but
  reading somebody's private notes is a different act.
- **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an
  external-content index needs. Idempotent, like the column sync beside it.
  Terms are ANDed and then ORed: the caller is usually a model writing a whole
  question, and requiring every word loses the match on one absent term.
- **The attach button is a menu** — file, image, a web page, or a document from
  the library. Attaching a document copies it, because history must not change
  when a document is edited later.
- **A URL fetcher with an SSRF guard.** This server can reach the router, the
  other services on the box and LLeMbas itself, and the address can come from a
  model. Private ranges are refused *after resolution* and redirects are followed
  by hand so every hop is checked. An admin can open it deliberately.
- **Model capabilities split** into protocol support and a toggle per built-in
  tool. Rows predating the split have no `tool_*` keys, and absent counts as on
  when `tools` is on — otherwise an upgrade silently takes web search away from
  every model already configured for it.

Also fixes the test fixture, which built the schema with `create_all` and so ran
against a database without the FTS tables production has; it now runs
`sync_schema`, the same path startup takes.

430 tests, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-07-21 19:43:57 +02:00
parent 3ad4c82b86
commit 1eba860d39
49 changed files with 5028 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/library/documents.py
+166
View File
@@ -0,0 +1,166 @@
"""The knowledge library: documents a person has collected.
Ingestion is deliberately **not** written here. A knowledge document and a chat
attachment are the same processing problem -- sniff the bytes, downscale the
image, extract the PDF once -- so both go through
``services.files.prepare``. Keeping one pipeline is what guarantees the same
PDF produces the same text whichever way it arrived, and it is why `Document`
carries the same content columns as `Attachment`.
"""
from __future__ import annotations
import logging
import secrets
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.orm import Session as DBSession
from lembas.config import settings
from lembas.db.models import SOURCE_LINK, SOURCE_UPLOAD, Document, User
from lembas.services import files as files_service
from lembas.services import sharing
from lembas.services.fetch import Fetched
from lembas.services.library.fts import search_ids
log = logging.getLogger(__name__)
INDEX = "documents_fts"
# How much of a document's text a search result carries back to the model. A
# whole 100-page extract would swallow the context window; this is enough to
# judge relevance and to answer from, and `knowledge_get` fetches the rest.
SNIPPET_CHARS = 1200
def library_dir() -> Path:
"""Where library files live, beside but separate from chat attachments."""
path = settings.uploads_dir / "library"
path.mkdir(parents=True, exist_ok=True)
return path
def stored_path(stored_name: str) -> Path | None:
"""Resolve a stored name, refusing anything outside the library directory.
The same check as ``services.files.stored_path``, against a different root.
"""
if not stored_name or "/" in stored_name or "\\" in stored_name or stored_name.startswith("."):
return None
base = library_dir().resolve()
path = (base / stored_name).resolve()
try:
path.relative_to(base)
except ValueError:
return None
return path if path.is_file() else None
# --- Creating ----------------------------------------------------------------
def store_upload(
db: DBSession, *, owner: User, payload: bytes, filename: str, title: str = ""
) -> Document:
"""Add an uploaded file to the library. Raises files.FileError if unusable."""
prepared = files_service.prepare(payload, filename)
stored_name = f"{secrets.token_hex(16)}{prepared.extension}"
(library_dir() / stored_name).write_bytes(prepared.payload)
display = files_service.safe_display_name(filename)
document = Document(
owner_id=owner.id,
title=(title.strip() or display)[:300],
source=SOURCE_UPLOAD,
filename=display,
stored_name=stored_name,
media_type=prepared.media_type,
size_bytes=len(prepared.payload),
kind=prepared.kind,
width=prepared.width,
height=prepared.height,
extracted_text=prepared.extracted_text,
pages=prepared.pages,
truncated=prepared.truncated,
extraction_error=prepared.extraction_error,
)
db.add(document)
db.commit()
log.info("library: stored %r (%s) for %s", document.title, document.kind, owner.email)
return document
def store_page(db: DBSession, *, owner: User, page: Fetched) -> Document:
"""Add a fetched web page to the library.
Saved as text rather than as the original HTML: the point of keeping it is
what it said, and the markup would have to be reduced again on every read.
"""
document = Document(
owner_id=owner.id,
title=page.title[:300] or page.url[:300],
source=SOURCE_LINK,
source_url=page.url,
filename="",
media_type="text/plain",
size_bytes=len(page.text.encode("utf-8")),
kind="text",
extracted_text=page.text,
truncated=page.truncated,
)
db.add(document)
db.commit()
log.info("library: saved page %r for %s", document.title, owner.email)
return document
# --- Reading -----------------------------------------------------------------
def visible(db: DBSession, user: User | None):
return select(Document).where(sharing.visible_to(Document, user))
def get(db: DBSession, document_id: str, user: User | None) -> Document | None:
document = db.get(Document, document_id)
if document is None or not sharing.can_read(db, document, user):
return None
return document
def search(
db: DBSession, user: User | None, needle: str, *, limit: int = 10
) -> list[Document]:
"""Documents matching `needle` that this user may see, best match first.
The index is searched first and the visibility filter applied to the rows
it returned. That order matters: filtering afterwards is what makes it
impossible for a hit on somebody else's document to leak, even as a count.
"""
hits = search_ids(db, INDEX, needle, limit=limit * 4)
if not hits:
return []
order = {hit.id: position for position, hit in enumerate(hits)}
rows = list(
db.scalars(visible(db, user).where(Document.id.in_(list(order))))
)
rows.sort(key=lambda document: order.get(document.id, len(order)))
return rows[:limit]
def snippet(document: Document) -> str:
"""The part of a document a search result carries."""
text = (document.extracted_text or "").strip()
if len(text) <= SNIPPET_CHARS:
return text
return text[:SNIPPET_CHARS].rstrip() + "…"
# --- Removing ----------------------------------------------------------------
def delete(db: DBSession, document: Document) -> None:
path = stored_path(document.stored_name)
if path is not None:
path.unlink(missing_ok=True)
# Shares carry no foreign key to their resource, so nothing cascades.
sharing.forget_resource(db, document)
db.delete(document)
db.commit()