Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Knowledge, notes, memory and skills, and a harness to make them used

Browse Source 
Four places a model can reach for, differing in who writes a record and how it
gets in front of the model.

**Knowledge** is uploaded by a person and searched by the model. It goes through
`services/files.py:prepare` — the same pipeline as a chat attachment — so the
same PDF produces the same text whichever way it arrived, and `Document` carries
the same content columns as `Attachment` for the same reason.

**Notes** are written by the model and edited by you. Too long to inject, so
they are searched.

**Memory** is short facts, and every one of them goes into every request. That
single decision is where the rest of its design comes from: records are capped
short, the block has a budget, there is no search tool because the model is
already looking at them, and they are not shareable — a record about a person is
not content to hand round.

**Skills** are saved procedures. Only the name and description are injected; the
body is fetched when the model decides one applies, which is what makes a
hundred skills affordable. A model may write and revise its own — the safety
story is not a gate but a record: every revision is kept, attributed and
revertible. A model that has just read a hostile page can save a skill that
outlives the conversation, and the honest mitigation is that it is visible and
undoable rather than that it was prevented.

**The harness** is why any of it gets used. A model handed a tools array
ignores it and answers from recall, because nothing in the request suggests
otherwise. `services/harness.py` assembles a preamble from what this chat
actually has: when to reach for each tool, the memories, the skill index.

This is an exception to "system prompts are precedence, not concatenation", and
a deliberate one. That rule governs the three *authored* layers and is
untouched — exactly one still wins. The harness is a different axis: it
describes the machinery rather than the behaviour, nobody authored it, and there
is nothing for it to disagree with. It is prepended to whichever authored prompt
won, in one system message, since several endpoints reject a second.

Supporting changes:

- **Sharing**, in one helper. `visible_to()` is the only definition of who can
  see a library item and every listing and tool goes through it. Sharing grants
  *reading*; two people editing one note with no history and no merge is worse
  than copying it. **Administrators do not bypass this** — they bypass
  permissions elsewhere because an admin can grant themselves those anyway, but
  reading somebody's private notes is a different act.
- **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an
  external-content index needs. Idempotent, like the column sync beside it.
  Terms are ANDed and then ORed: the caller is usually a model writing a whole
  question, and requiring every word loses the match on one absent term.
- **The attach button is a menu** — file, image, a web page, or a document from
  the library. Attaching a document copies it, because history must not change
  when a document is edited later.
- **A URL fetcher with an SSRF guard.** This server can reach the router, the
  other services on the box and LLeMbas itself, and the address can come from a
  model. Private ranges are refused *after resolution* and redirects are followed
  by hand so every hop is checked. An admin can open it deliberately.
- **Model capabilities split** into protocol support and a toggle per built-in
  tool. Rows predating the split have no `tool_*` keys, and absent counts as on
  when `tools` is on — otherwise an upgrade silently takes web search away from
  every model already configured for it.

Also fixes the test fixture, which built the schema with `create_all` and so ran
against a database without the FTS tables production has; it now runs
`sync_schema`, the same path startup takes.

430 tests, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-07-21 19:43:57 +02:00
parent a8b7b5fc14
commit 21001f2eb8
51 changed files with 5135 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/fetch.py
+221
View File
@@ -0,0 +1,221 @@
"""Fetching a web page so it can be kept, or read to a model.
Two things this deliberately does not do.
**It does not try to be clever about extraction.** No readability heuristics, no
main-column detection: script and style go, tags are dropped, whitespace is
collapsed. A clever extractor that silently discards the part somebody wanted is
worse than a plain one that keeps everything, and it would be a dependency.
**It does not trust the URL.** This runs on a server that can very likely reach
a router's admin page, a metadata endpoint, and every other service on the same
machine -- LLeMbas itself included. A fetcher that takes a URL from a user, or
worse from a model, is a request-forgery hole unless something stops it, so
addresses are checked after resolution and redirects are followed by hand.
"""
from __future__ import annotations
import ipaddress
import logging
import re
import socket
from dataclasses import dataclass
from urllib.parse import urlparse, urlunparse
import httpx
import nh3
log = logging.getLogger(__name__)
# Pages are kept as text, so the ceiling is about what is worth reading rather
# than what will fit on disk.
MAX_PAGE_BYTES = 5 * 1024 * 1024
MAX_TEXT_CHARS = 120_000
MAX_REDIRECTS = 5
TIMEOUT = 20.0
# Sent because a plain httpx user agent is blocked by a good number of sites,
# and being honest about what this is beats impersonating a browser.
USER_AGENT = "Mozilla/5.0 (compatible; LLeMbas/1.0; +https://github.com/homer/LLeMbas)"
# <head> goes wholesale, which takes script, style and the title with it. The
# title is pulled out of the raw HTML first, so removing it here is what stops
# it appearing again as the opening line of the body.
_DROPPED = re.compile(
r"<(head|script|style|noscript|template|svg)\b[^>]*>.*?</\1>",
re.IGNORECASE | re.DOTALL,
)
_TITLE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
# Tags that end a line of prose. Turning them into newlines before the tags are
# stripped is the difference between readable text and one enormous paragraph.
_BREAKS = re.compile(
r"</(p|div|section|article|li|tr|h[1-6]|blockquote|pre)\s*>|<br\s*/?>",
re.IGNORECASE,
)
class FetchError(Exception):
"""A refused or failed fetch, with a message fit to show a user."""
def __init__(self, message: str) -> None:
super().__init__(message)
self.message = message
@dataclass
class Fetched:
url: str
title: str
text: str
truncated: bool = False
def _is_public(address: str) -> bool:
"""Whether an IP is one this server should be willing to fetch from.
Loopback reaches LLeMbas and every other local service. Private ranges reach
the rest of the network the server sits on. Link-local covers cloud metadata
endpoints, which is where credentials live.
"""
try:
ip = ipaddress.ip_address(address)
except ValueError:
return False
return not (
ip.is_private
or ip.is_loopback
or ip.is_link_local
or ip.is_multicast
or ip.is_reserved
or ip.is_unspecified
)
def check_url(url: str, *, allow_private: bool = False) -> str:
"""Validate a URL and return it normalised. Raises FetchError if refused."""
try:
parsed = urlparse(url.strip())
except ValueError as exc:
raise FetchError("That does not look like a URL.") from exc
if parsed.scheme not in ("http", "https"):
raise FetchError("Only http and https addresses can be fetched.")
if not parsed.hostname:
raise FetchError("That URL has no host.")
if not allow_private:
try:
# Resolved, not parsed: a hostname pointing at 127.0.0.1 is the
# obvious way past a check that only looks at the text of the URL.
resolved = socket.getaddrinfo(parsed.hostname, None)
except socket.gaierror as exc:
raise FetchError(f"Could not resolve {parsed.hostname}.") from exc
addresses = {info[4][0] for info in resolved}
# Every address, not any: a name resolving to one public and one private
# address must not be usable to reach the private one.
if not addresses or not all(_is_public(address) for address in addresses):
raise FetchError(
f"{parsed.hostname} resolves to a private or local address. "
"An administrator can allow this under Admin → Web search if "
"fetching from this network is intended."
)
return urlunparse(parsed)
def html_to_text(html: str) -> tuple[str, str]:
"""Reduce a page to (title, text)."""
title_match = _TITLE.search(html)
title = ""
if title_match:
title = " ".join(nh3.clean(title_match.group(1), tags=set()).split())
body = _DROPPED.sub(" ", html)
body = _BREAKS.sub("\n", body)
# nh3 with no allowed tags leaves the text and escapes nothing structural;
# it is the same sanitiser the rest of the application trusts.
body = nh3.clean(body, tags=set(), attributes={})
import html as html_module
body = html_module.unescape(body)
lines = [" ".join(line.split()) for line in body.splitlines()]
# Collapse runs of blank lines, which a stripped page is mostly made of.
text, blank = [], False
for line in lines:
if line:
text.append(line)
blank = False
elif not blank:
text.append("")
blank = True
return title, "\n".join(text).strip()
async def fetch(url: str, *, allow_private: bool = False) -> Fetched:
"""Retrieve a page and reduce it to text.
Redirects are followed by hand so every hop can be checked. httpx's own
following would validate the first address and then happily land on
localhost.
"""
current = check_url(url, allow_private=allow_private)
try:
async with httpx.AsyncClient(
timeout=TIMEOUT, follow_redirects=False, headers={"User-Agent": USER_AGENT}
) as client:
for _ in range(MAX_REDIRECTS + 1):
response = await client.get(current)
if response.is_redirect:
location = response.headers.get("location", "")
if not location:
raise FetchError("That page redirected to nowhere.")
current = check_url(
str(response.url.join(location)), allow_private=allow_private
)
continue
if response.status_code >= 400:
raise FetchError(
f"{current} returned HTTP {response.status_code}."
)
break
else:
raise FetchError("That page redirected too many times.")
except httpx.RequestError as exc:
raise FetchError(f"Could not reach {current}: {exc}") from exc
payload = response.content[:MAX_PAGE_BYTES]
content_type = response.headers.get("content-type", "")
if "html" in content_type or payload[:512].lstrip()[:1] == b"<":
title, text = html_to_text(payload.decode(response.encoding or "utf-8", "replace"))
elif content_type.startswith("text/") or not content_type:
title, text = "", payload.decode(response.encoding or "utf-8", "replace")
else:
raise FetchError(
f"That address is {content_type or 'not text'}, which cannot be saved "
"as a page. Attach it as a file instead."
)
truncated = len(text) > MAX_TEXT_CHARS
if not text.strip():
raise FetchError(
"Nothing readable was found at that address. It may be a page that "
"builds itself with JavaScript, which this cannot run."
)
return Fetched(
url=current,
title=title or urlparse(current).netloc or current,
text=text[:MAX_TEXT_CHARS],
truncated=truncated,
)
__all__ = ["FetchError", "Fetched", "check_url", "fetch", "html_to_text"]