Jaroslav Beneš
21001f2eb8
Four places a model can reach for, differing in who writes a record and how it gets in front of the model. **Knowledge** is uploaded by a person and searched by the model. It goes through `services/files.py:prepare` — the same pipeline as a chat attachment — so the same PDF produces the same text whichever way it arrived, and `Document` carries the same content columns as `Attachment` for the same reason. **Notes** are written by the model and edited by you. Too long to inject, so they are searched. **Memory** is short facts, and every one of them goes into every request. That single decision is where the rest of its design comes from: records are capped short, the block has a budget, there is no search tool because the model is already looking at them, and they are not shareable — a record about a person is not content to hand round. **Skills** are saved procedures. Only the name and description are injected; the body is fetched when the model decides one applies, which is what makes a hundred skills affordable. A model may write and revise its own — the safety story is not a gate but a record: every revision is kept, attributed and revertible. A model that has just read a hostile page can save a skill that outlives the conversation, and the honest mitigation is that it is visible and undoable rather than that it was prevented. **The harness** is why any of it gets used. A model handed a tools array ignores it and answers from recall, because nothing in the request suggests otherwise. `services/harness.py` assembles a preamble from what this chat actually has: when to reach for each tool, the memories, the skill index. This is an exception to "system prompts are precedence, not concatenation", and a deliberate one. That rule governs the three *authored* layers and is untouched — exactly one still wins. The harness is a different axis: it describes the machinery rather than the behaviour, nobody authored it, and there is nothing for it to disagree with. It is prepended to whichever authored prompt won, in one system message, since several endpoints reject a second. Supporting changes: - **Sharing**, in one helper. `visible_to()` is the only definition of who can see a library item and every listing and tool goes through it. Sharing grants *reading*; two people editing one note with no history and no merge is worse than copying it. **Administrators do not bypass this** — they bypass permissions elsewhere because an admin can grant themselves those anyway, but reading somebody's private notes is a different act. - **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an external-content index needs. Idempotent, like the column sync beside it. Terms are ANDed and then ORed: the caller is usually a model writing a whole question, and requiring every word loses the match on one absent term. - **The attach button is a menu** — file, image, a web page, or a document from the library. Attaching a document copies it, because history must not change when a document is edited later. - **A URL fetcher with an SSRF guard.** This server can reach the router, the other services on the box and LLeMbas itself, and the address can come from a model. Private ranges are refused *after resolution* and redirects are followed by hand so every hop is checked. An admin can open it deliberately. - **Model capabilities split** into protocol support and a toggle per built-in tool. Rows predating the split have no `tool_*` keys, and absent counts as on when `tools` is on — otherwise an upgrade silently takes web search away from every model already configured for it. Also fixes the test fixture, which built the schema with `create_all` and so ran against a database without the FTS tables production has; it now runs `sync_schema`, the same path startup takes. 430 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
96 lines
3.7 KiB
Python
96 lines
3.7 KiB
Python
"""Querying the full-text indexes.
|
|
|
|
One helper for all three stores. The interesting part is turning what somebody
|
|
typed into something FTS5 will accept: its MATCH syntax has operators (`AND`,
|
|
`NEAR`, `*`, `^`, `:`) and a quoting rule, so a bare question mark or an
|
|
unbalanced quote is a syntax error rather than a search that finds nothing.
|
|
|
|
Every token is therefore quoted and the operators are dropped. That costs the
|
|
ability to type an FTS expression on purpose, which nobody was going to do, and
|
|
buys a search box that cannot be made to throw.
|
|
|
|
Search returns ids and leaves loading to the caller, which is what keeps the
|
|
visibility filter in one place: `services.sharing.visible_to` is applied to the
|
|
row query, not here.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import re
|
|
from dataclasses import dataclass
|
|
|
|
from sqlalchemy import text
|
|
from sqlalchemy.orm import Session as DBSession
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
# Anything that is not a word character or an apostrophe is a separator. Keeps
|
|
# accented letters (\w is Unicode-aware here) and loses the operators.
|
|
_TOKENS = re.compile(r"[^\W_]+(?:'[^\W_]+)*", re.UNICODE)
|
|
|
|
MAX_TERMS = 24
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class SearchHit:
|
|
id: str
|
|
rank: float
|
|
|
|
|
|
def _terms(needle: str) -> list[str]:
|
|
tokens = _TOKENS.findall(needle or "")[:MAX_TERMS]
|
|
# Doubling any embedded quote is the FTS5 escape; tokens cannot contain one
|
|
# after the regex above, but the rule is written out so it stays true if the
|
|
# pattern is ever loosened.
|
|
return ['"' + token.replace('"', '""') + '"' for token in tokens]
|
|
|
|
|
|
def fts_query(needle: str, *, operator: str = "AND") -> str:
|
|
"""Turn typed text into a safe FTS5 MATCH expression."""
|
|
terms = _terms(needle)
|
|
return f" {operator} ".join(terms) if terms else ""
|
|
|
|
|
|
def search_ids(
|
|
db: DBSession, index: str, needle: str, *, limit: int = 20
|
|
) -> list[SearchHit]:
|
|
"""Ids matching `needle`, best first.
|
|
|
|
`index` is a table name from db.migrations.FTS_INDEXES and never comes from
|
|
a request -- it is interpolated because SQLite cannot parameterise an
|
|
identifier, so it must stay that way.
|
|
|
|
Every term is required first, then any of them. AND alone is right for a
|
|
search box, where more words should narrow the result -- but the caller here
|
|
is usually a *model*, which writes "who built the west gate of Moria and
|
|
what is its password" rather than "moria gate". One word absent from the
|
|
document then loses the match entirely. Falling back to OR keeps precision
|
|
where it works and recall where it does not, and bm25 sorts the difference
|
|
out: documents matching more terms rank higher anyway.
|
|
"""
|
|
if not fts_query(needle):
|
|
return []
|
|
|
|
def run(query: str) -> list[SearchHit]:
|
|
try:
|
|
rows = db.execute(
|
|
text(
|
|
f"SELECT id, bm25({index}) AS rank FROM {index} " # noqa: S608 - see above
|
|
f"WHERE {index} MATCH :q ORDER BY rank LIMIT :limit"
|
|
),
|
|
{"q": query, "limit": max(1, min(limit, 100))},
|
|
).fetchall()
|
|
except Exception: # noqa: BLE001 - a broken index must not break the page
|
|
log.exception("full-text search failed on %s", index)
|
|
# Rolled back because a failed statement leaves the session
|
|
# unusable: without this, one broken search turns into every later
|
|
# query in the same request failing too, which looks nothing like a
|
|
# search problem.
|
|
db.rollback()
|
|
return []
|
|
# bm25 returns a negative number, better matches being more negative.
|
|
return [SearchHit(id=row[0], rank=float(row[1])) for row in rows]
|
|
|
|
return run(fts_query(needle)) or run(fts_query(needle, operator="OR"))
|