Knowledge, notes, memory and skills, and a harness to make them used
Four places a model can reach for, differing in who writes a record and how it gets in front of the model. **Knowledge** is uploaded by a person and searched by the model. It goes through `services/files.py:prepare` — the same pipeline as a chat attachment — so the same PDF produces the same text whichever way it arrived, and `Document` carries the same content columns as `Attachment` for the same reason. **Notes** are written by the model and edited by you. Too long to inject, so they are searched. **Memory** is short facts, and every one of them goes into every request. That single decision is where the rest of its design comes from: records are capped short, the block has a budget, there is no search tool because the model is already looking at them, and they are not shareable — a record about a person is not content to hand round. **Skills** are saved procedures. Only the name and description are injected; the body is fetched when the model decides one applies, which is what makes a hundred skills affordable. A model may write and revise its own — the safety story is not a gate but a record: every revision is kept, attributed and revertible. A model that has just read a hostile page can save a skill that outlives the conversation, and the honest mitigation is that it is visible and undoable rather than that it was prevented. **The harness** is why any of it gets used. A model handed a tools array ignores it and answers from recall, because nothing in the request suggests otherwise. `services/harness.py` assembles a preamble from what this chat actually has: when to reach for each tool, the memories, the skill index. This is an exception to "system prompts are precedence, not concatenation", and a deliberate one. That rule governs the three *authored* layers and is untouched — exactly one still wins. The harness is a different axis: it describes the machinery rather than the behaviour, nobody authored it, and there is nothing for it to disagree with. It is prepended to whichever authored prompt won, in one system message, since several endpoints reject a second. Supporting changes: - **Sharing**, in one helper. `visible_to()` is the only definition of who can see a library item and every listing and tool goes through it. Sharing grants *reading*; two people editing one note with no history and no merge is worse than copying it. **Administrators do not bypass this** — they bypass permissions elsewhere because an admin can grant themselves those anyway, but reading somebody's private notes is a different act. - **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an external-content index needs. Idempotent, like the column sync beside it. Terms are ANDed and then ORed: the caller is usually a model writing a whole question, and requiring every word loses the match on one absent term. - **The attach button is a menu** — file, image, a web page, or a document from the library. Attaching a document copies it, because history must not change when a document is edited later. - **A URL fetcher with an SSRF guard.** This server can reach the router, the other services on the box and LLeMbas itself, and the address can come from a model. Private ranges are refused *after resolution* and redirects are followed by hand so every hop is checked. An admin can open it deliberately. - **Model capabilities split** into protocol support and a toggle per built-in tool. Rows predating the split have no `tool_*` keys, and absent counts as on when `tools` is on — otherwise an upgrade silently takes web search away from every model already configured for it. Also fixes the test fixture, which built the schema with `create_all` and so ran against a database without the FTS tables production has; it now runs `sync_schema`, the same path startup takes. 430 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
@@ -0,0 +1,18 @@
|
||||
"""The four stores the model can reach for.
|
||||
|
||||
Knowledge, notes and skills are searched; memory is small enough to be handed
|
||||
over whole. Everything here answers to one visibility rule -- see
|
||||
``services.sharing`` -- and nothing here queries a table without it.
|
||||
"""
|
||||
|
||||
from lembas.services.library.fts import SearchHit, fts_query, search_ids
|
||||
from lembas.services.library.memories import MAX_MEMORY_CHARS
|
||||
from lembas.services.library.skills import SKILL_NAME_PATTERN
|
||||
|
||||
__all__ = [
|
||||
"MAX_MEMORY_CHARS",
|
||||
"SKILL_NAME_PATTERN",
|
||||
"SearchHit",
|
||||
"fts_query",
|
||||
"search_ids",
|
||||
]
|
||||
@@ -0,0 +1,166 @@
|
||||
"""The knowledge library: documents a person has collected.
|
||||
|
||||
Ingestion is deliberately **not** written here. A knowledge document and a chat
|
||||
attachment are the same processing problem -- sniff the bytes, downscale the
|
||||
image, extract the PDF once -- so both go through
|
||||
``services.files.prepare``. Keeping one pipeline is what guarantees the same
|
||||
PDF produces the same text whichever way it arrived, and it is why `Document`
|
||||
carries the same content columns as `Attachment`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import secrets
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
from lembas.config import settings
|
||||
from lembas.db.models import SOURCE_LINK, SOURCE_UPLOAD, Document, User
|
||||
from lembas.services import files as files_service
|
||||
from lembas.services import sharing
|
||||
from lembas.services.fetch import Fetched
|
||||
from lembas.services.library.fts import search_ids
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
INDEX = "documents_fts"
|
||||
|
||||
# How much of a document's text a search result carries back to the model. A
|
||||
# whole 100-page extract would swallow the context window; this is enough to
|
||||
# judge relevance and to answer from, and `knowledge_get` fetches the rest.
|
||||
SNIPPET_CHARS = 1200
|
||||
|
||||
|
||||
def library_dir() -> Path:
|
||||
"""Where library files live, beside but separate from chat attachments."""
|
||||
path = settings.uploads_dir / "library"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def stored_path(stored_name: str) -> Path | None:
|
||||
"""Resolve a stored name, refusing anything outside the library directory.
|
||||
|
||||
The same check as ``services.files.stored_path``, against a different root.
|
||||
"""
|
||||
if not stored_name or "/" in stored_name or "\\" in stored_name or stored_name.startswith("."):
|
||||
return None
|
||||
base = library_dir().resolve()
|
||||
path = (base / stored_name).resolve()
|
||||
try:
|
||||
path.relative_to(base)
|
||||
except ValueError:
|
||||
return None
|
||||
return path if path.is_file() else None
|
||||
|
||||
|
||||
# --- Creating ----------------------------------------------------------------
|
||||
def store_upload(
|
||||
db: DBSession, *, owner: User, payload: bytes, filename: str, title: str = ""
|
||||
) -> Document:
|
||||
"""Add an uploaded file to the library. Raises files.FileError if unusable."""
|
||||
prepared = files_service.prepare(payload, filename)
|
||||
|
||||
stored_name = f"{secrets.token_hex(16)}{prepared.extension}"
|
||||
(library_dir() / stored_name).write_bytes(prepared.payload)
|
||||
|
||||
display = files_service.safe_display_name(filename)
|
||||
document = Document(
|
||||
owner_id=owner.id,
|
||||
title=(title.strip() or display)[:300],
|
||||
source=SOURCE_UPLOAD,
|
||||
filename=display,
|
||||
stored_name=stored_name,
|
||||
media_type=prepared.media_type,
|
||||
size_bytes=len(prepared.payload),
|
||||
kind=prepared.kind,
|
||||
width=prepared.width,
|
||||
height=prepared.height,
|
||||
extracted_text=prepared.extracted_text,
|
||||
pages=prepared.pages,
|
||||
truncated=prepared.truncated,
|
||||
extraction_error=prepared.extraction_error,
|
||||
)
|
||||
db.add(document)
|
||||
db.commit()
|
||||
log.info("library: stored %r (%s) for %s", document.title, document.kind, owner.email)
|
||||
return document
|
||||
|
||||
|
||||
def store_page(db: DBSession, *, owner: User, page: Fetched) -> Document:
|
||||
"""Add a fetched web page to the library.
|
||||
|
||||
Saved as text rather than as the original HTML: the point of keeping it is
|
||||
what it said, and the markup would have to be reduced again on every read.
|
||||
"""
|
||||
document = Document(
|
||||
owner_id=owner.id,
|
||||
title=page.title[:300] or page.url[:300],
|
||||
source=SOURCE_LINK,
|
||||
source_url=page.url,
|
||||
filename="",
|
||||
media_type="text/plain",
|
||||
size_bytes=len(page.text.encode("utf-8")),
|
||||
kind="text",
|
||||
extracted_text=page.text,
|
||||
truncated=page.truncated,
|
||||
)
|
||||
db.add(document)
|
||||
db.commit()
|
||||
log.info("library: saved page %r for %s", document.title, owner.email)
|
||||
return document
|
||||
|
||||
|
||||
# --- Reading -----------------------------------------------------------------
|
||||
def visible(db: DBSession, user: User | None):
|
||||
return select(Document).where(sharing.visible_to(Document, user))
|
||||
|
||||
|
||||
def get(db: DBSession, document_id: str, user: User | None) -> Document | None:
|
||||
document = db.get(Document, document_id)
|
||||
if document is None or not sharing.can_read(db, document, user):
|
||||
return None
|
||||
return document
|
||||
|
||||
|
||||
def search(
|
||||
db: DBSession, user: User | None, needle: str, *, limit: int = 10
|
||||
) -> list[Document]:
|
||||
"""Documents matching `needle` that this user may see, best match first.
|
||||
|
||||
The index is searched first and the visibility filter applied to the rows
|
||||
it returned. That order matters: filtering afterwards is what makes it
|
||||
impossible for a hit on somebody else's document to leak, even as a count.
|
||||
"""
|
||||
hits = search_ids(db, INDEX, needle, limit=limit * 4)
|
||||
if not hits:
|
||||
return []
|
||||
|
||||
order = {hit.id: position for position, hit in enumerate(hits)}
|
||||
rows = list(
|
||||
db.scalars(visible(db, user).where(Document.id.in_(list(order))))
|
||||
)
|
||||
rows.sort(key=lambda document: order.get(document.id, len(order)))
|
||||
return rows[:limit]
|
||||
|
||||
|
||||
def snippet(document: Document) -> str:
|
||||
"""The part of a document a search result carries."""
|
||||
text = (document.extracted_text or "").strip()
|
||||
if len(text) <= SNIPPET_CHARS:
|
||||
return text
|
||||
return text[:SNIPPET_CHARS].rstrip() + "…"
|
||||
|
||||
|
||||
# --- Removing ----------------------------------------------------------------
|
||||
def delete(db: DBSession, document: Document) -> None:
|
||||
path = stored_path(document.stored_name)
|
||||
if path is not None:
|
||||
path.unlink(missing_ok=True)
|
||||
# Shares carry no foreign key to their resource, so nothing cascades.
|
||||
sharing.forget_resource(db, document)
|
||||
db.delete(document)
|
||||
db.commit()
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Querying the full-text indexes.
|
||||
|
||||
One helper for all three stores. The interesting part is turning what somebody
|
||||
typed into something FTS5 will accept: its MATCH syntax has operators (`AND`,
|
||||
`NEAR`, `*`, `^`, `:`) and a quoting rule, so a bare question mark or an
|
||||
unbalanced quote is a syntax error rather than a search that finds nothing.
|
||||
|
||||
Every token is therefore quoted and the operators are dropped. That costs the
|
||||
ability to type an FTS expression on purpose, which nobody was going to do, and
|
||||
buys a search box that cannot be made to throw.
|
||||
|
||||
Search returns ids and leaves loading to the caller, which is what keeps the
|
||||
visibility filter in one place: `services.sharing.visible_to` is applied to the
|
||||
row query, not here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Anything that is not a word character or an apostrophe is a separator. Keeps
|
||||
# accented letters (\w is Unicode-aware here) and loses the operators.
|
||||
_TOKENS = re.compile(r"[^\W_]+(?:'[^\W_]+)*", re.UNICODE)
|
||||
|
||||
MAX_TERMS = 24
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SearchHit:
|
||||
id: str
|
||||
rank: float
|
||||
|
||||
|
||||
def _terms(needle: str) -> list[str]:
|
||||
tokens = _TOKENS.findall(needle or "")[:MAX_TERMS]
|
||||
# Doubling any embedded quote is the FTS5 escape; tokens cannot contain one
|
||||
# after the regex above, but the rule is written out so it stays true if the
|
||||
# pattern is ever loosened.
|
||||
return ['"' + token.replace('"', '""') + '"' for token in tokens]
|
||||
|
||||
|
||||
def fts_query(needle: str, *, operator: str = "AND") -> str:
|
||||
"""Turn typed text into a safe FTS5 MATCH expression."""
|
||||
terms = _terms(needle)
|
||||
return f" {operator} ".join(terms) if terms else ""
|
||||
|
||||
|
||||
def search_ids(
|
||||
db: DBSession, index: str, needle: str, *, limit: int = 20
|
||||
) -> list[SearchHit]:
|
||||
"""Ids matching `needle`, best first.
|
||||
|
||||
`index` is a table name from db.migrations.FTS_INDEXES and never comes from
|
||||
a request -- it is interpolated because SQLite cannot parameterise an
|
||||
identifier, so it must stay that way.
|
||||
|
||||
Every term is required first, then any of them. AND alone is right for a
|
||||
search box, where more words should narrow the result -- but the caller here
|
||||
is usually a *model*, which writes "who built the west gate of Moria and
|
||||
what is its password" rather than "moria gate". One word absent from the
|
||||
document then loses the match entirely. Falling back to OR keeps precision
|
||||
where it works and recall where it does not, and bm25 sorts the difference
|
||||
out: documents matching more terms rank higher anyway.
|
||||
"""
|
||||
if not fts_query(needle):
|
||||
return []
|
||||
|
||||
def run(query: str) -> list[SearchHit]:
|
||||
try:
|
||||
rows = db.execute(
|
||||
text(
|
||||
f"SELECT id, bm25({index}) AS rank FROM {index} " # noqa: S608 - see above
|
||||
f"WHERE {index} MATCH :q ORDER BY rank LIMIT :limit"
|
||||
),
|
||||
{"q": query, "limit": max(1, min(limit, 100))},
|
||||
).fetchall()
|
||||
except Exception: # noqa: BLE001 - a broken index must not break the page
|
||||
log.exception("full-text search failed on %s", index)
|
||||
# Rolled back because a failed statement leaves the session
|
||||
# unusable: without this, one broken search turns into every later
|
||||
# query in the same request failing too, which looks nothing like a
|
||||
# search problem.
|
||||
db.rollback()
|
||||
return []
|
||||
# bm25 returns a negative number, better matches being more negative.
|
||||
return [SearchHit(id=row[0], rank=float(row[1])) for row in rows]
|
||||
|
||||
return run(fts_query(needle)) or run(fts_query(needle, operator="OR"))
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Memory: short facts, in front of the model on every turn.
|
||||
|
||||
The whole design follows from being injected rather than searched.
|
||||
|
||||
* Each record is **capped short**, because every one of them costs tokens on
|
||||
every request forever. A tool that writes an essay gets it trimmed and is
|
||||
told so, rather than the write failing -- the model can then decide to put
|
||||
the long version in a note.
|
||||
* There is a **budget** for the block as a whole. Past it the oldest are left
|
||||
out rather than the request growing without limit; the user can see the whole
|
||||
list in their settings and prune it.
|
||||
* There is **no search tool**. Searching something the model is already looking
|
||||
at is a round trip for nothing.
|
||||
* They are **not shareable**. A record about a person is not content to hand
|
||||
round, and nobody asked to share their memories with a group.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, Memory, User
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# One fact, not a paragraph. Long enough for "prefers metric units and a 24-hour
|
||||
# clock", short enough that fifty of them are still affordable.
|
||||
MAX_MEMORY_CHARS = 400
|
||||
|
||||
# Ceiling on the injected block. Reached, the oldest records drop out of the
|
||||
# prompt -- they are still listed in settings, so nothing disappears silently.
|
||||
MAX_TOTAL_CHARS = 4000
|
||||
|
||||
# A hard stop on how many can exist, so an enthusiastic model cannot fill a
|
||||
# database with variations on one fact.
|
||||
MAX_RECORDS = 200
|
||||
|
||||
|
||||
def all_for(db: DBSession, user: User | None) -> list[Memory]:
|
||||
if user is None:
|
||||
return []
|
||||
return list(
|
||||
db.scalars(
|
||||
select(Memory).where(Memory.owner_id == user.id).order_by(Memory.created_at)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def get(db: DBSession, memory_id: str, user: User | None) -> Memory | None:
|
||||
memory = db.get(Memory, memory_id)
|
||||
if memory is None or user is None or memory.owner_id != user.id:
|
||||
return None
|
||||
return memory
|
||||
|
||||
|
||||
def add(db: DBSession, *, owner: User, content: str, author: str = AUTHOR_MODEL) -> Memory:
|
||||
"""Record a fact. Raises ValueError when there is no room or nothing to say."""
|
||||
content = " ".join((content or "").split())
|
||||
if not content:
|
||||
raise ValueError("A memory cannot be empty.")
|
||||
|
||||
count = db.scalar(
|
||||
select(func.count()).select_from(Memory).where(Memory.owner_id == owner.id)
|
||||
)
|
||||
if (count or 0) >= MAX_RECORDS:
|
||||
raise ValueError(
|
||||
f"There are already {MAX_RECORDS} memories. Remove one first, or put "
|
||||
f"this in a note instead."
|
||||
)
|
||||
|
||||
memory = Memory(
|
||||
owner_id=owner.id,
|
||||
content=content[:MAX_MEMORY_CHARS],
|
||||
author=author if author in (AUTHOR_USER, AUTHOR_MODEL) else AUTHOR_MODEL,
|
||||
)
|
||||
db.add(memory)
|
||||
db.commit()
|
||||
return memory
|
||||
|
||||
|
||||
def update(db: DBSession, memory: Memory, content: str) -> Memory:
|
||||
content = " ".join((content or "").split())
|
||||
if not content:
|
||||
raise ValueError("A memory cannot be empty.")
|
||||
memory.content = content[:MAX_MEMORY_CHARS]
|
||||
db.commit()
|
||||
return memory
|
||||
|
||||
|
||||
def delete(db: DBSession, memory: Memory) -> None:
|
||||
db.delete(memory)
|
||||
db.commit()
|
||||
|
||||
|
||||
def block(db: DBSession, user: User | None) -> str:
|
||||
"""The memories as they appear in the prompt, within the budget.
|
||||
|
||||
Oldest first, and truncation drops the *newest* -- a fact that has survived
|
||||
a long time is more likely to be a standing preference than something said
|
||||
once this morning.
|
||||
"""
|
||||
records = all_for(db, user)
|
||||
if not records:
|
||||
return ""
|
||||
|
||||
lines: list[str] = []
|
||||
total = 0
|
||||
for memory in records:
|
||||
line = f"- {memory.content}"
|
||||
if total + len(line) > MAX_TOTAL_CHARS:
|
||||
lines.append(f"- (…{len(records) - len(lines)} more, see your settings)")
|
||||
break
|
||||
lines.append(line)
|
||||
total += len(line)
|
||||
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Notes: what the model wrote down, and what a person wrote for it.
|
||||
|
||||
Longer and more specific than a memory, and not injected. A dozen notes would
|
||||
fill a context window on their own, so the model searches for the one it needs
|
||||
-- which is also why a note has a title worth reading: it is what a search
|
||||
result shows.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, Note, User
|
||||
from lembas.services import sharing
|
||||
from lembas.services.library.fts import search_ids
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
INDEX = "notes_fts"
|
||||
|
||||
MAX_TITLE_CHARS = 300
|
||||
MAX_BODY_CHARS = 40_000
|
||||
SNIPPET_CHARS = 800
|
||||
|
||||
|
||||
def visible(db: DBSession, user: User | None):
|
||||
return select(Note).where(sharing.visible_to(Note, user))
|
||||
|
||||
|
||||
def get(db: DBSession, note_id: str, user: User | None) -> Note | None:
|
||||
note = db.get(Note, note_id)
|
||||
if note is None or not sharing.can_read(db, note, user):
|
||||
return None
|
||||
return note
|
||||
|
||||
|
||||
def recent(db: DBSession, user: User | None, *, limit: int = 20) -> list[Note]:
|
||||
return list(
|
||||
db.scalars(visible(db, user).order_by(Note.updated_at.desc()).limit(limit))
|
||||
)
|
||||
|
||||
|
||||
def search(db: DBSession, user: User | None, needle: str, *, limit: int = 10) -> list[Note]:
|
||||
"""Notes matching `needle` that this user may see, best match first."""
|
||||
hits = search_ids(db, INDEX, needle, limit=limit * 4)
|
||||
if not hits:
|
||||
return []
|
||||
order = {hit.id: position for position, hit in enumerate(hits)}
|
||||
rows = list(db.scalars(visible(db, user).where(Note.id.in_(list(order)))))
|
||||
rows.sort(key=lambda note: order.get(note.id, len(order)))
|
||||
return rows[:limit]
|
||||
|
||||
|
||||
def create(
|
||||
db: DBSession, *, owner: User, title: str, body: str, author: str = AUTHOR_USER
|
||||
) -> Note:
|
||||
note = Note(
|
||||
owner_id=owner.id,
|
||||
title=(title.strip() or "Untitled")[:MAX_TITLE_CHARS],
|
||||
body=body.strip()[:MAX_BODY_CHARS],
|
||||
author=author if author in (AUTHOR_USER, AUTHOR_MODEL) else AUTHOR_USER,
|
||||
)
|
||||
db.add(note)
|
||||
db.commit()
|
||||
return note
|
||||
|
||||
|
||||
def update(db: DBSession, note: Note, *, title: str | None = None, body: str | None = None) -> Note:
|
||||
"""Change a note. Absent arguments are left alone, which is what lets a tool
|
||||
edit only the body without having to send the title back."""
|
||||
if title is not None and title.strip():
|
||||
note.title = title.strip()[:MAX_TITLE_CHARS]
|
||||
if body is not None:
|
||||
note.body = body.strip()[:MAX_BODY_CHARS]
|
||||
db.commit()
|
||||
return note
|
||||
|
||||
|
||||
def delete(db: DBSession, note: Note) -> None:
|
||||
sharing.forget_resource(db, note)
|
||||
db.delete(note)
|
||||
db.commit()
|
||||
|
||||
|
||||
def snippet(note: Note) -> str:
|
||||
text = (note.body or "").strip()
|
||||
if len(text) <= SNIPPET_CHARS:
|
||||
return text
|
||||
return text[:SNIPPET_CHARS].rstrip() + "…"
|
||||
@@ -0,0 +1,207 @@
|
||||
"""Skills: named instructions the model can choose to follow.
|
||||
|
||||
Two fields carry the design.
|
||||
|
||||
`description` is what gets injected -- one line per skill, for every skill --
|
||||
and is therefore the only thing the model has to go on when deciding whether a
|
||||
skill is relevant. A description that does not say *when* to use the skill makes
|
||||
it invisible in practice.
|
||||
|
||||
`body` is fetched only when the model decides to use it. That split is what
|
||||
makes a hundred skills affordable: the index costs a line each, the instructions
|
||||
cost nothing until wanted.
|
||||
|
||||
**A model may rewrite its own skills**, which is the point -- it is how it
|
||||
learns a procedure once instead of being told every time. The safety story is
|
||||
not a gate but a record: every write snapshots what was there first, so a change
|
||||
can be read and undone. A skill written after reading a hostile web page is a
|
||||
real risk, and the honest mitigation is that it is visible, attributed and
|
||||
revertible rather than that it was somehow prevented.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, Skill, SkillRevision, User
|
||||
from lembas.services import sharing
|
||||
from lembas.services.library.fts import search_ids
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
INDEX = "skills_fts"
|
||||
|
||||
# A name the model can quote back without getting it wrong.
|
||||
SKILL_NAME_PATTERN = re.compile(r"^[a-z0-9][a-z0-9-]{1,60}$")
|
||||
|
||||
MAX_DESCRIPTION_CHARS = 400
|
||||
MAX_BODY_CHARS = 20_000
|
||||
|
||||
# The index goes into every request, so it has a ceiling like memory does.
|
||||
MAX_INDEX_SKILLS = 60
|
||||
|
||||
|
||||
class SkillError(Exception):
|
||||
"""A rejected skill write, with a message fit for the model or the user."""
|
||||
|
||||
|
||||
def slugify(name: str) -> str:
|
||||
cleaned = re.sub(r"[^a-z0-9]+", "-", (name or "").strip().lower()).strip("-")
|
||||
return cleaned[:60]
|
||||
|
||||
|
||||
def visible(db: DBSession, user: User | None):
|
||||
return select(Skill).where(sharing.visible_to(Skill, user))
|
||||
|
||||
|
||||
def get(db: DBSession, skill_id: str, user: User | None) -> Skill | None:
|
||||
skill = db.get(Skill, skill_id)
|
||||
if skill is None or not sharing.can_read(db, skill, user):
|
||||
return None
|
||||
return skill
|
||||
|
||||
|
||||
def by_name(db: DBSession, name: str, user: User | None) -> Skill | None:
|
||||
"""Look one up the way the model refers to it."""
|
||||
if user is None:
|
||||
return None
|
||||
return db.scalar(visible(db, user).where(Skill.name == slugify(name)))
|
||||
|
||||
|
||||
def enabled_for(db: DBSession, user: User | None) -> list[Skill]:
|
||||
"""Skills that should appear in the index, oldest first for a stable order."""
|
||||
if user is None:
|
||||
return []
|
||||
return list(
|
||||
db.scalars(
|
||||
visible(db, user)
|
||||
.where(Skill.enabled.is_(True))
|
||||
.order_by(Skill.name)
|
||||
.limit(MAX_INDEX_SKILLS)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def search(db: DBSession, user: User | None, needle: str, *, limit: int = 10) -> list[Skill]:
|
||||
hits = search_ids(db, INDEX, needle, limit=limit * 4)
|
||||
if not hits:
|
||||
return []
|
||||
order = {hit.id: position for position, hit in enumerate(hits)}
|
||||
rows = list(db.scalars(visible(db, user).where(Skill.id.in_(list(order)))))
|
||||
rows.sort(key=lambda skill: order.get(skill.id, len(order)))
|
||||
return rows[:limit]
|
||||
|
||||
|
||||
def snapshot(db: DBSession, skill: Skill, *, author: str, note: str = "") -> SkillRevision:
|
||||
"""Record what a skill looked like before it is changed."""
|
||||
revision = SkillRevision(
|
||||
skill_id=skill.id,
|
||||
description=skill.description,
|
||||
body=skill.body,
|
||||
author=author,
|
||||
note=note[:200],
|
||||
)
|
||||
db.add(revision)
|
||||
return revision
|
||||
|
||||
|
||||
def create(
|
||||
db: DBSession,
|
||||
*,
|
||||
owner: User,
|
||||
name: str,
|
||||
description: str,
|
||||
body: str,
|
||||
author: str = AUTHOR_USER,
|
||||
) -> Skill:
|
||||
slug = slugify(name)
|
||||
if not SKILL_NAME_PATTERN.match(slug):
|
||||
raise SkillError(
|
||||
"A skill name must be two or more letters, numbers or hyphens, "
|
||||
"such as 'weekly-report'."
|
||||
)
|
||||
if by_name(db, slug, owner) is not None:
|
||||
raise SkillError(f"A skill called {slug!r} already exists. Edit it instead.")
|
||||
if not description.strip():
|
||||
raise SkillError(
|
||||
"A skill needs a description saying when to use it — it is the only "
|
||||
"thing shown until the skill is opened."
|
||||
)
|
||||
|
||||
skill = Skill(
|
||||
owner_id=owner.id,
|
||||
name=slug,
|
||||
description=description.strip()[:MAX_DESCRIPTION_CHARS],
|
||||
body=body.strip()[:MAX_BODY_CHARS],
|
||||
author=author if author in (AUTHOR_USER, AUTHOR_MODEL) else AUTHOR_USER,
|
||||
)
|
||||
db.add(skill)
|
||||
db.commit()
|
||||
log.info("skill %r created by %s", slug, author)
|
||||
return skill
|
||||
|
||||
|
||||
def update(
|
||||
db: DBSession,
|
||||
skill: Skill,
|
||||
*,
|
||||
description: str | None = None,
|
||||
body: str | None = None,
|
||||
enabled: bool | None = None,
|
||||
author: str = AUTHOR_USER,
|
||||
note: str = "",
|
||||
) -> Skill:
|
||||
"""Change a skill, keeping what it was.
|
||||
|
||||
The snapshot happens before the change and in the same transaction, so
|
||||
there is no window where a skill has been rewritten with no record of what
|
||||
it used to say.
|
||||
"""
|
||||
changing = (description is not None and description.strip() != skill.description) or (
|
||||
body is not None and body.strip() != skill.body
|
||||
)
|
||||
if changing:
|
||||
snapshot(db, skill, author=author, note=note)
|
||||
|
||||
if description is not None and description.strip():
|
||||
skill.description = description.strip()[:MAX_DESCRIPTION_CHARS]
|
||||
if body is not None:
|
||||
skill.body = body.strip()[:MAX_BODY_CHARS]
|
||||
if enabled is not None:
|
||||
skill.enabled = enabled
|
||||
if changing:
|
||||
skill.author = author if author in (AUTHOR_USER, AUTHOR_MODEL) else skill.author
|
||||
|
||||
db.commit()
|
||||
return skill
|
||||
|
||||
|
||||
def revert(db: DBSession, skill: Skill, revision: SkillRevision, *, author: str) -> Skill:
|
||||
"""Put a skill back to an earlier revision.
|
||||
|
||||
The revert is itself a change, so the current state is snapshotted first --
|
||||
going back is undoable too.
|
||||
"""
|
||||
snapshot(db, skill, author=author, note="before revert")
|
||||
skill.description = revision.description
|
||||
skill.body = revision.body
|
||||
db.commit()
|
||||
return skill
|
||||
|
||||
|
||||
def delete(db: DBSession, skill: Skill) -> None:
|
||||
sharing.forget_resource(db, skill)
|
||||
db.delete(skill)
|
||||
db.commit()
|
||||
|
||||
|
||||
def index_block(db: DBSession, user: User | None) -> str:
|
||||
"""The one-line-per-skill listing that goes into the prompt."""
|
||||
skills = enabled_for(db, user)
|
||||
if not skills:
|
||||
return ""
|
||||
return "\n".join(f"- {skill.name}: {skill.description}" for skill in skills)
|
||||
Reference in New Issue
Block a user