Knowledge, notes, memory and skills, and a harness to make them used
Four places a model can reach for, differing in who writes a record and how it gets in front of the model. **Knowledge** is uploaded by a person and searched by the model. It goes through `services/files.py:prepare` — the same pipeline as a chat attachment — so the same PDF produces the same text whichever way it arrived, and `Document` carries the same content columns as `Attachment` for the same reason. **Notes** are written by the model and edited by you. Too long to inject, so they are searched. **Memory** is short facts, and every one of them goes into every request. That single decision is where the rest of its design comes from: records are capped short, the block has a budget, there is no search tool because the model is already looking at them, and they are not shareable — a record about a person is not content to hand round. **Skills** are saved procedures. Only the name and description are injected; the body is fetched when the model decides one applies, which is what makes a hundred skills affordable. A model may write and revise its own — the safety story is not a gate but a record: every revision is kept, attributed and revertible. A model that has just read a hostile page can save a skill that outlives the conversation, and the honest mitigation is that it is visible and undoable rather than that it was prevented. **The harness** is why any of it gets used. A model handed a tools array ignores it and answers from recall, because nothing in the request suggests otherwise. `services/harness.py` assembles a preamble from what this chat actually has: when to reach for each tool, the memories, the skill index. This is an exception to "system prompts are precedence, not concatenation", and a deliberate one. That rule governs the three *authored* layers and is untouched — exactly one still wins. The harness is a different axis: it describes the machinery rather than the behaviour, nobody authored it, and there is nothing for it to disagree with. It is prepended to whichever authored prompt won, in one system message, since several endpoints reject a second. Supporting changes: - **Sharing**, in one helper. `visible_to()` is the only definition of who can see a library item and every listing and tool goes through it. Sharing grants *reading*; two people editing one note with no history and no merge is worse than copying it. **Administrators do not bypass this** — they bypass permissions elsewhere because an admin can grant themselves those anyway, but reading somebody's private notes is a different act. - **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an external-content index needs. Idempotent, like the column sync beside it. Terms are ANDed and then ORed: the caller is usually a model writing a whole question, and requiring every word loses the match on one absent term. - **The attach button is a menu** — file, image, a web page, or a document from the library. Attaching a document copies it, because history must not change when a document is edited later. - **A URL fetcher with an SSRF guard.** This server can reach the router, the other services on the box and LLeMbas itself, and the address can come from a model. Private ranges are refused *after resolution* and redirects are followed by hand so every hop is checked. An admin can open it deliberately. - **Model capabilities split** into protocol support and a toggle per built-in tool. Rows predating the split have no `tool_*` keys, and absent counts as on when `tools` is on — otherwise an upgrade silently takes web search away from every model already configured for it. Also fixes the test fixture, which built the schema with `create_all` and so ran against a database without the FTS tables production has; it now runs `sync_schema`, the same path startup takes. 430 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
@@ -96,6 +96,93 @@ def _add_column_sql(table: Table, column: Column, dialect) -> str | None:
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
# --- Full-text search --------------------------------------------------------
|
||||
# The library stores are searched rather than listed, and LIKE over a few
|
||||
# hundred documents ranks nothing and matches badly. SQLite ships FTS5, so the
|
||||
# index costs no dependency and works offline like everything else here.
|
||||
#
|
||||
# These are the one part of the schema this module's model-diffing cannot
|
||||
# derive: an FTS5 virtual table is not a SQLAlchemy model, has no columns to
|
||||
# compare, and needs triggers to stay in step with the table it shadows. So it
|
||||
# is written out -- but written out *idempotently*, with IF NOT EXISTS
|
||||
# throughout, which keeps it the same kind of thing as the column sync: run it
|
||||
# at every startup and it converges.
|
||||
#
|
||||
# `content=` makes each index external-content: the text is not stored twice,
|
||||
# and the triggers below are what the FTS5 documentation calls for to keep an
|
||||
# external-content index correct through updates and deletes.
|
||||
FTS_INDEXES: tuple[tuple[str, str, tuple[str, ...]], ...] = (
|
||||
("documents_fts", "documents", ("title", "description", "extracted_text")),
|
||||
("notes_fts", "notes", ("title", "body")),
|
||||
("skills_fts", "skills", ("name", "description", "body")),
|
||||
)
|
||||
|
||||
|
||||
def _fts_statements(index: str, table: str, columns: tuple[str, ...]) -> list[str]:
|
||||
# `id` rides along UNINDEXED so a match can be turned straight back into an
|
||||
# ORM row. The alternative is joining on rowid, which SQLAlchemy models do
|
||||
# not expose and which changes under VACUUM.
|
||||
columns = ("id", *columns)
|
||||
column_list = ", ".join(columns)
|
||||
declared = ", ".join(
|
||||
f"{name} UNINDEXED" if name == "id" else name for name in columns
|
||||
)
|
||||
new_values = ", ".join(f"new.{name}" for name in columns)
|
||||
old_values = ", ".join(f"old.{name}" for name in columns)
|
||||
|
||||
return [
|
||||
f"CREATE VIRTUAL TABLE IF NOT EXISTS {index} USING fts5("
|
||||
f"{declared}, content='{table}', content_rowid='rowid')",
|
||||
# 'delete' rows carry the old values because an external-content index
|
||||
# cannot look them up itself once the source row has gone.
|
||||
f"""CREATE TRIGGER IF NOT EXISTS {index}_ai AFTER INSERT ON {table} BEGIN
|
||||
INSERT INTO {index}(rowid, {column_list}) VALUES (new.rowid, {new_values});
|
||||
END""",
|
||||
f"""CREATE TRIGGER IF NOT EXISTS {index}_ad AFTER DELETE ON {table} BEGIN
|
||||
INSERT INTO {index}({index}, rowid, {column_list})
|
||||
VALUES ('delete', old.rowid, {old_values});
|
||||
END""",
|
||||
f"""CREATE TRIGGER IF NOT EXISTS {index}_au AFTER UPDATE ON {table} BEGIN
|
||||
INSERT INTO {index}({index}, rowid, {column_list})
|
||||
VALUES ('delete', old.rowid, {old_values});
|
||||
INSERT INTO {index}(rowid, {column_list}) VALUES (new.rowid, {new_values});
|
||||
END""",
|
||||
]
|
||||
|
||||
|
||||
def ensure_fts(engine: Engine) -> list[str]:
|
||||
"""Create the search indexes and their triggers if they are missing.
|
||||
|
||||
Returns the indexes it created. A failure here is logged and swallowed:
|
||||
search degrading to "finds nothing" is bad, but it is much better than the
|
||||
application refusing to start.
|
||||
"""
|
||||
created: list[str] = []
|
||||
inspector = inspect(engine)
|
||||
known = set(inspector.get_table_names())
|
||||
|
||||
with engine.begin() as connection:
|
||||
for index, table, columns in FTS_INDEXES:
|
||||
if table not in known:
|
||||
continue
|
||||
fresh = index not in known
|
||||
for statement in _fts_statements(index, table, columns):
|
||||
connection.execute(text(statement))
|
||||
if fresh:
|
||||
# Backfill anything already in the table. Only on creation --
|
||||
# the triggers keep it current from then on.
|
||||
column_list = ", ".join(("id", *columns))
|
||||
connection.execute(
|
||||
text(
|
||||
f"INSERT INTO {index}(rowid, {column_list}) "
|
||||
f"SELECT rowid, {column_list} FROM {table}"
|
||||
)
|
||||
)
|
||||
created.append(index)
|
||||
|
||||
return created
|
||||
|
||||
|
||||
def sync_schema(engine: Engine) -> list[str]:
|
||||
"""Bring the database up to the declared schema. Returns what it changed."""
|
||||
import lembas.db.models # noqa: F401 (registers every table on the metadata)
|
||||
@@ -125,6 +212,12 @@ def sync_schema(engine: Engine) -> list[str]:
|
||||
changes.append(f"add column {table.name}.{column.name}")
|
||||
log.info("schema: %s", statement)
|
||||
|
||||
try:
|
||||
for index in ensure_fts(engine):
|
||||
changes.append(f"create search index {index}")
|
||||
except Exception: # noqa: BLE001 - search is not worth refusing to start over
|
||||
log.exception("could not create the full-text search indexes")
|
||||
|
||||
if changes:
|
||||
log.info("schema synchronised: %d change(s)", len(changes))
|
||||
for step in MANUAL_STEPS:
|
||||
|
||||
@@ -21,6 +21,23 @@ from lembas.db.models.chat import (
|
||||
Message,
|
||||
)
|
||||
from lembas.db.models.connection import Connection, Model, model_groups
|
||||
from lembas.db.models.library import (
|
||||
AUTHOR_MODEL,
|
||||
AUTHOR_USER,
|
||||
PRINCIPAL_GROUP,
|
||||
PRINCIPAL_USER,
|
||||
RESOURCE_DOCUMENT,
|
||||
RESOURCE_NOTE,
|
||||
RESOURCE_SKILL,
|
||||
SOURCE_LINK,
|
||||
SOURCE_UPLOAD,
|
||||
Document,
|
||||
Memory,
|
||||
Note,
|
||||
Share,
|
||||
Skill,
|
||||
SkillRevision,
|
||||
)
|
||||
from lembas.db.models.setting import Setting
|
||||
from lembas.db.models.user import (
|
||||
ROLE_ADMIN,
|
||||
@@ -32,25 +49,40 @@ from lembas.db.models.user import (
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"AUTHOR_MODEL",
|
||||
"AUTHOR_USER",
|
||||
"Attachment",
|
||||
"KIND_DOCUMENT",
|
||||
"KIND_IMAGE",
|
||||
"KIND_TEXT",
|
||||
"PRINCIPAL_GROUP",
|
||||
"PRINCIPAL_USER",
|
||||
"RESOURCE_DOCUMENT",
|
||||
"RESOURCE_NOTE",
|
||||
"RESOURCE_SKILL",
|
||||
"ROLE_ADMIN",
|
||||
"ROLE_ASSISTANT",
|
||||
"ROLE_PENDING",
|
||||
"ROLE_SYSTEM",
|
||||
"ROLE_TOOL",
|
||||
"ROLE_USER",
|
||||
"SOURCE_LINK",
|
||||
"SOURCE_UPLOAD",
|
||||
"Chat",
|
||||
"Connection",
|
||||
"Document",
|
||||
"Folder",
|
||||
"Group",
|
||||
"Memory",
|
||||
"Message",
|
||||
"Model",
|
||||
"model_groups",
|
||||
"Note",
|
||||
"Session",
|
||||
"Setting",
|
||||
"Share",
|
||||
"Skill",
|
||||
"SkillRevision",
|
||||
"User",
|
||||
"model_groups",
|
||||
"user_groups",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
"""What the model can reach for: knowledge, notes, memory and skills.
|
||||
|
||||
Four stores rather than one, because they differ in the two ways that matter --
|
||||
who writes a record, and how a record reaches the model:
|
||||
|
||||
* **Document** is uploaded by a person and searched by the model. It is the
|
||||
only one holding a file, and it is deliberately shaped like ``Attachment``:
|
||||
both come out of ``services.files.prepare`` and carry the same processed
|
||||
content.
|
||||
* **Note** is written by the model and edited by a person. Long enough that it
|
||||
has to be searched rather than injected.
|
||||
* **Memory** is one short fact, and *is* injected -- every one of them, every
|
||||
turn, up to a budget. Anything that would not survive that treatment belongs
|
||||
in a note.
|
||||
* **Skill** is a named instruction document. Its description is injected so the
|
||||
model knows the skill exists; the body is fetched only when it decides to use
|
||||
it, which is what keeps a hundred skills affordable.
|
||||
|
||||
Everything except Memory can be shared -- see ``Share`` below and
|
||||
``services.sharing``. Memory cannot: a record about a person is not content to
|
||||
hand round, and "share my memories with the team" is a question nobody asked.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import Boolean, ForeignKey, Index, Integer, String, Text, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from lembas.db.base import Base, Timestamps, UUIDPrimaryKey
|
||||
|
||||
# Who wrote a record. Not decoration: a skill the model wrote itself is the one
|
||||
# worth looking at twice when its behaviour changes unexpectedly.
|
||||
AUTHOR_USER = "user"
|
||||
AUTHOR_MODEL = "model"
|
||||
|
||||
# Where a document came from.
|
||||
SOURCE_UPLOAD = "upload"
|
||||
SOURCE_LINK = "link"
|
||||
|
||||
# Resource kinds that can be shared. Values are stored, so they are part of the
|
||||
# schema rather than an implementation detail.
|
||||
RESOURCE_DOCUMENT = "document"
|
||||
RESOURCE_NOTE = "note"
|
||||
RESOURCE_SKILL = "skill"
|
||||
|
||||
PRINCIPAL_USER = "user"
|
||||
PRINCIPAL_GROUP = "group"
|
||||
|
||||
|
||||
class Document(UUIDPrimaryKey, Timestamps, Base):
|
||||
"""One item in a knowledge library: a file, an image or a saved web page.
|
||||
|
||||
The content columns mirror ``Attachment`` exactly because both are produced
|
||||
by ``services.files.prepare`` -- images downscaled, PDF text extracted once,
|
||||
type decided by sniffing bytes. Keeping the shapes identical is what lets a
|
||||
document be attached to a message by copying rather than converting.
|
||||
"""
|
||||
|
||||
__tablename__ = "documents"
|
||||
|
||||
owner_id: Mapped[str] = mapped_column(
|
||||
String(32), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
|
||||
title: Mapped[str] = mapped_column(String(300), nullable=False)
|
||||
description: Mapped[str] = mapped_column(Text, default="")
|
||||
|
||||
source: Mapped[str] = mapped_column(String(16), default=SOURCE_UPLOAD, nullable=False)
|
||||
# Set for a saved web page, so it can be re-fetched and cited.
|
||||
source_url: Mapped[str] = mapped_column(Text, default="")
|
||||
|
||||
# --- The same content columns as Attachment ---
|
||||
filename: Mapped[str] = mapped_column(String(300), default="")
|
||||
stored_name: Mapped[str] = mapped_column(String(120), default="")
|
||||
media_type: Mapped[str] = mapped_column(String(100), default="")
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
kind: Mapped[str] = mapped_column(String(16), default="text", nullable=False)
|
||||
width: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
height: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
extracted_text: Mapped[str] = mapped_column(Text, default="")
|
||||
pages: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
truncated: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
|
||||
extraction_error: Mapped[str] = mapped_column(Text, default="")
|
||||
|
||||
@property
|
||||
def is_image(self) -> bool:
|
||||
return self.kind == "image"
|
||||
|
||||
@property
|
||||
def human_size(self) -> str:
|
||||
size = float(self.size_bytes)
|
||||
for unit in ("B", "KB", "MB"):
|
||||
if size < 1024 or unit == "MB":
|
||||
return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}"
|
||||
size /= 1024
|
||||
return f"{size:.1f} MB"
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Document {self.title!r}>"
|
||||
|
||||
|
||||
class Note(UUIDPrimaryKey, Timestamps, Base):
|
||||
"""Something the model wrote down, or a person did.
|
||||
|
||||
Longer and more specific than a memory. Not injected: a handful of notes
|
||||
would fill a context window on their own, so the model searches for the one
|
||||
it needs.
|
||||
"""
|
||||
|
||||
__tablename__ = "notes"
|
||||
|
||||
owner_id: Mapped[str] = mapped_column(
|
||||
String(32), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
title: Mapped[str] = mapped_column(String(300), nullable=False)
|
||||
body: Mapped[str] = mapped_column(Text, default="")
|
||||
author: Mapped[str] = mapped_column(String(16), default=AUTHOR_USER, nullable=False)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Note {self.title!r}>"
|
||||
|
||||
|
||||
class Memory(UUIDPrimaryKey, Timestamps, Base):
|
||||
"""One short fact, in front of the model on every turn.
|
||||
|
||||
Deliberately not shareable and deliberately small. The length cap is
|
||||
enforced in the service rather than by the column, so an over-long write
|
||||
from a tool is trimmed with an explanation instead of failing the turn.
|
||||
"""
|
||||
|
||||
__tablename__ = "memories"
|
||||
|
||||
owner_id: Mapped[str] = mapped_column(
|
||||
String(32), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
content: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
author: Mapped[str] = mapped_column(String(16), default=AUTHOR_MODEL, nullable=False)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Memory {self.content[:40]!r}>"
|
||||
|
||||
|
||||
class Skill(UUIDPrimaryKey, Timestamps, Base):
|
||||
"""A named set of instructions the model can choose to follow.
|
||||
|
||||
`description` is the load-bearing field: it is what gets injected, and it is
|
||||
the only thing the model has to decide whether the skill is relevant. The
|
||||
body is fetched with a tool.
|
||||
"""
|
||||
|
||||
__tablename__ = "skills"
|
||||
__table_args__ = (UniqueConstraint("owner_id", "name"),)
|
||||
|
||||
owner_id: Mapped[str] = mapped_column(
|
||||
String(32), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
# Slug, referenced by the model when it asks for the body.
|
||||
name: Mapped[str] = mapped_column(String(120), nullable=False)
|
||||
description: Mapped[str] = mapped_column(Text, default="")
|
||||
body: Mapped[str] = mapped_column(Text, default="")
|
||||
enabled: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
|
||||
author: Mapped[str] = mapped_column(String(16), default=AUTHOR_USER, nullable=False)
|
||||
|
||||
revisions: Mapped[list[SkillRevision]] = relationship(
|
||||
back_populates="skill",
|
||||
cascade="all, delete-orphan",
|
||||
order_by="SkillRevision.created_at.desc()",
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Skill {self.name}>"
|
||||
|
||||
|
||||
class SkillRevision(UUIDPrimaryKey, Timestamps, Base):
|
||||
"""The state of a skill before a change.
|
||||
|
||||
A model may rewrite its own skills, so every write snapshots what was there
|
||||
first. That is the whole safety story for self-modification: not a gate, but
|
||||
a record and a way back.
|
||||
"""
|
||||
|
||||
__tablename__ = "skill_revisions"
|
||||
|
||||
skill_id: Mapped[str] = mapped_column(
|
||||
String(32), ForeignKey("skills.id", ondelete="CASCADE"), nullable=False, index=True
|
||||
)
|
||||
description: Mapped[str] = mapped_column(Text, default="")
|
||||
body: Mapped[str] = mapped_column(Text, default="")
|
||||
# Who made the change this revision is the "before" of.
|
||||
author: Mapped[str] = mapped_column(String(16), default=AUTHOR_USER, nullable=False)
|
||||
note: Mapped[str] = mapped_column(String(200), default="")
|
||||
|
||||
skill: Mapped[Skill] = relationship(back_populates="revisions")
|
||||
|
||||
|
||||
class Share(UUIDPrimaryKey, Timestamps, Base):
|
||||
"""One grant of access to one resource.
|
||||
|
||||
A single table across documents, notes and skills rather than three
|
||||
association tables, because the rule is identical in all three cases and
|
||||
``services.sharing`` is the only thing that reads it.
|
||||
|
||||
A grant, never a denial -- the same principle as group permissions. Somebody
|
||||
who cannot see a resource simply has no row here.
|
||||
"""
|
||||
|
||||
__tablename__ = "shares"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"resource_type", "resource_id", "principal_type", "principal_id"
|
||||
),
|
||||
)
|
||||
|
||||
resource_type: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
resource_id: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
|
||||
principal_type: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
# No foreign key: this column points at users or groups depending on
|
||||
# principal_type, and SQLite cannot express that. services.sharing deletes
|
||||
# dangling rows when a user or group goes.
|
||||
principal_id: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<Share {self.resource_type}:{self.resource_id} -> {self.principal_type}>"
|
||||
|
||||
|
||||
Index("ix_shares_resource", Share.resource_type, Share.resource_id)
|
||||
Index("ix_shares_principal", Share.principal_type, Share.principal_id)
|
||||
Reference in New Issue
Block a user