Knowledge, notes, memory and skills, and a harness to make them used
Four places a model can reach for, differing in who writes a record and how it gets in front of the model. **Knowledge** is uploaded by a person and searched by the model. It goes through `services/files.py:prepare` — the same pipeline as a chat attachment — so the same PDF produces the same text whichever way it arrived, and `Document` carries the same content columns as `Attachment` for the same reason. **Notes** are written by the model and edited by you. Too long to inject, so they are searched. **Memory** is short facts, and every one of them goes into every request. That single decision is where the rest of its design comes from: records are capped short, the block has a budget, there is no search tool because the model is already looking at them, and they are not shareable — a record about a person is not content to hand round. **Skills** are saved procedures. Only the name and description are injected; the body is fetched when the model decides one applies, which is what makes a hundred skills affordable. A model may write and revise its own — the safety story is not a gate but a record: every revision is kept, attributed and revertible. A model that has just read a hostile page can save a skill that outlives the conversation, and the honest mitigation is that it is visible and undoable rather than that it was prevented. **The harness** is why any of it gets used. A model handed a tools array ignores it and answers from recall, because nothing in the request suggests otherwise. `services/harness.py` assembles a preamble from what this chat actually has: when to reach for each tool, the memories, the skill index. This is an exception to "system prompts are precedence, not concatenation", and a deliberate one. That rule governs the three *authored* layers and is untouched — exactly one still wins. The harness is a different axis: it describes the machinery rather than the behaviour, nobody authored it, and there is nothing for it to disagree with. It is prepended to whichever authored prompt won, in one system message, since several endpoints reject a second. Supporting changes: - **Sharing**, in one helper. `visible_to()` is the only definition of who can see a library item and every listing and tool goes through it. Sharing grants *reading*; two people editing one note with no history and no merge is worse than copying it. **Administrators do not bypass this** — they bypass permissions elsewhere because an admin can grant themselves those anyway, but reading somebody's private notes is a different act. - **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an external-content index needs. Idempotent, like the column sync beside it. Terms are ANDed and then ORed: the caller is usually a model writing a whole question, and requiring every word loses the match on one absent term. - **The attach button is a menu** — file, image, a web page, or a document from the library. Attaching a document copies it, because history must not change when a document is edited later. - **A URL fetcher with an SSRF guard.** This server can reach the router, the other services on the box and LLeMbas itself, and the address can come from a model. Private ranges are refused *after resolution* and redirects are followed by hand so every hop is checked. An admin can open it deliberately. - **Model capabilities split** into protocol support and a toggle per built-in tool. Rows predating the split have no `tool_*` keys, and absent counts as on when `tools` is on — otherwise an upgrade silently takes web search away from every model already configured for it. Also fixes the test fixture, which built the schema with `create_all` and so ran against a database without the FTS tables production has; it now runs `sync_schema`, the same path startup takes. 430 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
3ad4c82b86
commit
1eba860d39
@@ -0,0 +1,205 @@
|
||||
"""Who may see a document, a note or a skill.
|
||||
|
||||
One rule, in one place, for all three: you can see a resource if you own it, if
|
||||
it was shared with you by name, or if it was shared with a group you are in.
|
||||
|
||||
Everything that lists or searches a library store goes through `visible_to`.
|
||||
Writing the same condition into each query would work right up until one of
|
||||
them was written slightly differently, and the way that failure shows up is
|
||||
somebody reading somebody else's notes.
|
||||
|
||||
**Administrators are not exempt.** They are elsewhere in this codebase --
|
||||
`security.permissions.resolve` hands an admin every permission -- and that is
|
||||
right for configuration, because an admin can grant themselves those two clicks
|
||||
away. This is a different thing. Nobody made these records available to anyone,
|
||||
and being able to reach a database is not the same as being invited.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import ColumnElement, delete, or_, select
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
from lembas.db.models import (
|
||||
PRINCIPAL_GROUP,
|
||||
PRINCIPAL_USER,
|
||||
RESOURCE_DOCUMENT,
|
||||
RESOURCE_NOTE,
|
||||
RESOURCE_SKILL,
|
||||
Document,
|
||||
Note,
|
||||
Share,
|
||||
Skill,
|
||||
User,
|
||||
)
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# The mapping between a model class and the string stored in Share. Kept here
|
||||
# so no caller has to remember which literal goes with which table.
|
||||
RESOURCE_TYPES: dict[Any, str] = {
|
||||
Document: RESOURCE_DOCUMENT,
|
||||
Note: RESOURCE_NOTE,
|
||||
Skill: RESOURCE_SKILL,
|
||||
}
|
||||
|
||||
|
||||
def resource_type(model: Any) -> str:
|
||||
kind = RESOURCE_TYPES.get(model if isinstance(model, type) else type(model))
|
||||
if kind is None:
|
||||
raise ValueError(f"{model!r} is not a shareable resource")
|
||||
return kind
|
||||
|
||||
|
||||
def principal_ids(user: User | None) -> tuple[list[str], list[str]]:
|
||||
"""The ids a share could name to reach this user: themselves, their groups."""
|
||||
if user is None:
|
||||
return [], []
|
||||
return [user.id], [group.id for group in user.groups]
|
||||
|
||||
|
||||
def visible_to(model: Any, user: User | None) -> ColumnElement[bool]:
|
||||
"""A WHERE clause selecting the rows of `model` this user may see.
|
||||
|
||||
Returned as a condition rather than a query so callers can add their own
|
||||
filtering, ordering and pagination without this module knowing about any of
|
||||
it.
|
||||
"""
|
||||
if user is None:
|
||||
# Signed out sees nothing. Not an empty library -- no library.
|
||||
return model.id.is_(None)
|
||||
|
||||
users, groups = principal_ids(user)
|
||||
shared = select(Share.resource_id).where(
|
||||
Share.resource_type == resource_type(model),
|
||||
or_(
|
||||
(Share.principal_type == PRINCIPAL_USER) & Share.principal_id.in_(users),
|
||||
(Share.principal_type == PRINCIPAL_GROUP) & Share.principal_id.in_(groups or [""]),
|
||||
),
|
||||
)
|
||||
return or_(model.owner_id == user.id, model.id.in_(shared))
|
||||
|
||||
|
||||
def owned_by(model: Any, user: User | None) -> ColumnElement[bool]:
|
||||
"""Rows this user may *change*.
|
||||
|
||||
Sharing grants reading, never writing. Two people editing one note with no
|
||||
history and no merge is worse than the inconvenience of copying it.
|
||||
"""
|
||||
if user is None:
|
||||
return model.id.is_(None)
|
||||
return model.owner_id == user.id
|
||||
|
||||
|
||||
def can_read(db: DBSession, resource: Any, user: User | None) -> bool:
|
||||
if user is None or resource is None:
|
||||
return False
|
||||
if resource.owner_id == user.id:
|
||||
return True
|
||||
users, groups = principal_ids(user)
|
||||
found = db.scalar(
|
||||
select(Share.id).where(
|
||||
Share.resource_type == resource_type(resource),
|
||||
Share.resource_id == resource.id,
|
||||
or_(
|
||||
(Share.principal_type == PRINCIPAL_USER) & Share.principal_id.in_(users),
|
||||
(Share.principal_type == PRINCIPAL_GROUP)
|
||||
& Share.principal_id.in_(groups or [""]),
|
||||
),
|
||||
)
|
||||
)
|
||||
return found is not None
|
||||
|
||||
|
||||
def can_write(resource: Any, user: User | None) -> bool:
|
||||
return user is not None and resource is not None and resource.owner_id == user.id
|
||||
|
||||
|
||||
# --- Managing grants ---------------------------------------------------------
|
||||
def grants_for(db: DBSession, resource: Any) -> list[Share]:
|
||||
return list(
|
||||
db.scalars(
|
||||
select(Share).where(
|
||||
Share.resource_type == resource_type(resource),
|
||||
Share.resource_id == resource.id,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def set_grants(
|
||||
db: DBSession,
|
||||
resource: Any,
|
||||
*,
|
||||
user_ids: list[str],
|
||||
group_ids: list[str],
|
||||
) -> None:
|
||||
"""Replace a resource's shares with exactly these principals."""
|
||||
kind = resource_type(resource)
|
||||
db.execute(
|
||||
delete(Share).where(Share.resource_type == kind, Share.resource_id == resource.id)
|
||||
)
|
||||
|
||||
wanted = [(PRINCIPAL_USER, i) for i in dict.fromkeys(user_ids) if i] + [
|
||||
(PRINCIPAL_GROUP, i) for i in dict.fromkeys(group_ids) if i
|
||||
]
|
||||
for principal_type, principal_id in wanted:
|
||||
# Sharing with yourself is not wrong, just meaningless -- you own it.
|
||||
if principal_type == PRINCIPAL_USER and principal_id == resource.owner_id:
|
||||
continue
|
||||
db.add(
|
||||
Share(
|
||||
resource_type=kind,
|
||||
resource_id=resource.id,
|
||||
principal_type=principal_type,
|
||||
principal_id=principal_id,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def forget_resource(db: DBSession, resource: Any) -> None:
|
||||
"""Drop every share of a resource that is being deleted.
|
||||
|
||||
Shares carry no foreign key to their resource -- one column pointing at
|
||||
three tables cannot have one -- so nothing cascades and this has to be
|
||||
called explicitly.
|
||||
"""
|
||||
db.execute(
|
||||
delete(Share).where(
|
||||
Share.resource_type == resource_type(resource),
|
||||
Share.resource_id == resource.id,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def forget_principal(db: DBSession, principal_type: str, principal_id: str) -> int:
|
||||
"""Drop every share naming a user or group that has been deleted.
|
||||
|
||||
Same reason as above: no foreign key, so nothing cascades. Called when an
|
||||
account or a group goes; a stale row would otherwise grant access to
|
||||
whoever next received that id, which is not a risk worth carrying for the
|
||||
sake of a tidy delete.
|
||||
"""
|
||||
result = db.execute(
|
||||
delete(Share).where(
|
||||
Share.principal_type == principal_type, Share.principal_id == principal_id
|
||||
)
|
||||
)
|
||||
return result.rowcount or 0
|
||||
|
||||
|
||||
__all__ = [
|
||||
"can_read",
|
||||
"can_write",
|
||||
"forget_principal",
|
||||
"forget_resource",
|
||||
"grants_for",
|
||||
"owned_by",
|
||||
"resource_type",
|
||||
"set_grants",
|
||||
"visible_to",
|
||||
]
|
||||
Reference in New Issue
Block a user