Knowledge, notes, memory and skills, and a harness to make them used
Four places a model can reach for, differing in who writes a record and how it gets in front of the model. **Knowledge** is uploaded by a person and searched by the model. It goes through `services/files.py:prepare` — the same pipeline as a chat attachment — so the same PDF produces the same text whichever way it arrived, and `Document` carries the same content columns as `Attachment` for the same reason. **Notes** are written by the model and edited by you. Too long to inject, so they are searched. **Memory** is short facts, and every one of them goes into every request. That single decision is where the rest of its design comes from: records are capped short, the block has a budget, there is no search tool because the model is already looking at them, and they are not shareable — a record about a person is not content to hand round. **Skills** are saved procedures. Only the name and description are injected; the body is fetched when the model decides one applies, which is what makes a hundred skills affordable. A model may write and revise its own — the safety story is not a gate but a record: every revision is kept, attributed and revertible. A model that has just read a hostile page can save a skill that outlives the conversation, and the honest mitigation is that it is visible and undoable rather than that it was prevented. **The harness** is why any of it gets used. A model handed a tools array ignores it and answers from recall, because nothing in the request suggests otherwise. `services/harness.py` assembles a preamble from what this chat actually has: when to reach for each tool, the memories, the skill index. This is an exception to "system prompts are precedence, not concatenation", and a deliberate one. That rule governs the three *authored* layers and is untouched — exactly one still wins. The harness is a different axis: it describes the machinery rather than the behaviour, nobody authored it, and there is nothing for it to disagree with. It is prepended to whichever authored prompt won, in one system message, since several endpoints reject a second. Supporting changes: - **Sharing**, in one helper. `visible_to()` is the only definition of who can see a library item and every listing and tool goes through it. Sharing grants *reading*; two people editing one note with no history and no merge is worse than copying it. **Administrators do not bypass this** — they bypass permissions elsewhere because an admin can grant themselves those anyway, but reading somebody's private notes is a different act. - **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an external-content index needs. Idempotent, like the column sync beside it. Terms are ANDed and then ORed: the caller is usually a model writing a whole question, and requiring every word loses the match on one absent term. - **The attach button is a menu** — file, image, a web page, or a document from the library. Attaching a document copies it, because history must not change when a document is edited later. - **A URL fetcher with an SSRF guard.** This server can reach the router, the other services on the box and LLeMbas itself, and the address can come from a model. Private ranges are refused *after resolution* and redirects are followed by hand so every hop is checked. An admin can open it deliberately. - **Model capabilities split** into protocol support and a toggle per built-in tool. Rows predating the split have no `tool_*` keys, and absent counts as on when `tools` is on — otherwise an upgrade silently takes web search away from every model already configured for it. Also fixes the test fixture, which built the schema with `create_all` and so ran against a database without the FTS tables production has; it now runs `sync_schema`, the same path startup takes. 430 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
@@ -0,0 +1,450 @@
|
||||
"""The library: knowledge documents, notes, skills — and memory in settings.
|
||||
|
||||
List-plus-detail throughout, the same shape as the model admin: compact rows
|
||||
with search and pagination, and a full form on its own page. A library is
|
||||
expected to run to hundreds of items, and a page that renders a form per row is
|
||||
unusable at that size.
|
||||
|
||||
Every read goes through ``services.sharing.visible_to`` and every write through
|
||||
``owner_id``. Sharing grants reading only -- two people editing one note with no
|
||||
history and no merge is worse than the inconvenience of copying it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile, status
|
||||
from fastapi.responses import FileResponse, RedirectResponse, Response
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session as DBSession
|
||||
|
||||
from lembas.api.deps import Db, RequiredUser, require_permission
|
||||
from lembas.api.pages import sidebar_context
|
||||
from lembas.db.models import (
|
||||
AUTHOR_USER,
|
||||
PRINCIPAL_GROUP,
|
||||
PRINCIPAL_USER,
|
||||
Document,
|
||||
Group,
|
||||
Note,
|
||||
Skill,
|
||||
SkillRevision,
|
||||
User,
|
||||
)
|
||||
from lembas.security import permissions
|
||||
from lembas.services import files as files_service
|
||||
from lembas.services import settings_store, sharing
|
||||
from lembas.services.fetch import FetchError, fetch
|
||||
from lembas.services.library import documents as documents_service
|
||||
from lembas.services.library import memories as memories_service
|
||||
from lembas.services.library import notes as notes_service
|
||||
from lembas.services.library import skills as skills_service
|
||||
from lembas.services.markdown import render_markdown
|
||||
from lembas.web.templating import render
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(dependencies=[Depends(require_permission("library.use"))], tags=["library"])
|
||||
|
||||
PAGE_SIZE = 30
|
||||
|
||||
|
||||
def _page(db: DBSession, query, page: int):
|
||||
"""One page of a visibility-filtered query, plus what the pager needs."""
|
||||
total = db.scalar(select(func.count()).select_from(query.subquery())) or 0
|
||||
pages = max(1, (total + PAGE_SIZE - 1) // PAGE_SIZE)
|
||||
page = min(max(page, 1), pages)
|
||||
rows = list(db.scalars(query.offset((page - 1) * PAGE_SIZE).limit(PAGE_SIZE)))
|
||||
return rows, {"page": page, "pages": pages, "total": total}
|
||||
|
||||
|
||||
def _shared_context(db: DBSession, user: User, resource) -> dict:
|
||||
"""Everything the share panel on a detail page needs."""
|
||||
grants = sharing.grants_for(db, resource)
|
||||
return {
|
||||
"can_share": permissions.has(db, user, "library.share"),
|
||||
"groups": list(db.scalars(select(Group).order_by(Group.name))),
|
||||
"people": list(
|
||||
db.scalars(select(User).where(User.id != user.id).order_by(User.name))
|
||||
),
|
||||
"shared_users": [g.principal_id for g in grants if g.principal_type == PRINCIPAL_USER],
|
||||
"shared_groups": [g.principal_id for g in grants if g.principal_type == PRINCIPAL_GROUP],
|
||||
"is_owner": resource.owner_id == user.id,
|
||||
}
|
||||
|
||||
|
||||
def _apply_shares(db: DBSession, user: User, resource, form) -> None:
|
||||
if not permissions.has(db, user, "library.share") or resource.owner_id != user.id:
|
||||
return
|
||||
sharing.set_grants(
|
||||
db,
|
||||
resource,
|
||||
user_ids=form.getlist("share_user"),
|
||||
group_ids=form.getlist("share_group"),
|
||||
)
|
||||
|
||||
|
||||
# --- Shell -------------------------------------------------------------------
|
||||
@router.get("/library")
|
||||
async def library_home(user: RequiredUser):
|
||||
return RedirectResponse("/library/knowledge", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
# --- Knowledge ---------------------------------------------------------------
|
||||
@router.get("/library/knowledge")
|
||||
async def knowledge_list(
|
||||
request: Request, db: Db, user: RequiredUser, q: str = "", page: int = 1, saved: str = ""
|
||||
):
|
||||
if q.strip():
|
||||
# Search returns best-match order and its own limit, so it is not paged.
|
||||
rows = documents_service.search(db, user, q, limit=PAGE_SIZE)
|
||||
pager = {"page": 1, "pages": 1, "total": len(rows)}
|
||||
else:
|
||||
rows, pager = _page(
|
||||
db, documents_service.visible(db, user).order_by(Document.created_at.desc()), page
|
||||
)
|
||||
return render(
|
||||
request,
|
||||
"library/knowledge.html",
|
||||
{
|
||||
"section": "knowledge",
|
||||
"documents": rows,
|
||||
"q": q,
|
||||
"pager": pager,
|
||||
"saved": saved,
|
||||
**sidebar_context(db, user),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/library/knowledge/{document_id}")
|
||||
async def knowledge_detail(request: Request, db: Db, user: RequiredUser, document_id: str):
|
||||
document = documents_service.get(db, document_id, user)
|
||||
if document is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That document is not available.")
|
||||
return render(
|
||||
request,
|
||||
"library/knowledge_detail.html",
|
||||
{
|
||||
"section": "knowledge",
|
||||
"document": document,
|
||||
**_shared_context(db, user, document),
|
||||
**sidebar_context(db, user),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/documents")
|
||||
async def upload_document(
|
||||
db: Db, user: RequiredUser, file: UploadFile = File(...), title: str = Form("")
|
||||
) -> Response:
|
||||
payload = await file.read(files_service.MAX_UPLOAD_BYTES + 1)
|
||||
try:
|
||||
document = documents_service.store_upload(
|
||||
db, owner=user, payload=payload, filename=file.filename or "file", title=title
|
||||
)
|
||||
except files_service.FileError as exc:
|
||||
raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
|
||||
return RedirectResponse(
|
||||
f"/library/knowledge/{document.id}", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/documents/link")
|
||||
async def save_link(db: Db, user: RequiredUser, url: str = Form(...)) -> Response:
|
||||
config = settings_store.search(db)
|
||||
try:
|
||||
page = await fetch(url, allow_private=bool(config.get("allow_private_fetch")))
|
||||
except FetchError as exc:
|
||||
raise HTTPException(status.HTTP_400_BAD_REQUEST, exc.message) from exc
|
||||
document = documents_service.store_page(db, owner=user, page=page)
|
||||
return RedirectResponse(
|
||||
f"/library/knowledge/{document.id}", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/documents/{document_id}")
|
||||
async def update_document(
|
||||
request: Request, db: Db, user: RequiredUser, document_id: str
|
||||
) -> Response:
|
||||
document = documents_service.get(db, document_id, user)
|
||||
if document is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That document is not available.")
|
||||
if not sharing.can_write(document, user):
|
||||
raise HTTPException(status.HTTP_403_FORBIDDEN, "That document is not yours to change.")
|
||||
|
||||
form = await request.form()
|
||||
document.title = str(form.get("title", document.title)).strip()[:300] or document.title
|
||||
document.description = str(form.get("description", "")).strip()[:2000]
|
||||
db.commit()
|
||||
_apply_shares(db, user, document, form)
|
||||
return RedirectResponse(
|
||||
f"/library/knowledge/{document.id}", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/documents/{document_id}/delete")
|
||||
async def delete_document(db: Db, user: RequiredUser, document_id: str) -> Response:
|
||||
document = documents_service.get(db, document_id, user)
|
||||
if document is None or not sharing.can_write(document, user):
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That document is not available.")
|
||||
documents_service.delete(db, document)
|
||||
return RedirectResponse("/library/knowledge", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
@router.get("/api/library/documents/{document_id}/content")
|
||||
async def document_content(db: Db, user: RequiredUser, document_id: str) -> Response:
|
||||
"""Serve a document's file.
|
||||
|
||||
Non-images go out as attachments with nosniff, exactly as chat attachments
|
||||
do: an uploaded .html must not be able to execute in this origin.
|
||||
"""
|
||||
document = documents_service.get(db, document_id, user)
|
||||
if document is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That document is not available.")
|
||||
path = documents_service.stored_path(document.stored_name)
|
||||
if path is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That file is no longer on disk.")
|
||||
|
||||
headers = {"X-Content-Type-Options": "nosniff"}
|
||||
if not document.is_image:
|
||||
headers["Content-Disposition"] = f'attachment; filename="{document.filename}"'
|
||||
return FileResponse(path, media_type=document.media_type, headers=headers)
|
||||
|
||||
|
||||
# --- Notes -------------------------------------------------------------------
|
||||
@router.get("/library/notes")
|
||||
async def notes_list(request: Request, db: Db, user: RequiredUser, q: str = "", page: int = 1):
|
||||
if q.strip():
|
||||
rows = notes_service.search(db, user, q, limit=PAGE_SIZE)
|
||||
pager = {"page": 1, "pages": 1, "total": len(rows)}
|
||||
else:
|
||||
rows, pager = _page(
|
||||
db, notes_service.visible(db, user).order_by(Note.updated_at.desc()), page
|
||||
)
|
||||
return render(
|
||||
request,
|
||||
"library/notes.html",
|
||||
{
|
||||
"section": "notes",
|
||||
"notes": rows,
|
||||
"q": q,
|
||||
"pager": pager,
|
||||
**sidebar_context(db, user),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/library/notes/new")
|
||||
async def new_note(request: Request, db: Db, user: RequiredUser):
|
||||
return render(
|
||||
request,
|
||||
"library/note_detail.html",
|
||||
{"section": "notes", "note": None, **sidebar_context(db, user)},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/library/notes/{note_id}")
|
||||
async def note_detail(request: Request, db: Db, user: RequiredUser, note_id: str):
|
||||
note = notes_service.get(db, note_id, user)
|
||||
if note is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That note is not available.")
|
||||
return render(
|
||||
request,
|
||||
"library/note_detail.html",
|
||||
{
|
||||
"section": "notes",
|
||||
"note": note,
|
||||
"body_html": render_markdown(note.body),
|
||||
**_shared_context(db, user, note),
|
||||
**sidebar_context(db, user),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/notes")
|
||||
async def create_note(
|
||||
db: Db, user: RequiredUser, title: str = Form(""), body: str = Form("")
|
||||
) -> Response:
|
||||
note = notes_service.create(db, owner=user, title=title, body=body, author=AUTHOR_USER)
|
||||
return RedirectResponse(f"/library/notes/{note.id}", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
@router.post("/api/library/notes/{note_id}")
|
||||
async def update_note(request: Request, db: Db, user: RequiredUser, note_id: str) -> Response:
|
||||
note = notes_service.get(db, note_id, user)
|
||||
if note is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That note is not available.")
|
||||
if not sharing.can_write(note, user):
|
||||
raise HTTPException(status.HTTP_403_FORBIDDEN, "That note is not yours to change.")
|
||||
|
||||
form = await request.form()
|
||||
notes_service.update(db, note, title=str(form.get("title", "")), body=str(form.get("body", "")))
|
||||
_apply_shares(db, user, note, form)
|
||||
return RedirectResponse(f"/library/notes/{note.id}", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
@router.post("/api/library/notes/{note_id}/delete")
|
||||
async def delete_note(db: Db, user: RequiredUser, note_id: str) -> Response:
|
||||
note = notes_service.get(db, note_id, user)
|
||||
if note is None or not sharing.can_write(note, user):
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That note is not available.")
|
||||
notes_service.delete(db, note)
|
||||
return RedirectResponse("/library/notes", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
# --- Skills ------------------------------------------------------------------
|
||||
@router.get("/library/skills")
|
||||
async def skills_list(request: Request, db: Db, user: RequiredUser, q: str = "", page: int = 1):
|
||||
if q.strip():
|
||||
rows = skills_service.search(db, user, q, limit=PAGE_SIZE)
|
||||
pager = {"page": 1, "pages": 1, "total": len(rows)}
|
||||
else:
|
||||
rows, pager = _page(db, skills_service.visible(db, user).order_by(Skill.name), page)
|
||||
return render(
|
||||
request,
|
||||
"library/skills.html",
|
||||
{
|
||||
"section": "skills",
|
||||
"skills": rows,
|
||||
"q": q,
|
||||
"pager": pager,
|
||||
**sidebar_context(db, user),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/library/skills/new")
|
||||
async def new_skill(request: Request, db: Db, user: RequiredUser):
|
||||
return render(
|
||||
request,
|
||||
"library/skill_detail.html",
|
||||
{"section": "skills", "skill": None, **sidebar_context(db, user)},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/library/skills/{skill_id}")
|
||||
async def skill_detail(request: Request, db: Db, user: RequiredUser, skill_id: str):
|
||||
skill = skills_service.get(db, skill_id, user)
|
||||
if skill is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That skill is not available.")
|
||||
return render(
|
||||
request,
|
||||
"library/skill_detail.html",
|
||||
{
|
||||
"section": "skills",
|
||||
"skill": skill,
|
||||
"revisions": skill.revisions,
|
||||
**_shared_context(db, user, skill),
|
||||
**sidebar_context(db, user),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/skills")
|
||||
async def create_skill(
|
||||
db: Db,
|
||||
user: RequiredUser,
|
||||
name: str = Form(""),
|
||||
description: str = Form(""),
|
||||
body: str = Form(""),
|
||||
) -> Response:
|
||||
try:
|
||||
skill = skills_service.create(
|
||||
db, owner=user, name=name, description=description, body=body, author=AUTHOR_USER
|
||||
)
|
||||
except skills_service.SkillError as exc:
|
||||
raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
|
||||
return RedirectResponse(f"/library/skills/{skill.id}", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
@router.post("/api/library/skills/{skill_id}")
|
||||
async def update_skill(request: Request, db: Db, user: RequiredUser, skill_id: str) -> Response:
|
||||
skill = skills_service.get(db, skill_id, user)
|
||||
if skill is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That skill is not available.")
|
||||
if not sharing.can_write(skill, user):
|
||||
raise HTTPException(status.HTTP_403_FORBIDDEN, "That skill is not yours to change.")
|
||||
|
||||
form = await request.form()
|
||||
skills_service.update(
|
||||
db,
|
||||
skill,
|
||||
description=str(form.get("description", "")),
|
||||
body=str(form.get("body", "")),
|
||||
enabled="enabled" in form,
|
||||
author=AUTHOR_USER,
|
||||
note="edited by hand",
|
||||
)
|
||||
_apply_shares(db, user, skill, form)
|
||||
return RedirectResponse(f"/library/skills/{skill.id}", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
@router.post("/api/library/skills/{skill_id}/revert/{revision_id}")
|
||||
async def revert_skill(
|
||||
db: Db, user: RequiredUser, skill_id: str, revision_id: str
|
||||
) -> Response:
|
||||
skill = skills_service.get(db, skill_id, user)
|
||||
if skill is None or not sharing.can_write(skill, user):
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That skill is not available.")
|
||||
revision = db.get(SkillRevision, revision_id)
|
||||
if revision is None or revision.skill_id != skill.id:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That revision no longer exists.")
|
||||
|
||||
skills_service.revert(db, skill, revision, author=AUTHOR_USER)
|
||||
return RedirectResponse(f"/library/skills/{skill.id}", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
@router.post("/api/library/skills/{skill_id}/delete")
|
||||
async def delete_skill(db: Db, user: RequiredUser, skill_id: str) -> Response:
|
||||
skill = skills_service.get(db, skill_id, user)
|
||||
if skill is None or not sharing.can_write(skill, user):
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That skill is not available.")
|
||||
skills_service.delete(db, skill)
|
||||
return RedirectResponse("/library/skills", status_code=status.HTTP_303_SEE_OTHER)
|
||||
|
||||
|
||||
# --- Memory ------------------------------------------------------------------
|
||||
# Lives in Settings rather than in the library: it is a set of short facts about
|
||||
# the reader, not content they collected.
|
||||
@router.post("/api/library/memories")
|
||||
async def add_memory(db: Db, user: RequiredUser, content: str = Form("")) -> Response:
|
||||
try:
|
||||
memories_service.add(db, owner=user, content=content, author=AUTHOR_USER)
|
||||
except ValueError as exc:
|
||||
from urllib.parse import quote
|
||||
|
||||
return RedirectResponse(
|
||||
f"/settings?error={quote(str(exc))}", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
return RedirectResponse(
|
||||
"/settings?saved=Memory+added.", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/memories/{memory_id}")
|
||||
async def update_memory(
|
||||
db: Db, user: RequiredUser, memory_id: str, content: str = Form("")
|
||||
) -> Response:
|
||||
memory = memories_service.get(db, memory_id, user)
|
||||
if memory is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That memory no longer exists.")
|
||||
try:
|
||||
memories_service.update(db, memory, content)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status.HTTP_400_BAD_REQUEST, str(exc)) from exc
|
||||
return RedirectResponse(
|
||||
"/settings?saved=Memory+updated.", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/library/memories/{memory_id}/delete")
|
||||
async def delete_memory(db: Db, user: RequiredUser, memory_id: str) -> Response:
|
||||
memory = memories_service.get(db, memory_id, user)
|
||||
if memory is None:
|
||||
raise HTTPException(status.HTTP_404_NOT_FOUND, "That memory no longer exists.")
|
||||
memories_service.delete(db, memory)
|
||||
return RedirectResponse(
|
||||
"/settings?saved=Memory+removed.", status_code=status.HTTP_303_SEE_OTHER
|
||||
)
|
||||
Reference in New Issue
Block a user