Jaroslav BenešandClaude Opus 5
20bb569b00
Three pieces, and the first one is that they are all optional. Extraction stops being constants. Upload size, image edge, JPEG quality, PDF pages, extracted characters, orphan age and the text-extension list are settings now, read through a process-level snapshot rather than a session -- `prepare` and everything under it are called from routes, tool runners and the startup sweep, and several of those have no session in hand. Two things deliberately stayed constants: the decompression-bomb guard, which is a guard and not a preference, and ORPHAN_AGE, which would have been evaluated at import if it stayed in the signature and pinned the shipped 24 hours whatever anybody set. An embedding model is picked from the models an administrator flagged for it, and one that has since lost its flag is *named* rather than dropped from the picker: a setting that vanishes is one nobody can tell from a setting never made. Nothing here is required. Choosing none means no chunk rows, no requests, and retrieval.search returning exactly what fts.search_ids returns in exactly that order -- asserted, because it is what makes this safe to land on an instance that never asked for it. The two rankings are fused by reciprocal rank fusion: ranks and not scores, because bm25 is a corpus-dependent negative and cosine is 0..1, and normalising them onto one scale means picking a constant nobody can tune without a labelled set they do not have. RRF's one constant is famously insensitive and degrades to whichever list is non-empty -- which is what turns "no embedding model" into a branch that does not exist. A record scores as its best chunk rather than its average, or a long document about something else outranks a short one that says the thing. Width and model are stored beside every vector and a mismatch is skipped, because vectors from two spaces score against each other perfectly happily and mean nothing -- a search that works and is wrong is the worst failure this can have, and a model change now leaves stale rows ignored rather than trusted. Indexing is fired and forgotten, and how a change is noticed is a session event rather than a call in each of the ten library writers. That is a departure from this codebase's taste for explicit seams, for the reason tool_label is a Jinja global: a step every writer has to remember is one that gets forgotten, and here forgetting is silent -- the record saves, keyword search still finds it, and only its recall goes stale. Chunks are embedded before anything is deleted, so a failure leaves the old index rather than half a new one. Also: `embeddings` joins the model capabilities, and the three tool flags that had shipped with no checkbox -- canvas, scheduling and helpers -- have one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
106 lines
3.2 KiB
Python
106 lines
3.2 KiB
Python
"""Notes: what the model wrote down, and what a person wrote for it.
|
|
|
|
Longer and more specific than a memory, and not injected. A dozen notes would
|
|
fill a context window on their own, so the model searches for the one it needs
|
|
-- which is also why a note has a title worth reading: it is what a search
|
|
result shows.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
from sqlalchemy import select
|
|
from sqlalchemy.orm import Session as DBSession
|
|
|
|
from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, CHUNK_NOTE, Note, User
|
|
from lembas.services import sharing
|
|
from lembas.services.library import retrieval
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
INDEX = "notes_fts"
|
|
|
|
MAX_TITLE_CHARS = 300
|
|
MAX_BODY_CHARS = 40_000
|
|
SNIPPET_CHARS = 800
|
|
|
|
|
|
def visible(db: DBSession, user: User | None):
|
|
return select(Note).where(sharing.visible_to(Note, user))
|
|
|
|
|
|
def get(db: DBSession, note_id: str, user: User | None) -> Note | None:
|
|
note = db.get(Note, note_id)
|
|
if note is None or not sharing.can_read(db, note, user):
|
|
return None
|
|
return note
|
|
|
|
|
|
def recent(db: DBSession, user: User | None, *, limit: int = 20) -> list[Note]:
|
|
return list(
|
|
db.scalars(visible(db, user).order_by(Note.updated_at.desc()).limit(limit))
|
|
)
|
|
|
|
|
|
def search(
|
|
db: DBSession,
|
|
user: User | None,
|
|
needle: str,
|
|
*,
|
|
limit: int = 10,
|
|
vector: list[float] | None = None,
|
|
) -> list[Note]:
|
|
"""Notes matching `needle` that this user may see, best match first.
|
|
|
|
`vector` is the query already embedded, or None. It comes from the caller
|
|
rather than being worked out here because this is synchronous and embedding
|
|
is an HTTP request -- see `services/library/retrieval.py`. None means the
|
|
keyword search exactly as it always was.
|
|
"""
|
|
hits = retrieval.search(db, INDEX, needle, kind=CHUNK_NOTE, vector=vector, limit=limit * 4)
|
|
if not hits:
|
|
return []
|
|
order = {hit.id: position for position, hit in enumerate(hits)}
|
|
rows = list(db.scalars(visible(db, user).where(Note.id.in_(list(order)))))
|
|
rows.sort(key=lambda note: order.get(note.id, len(order)))
|
|
return rows[:limit]
|
|
|
|
|
|
def create(
|
|
db: DBSession, *, owner: User, title: str, body: str, author: str = AUTHOR_USER
|
|
) -> Note:
|
|
note = Note(
|
|
owner_id=owner.id,
|
|
title=(title.strip() or "Untitled")[:MAX_TITLE_CHARS],
|
|
body=body.strip()[:MAX_BODY_CHARS],
|
|
author=author if author in (AUTHOR_USER, AUTHOR_MODEL) else AUTHOR_USER,
|
|
)
|
|
db.add(note)
|
|
db.commit()
|
|
return note
|
|
|
|
|
|
def update(db: DBSession, note: Note, *, title: str | None = None, body: str | None = None) -> Note:
|
|
"""Change a note. Absent arguments are left alone, which is what lets a tool
|
|
edit only the body without having to send the title back."""
|
|
if title is not None and title.strip():
|
|
note.title = title.strip()[:MAX_TITLE_CHARS]
|
|
if body is not None:
|
|
note.body = body.strip()[:MAX_BODY_CHARS]
|
|
db.commit()
|
|
return note
|
|
|
|
|
|
def delete(db: DBSession, note: Note) -> None:
|
|
sharing.forget_resource(db, note)
|
|
db.delete(note)
|
|
db.commit()
|
|
|
|
|
|
def snippet(note: Note) -> str:
|
|
text = (note.body or "").strip()
|
|
if len(text) <= SNIPPET_CHARS:
|
|
return text
|
|
return text[:SNIPPET_CHARS].rstrip() + "…"
|