Finding a thing that does not use your words

Three pieces, and the first one is that they are all optional.

Extraction stops being constants. Upload size, image edge, JPEG quality, PDF
pages, extracted characters, orphan age and the text-extension list are settings
now, read through a process-level snapshot rather than a session -- `prepare` and
everything under it are called from routes, tool runners and the startup sweep,
and several of those have no session in hand. Two things deliberately stayed
constants: the decompression-bomb guard, which is a guard and not a preference,
and ORPHAN_AGE, which would have been evaluated at import if it stayed in the
signature and pinned the shipped 24 hours whatever anybody set.

An embedding model is picked from the models an administrator flagged for it, and
one that has since lost its flag is *named* rather than dropped from the picker:
a setting that vanishes is one nobody can tell from a setting never made. Nothing
here is required. Choosing none means no chunk rows, no requests, and
retrieval.search returning exactly what fts.search_ids returns in exactly that
order -- asserted, because it is what makes this safe to land on an instance that
never asked for it.

The two rankings are fused by reciprocal rank fusion: ranks and not scores,
because bm25 is a corpus-dependent negative and cosine is 0..1, and normalising
them onto one scale means picking a constant nobody can tune without a labelled
set they do not have. RRF's one constant is famously insensitive and degrades to
whichever list is non-empty -- which is what turns "no embedding model" into a
branch that does not exist.

A record scores as its best chunk rather than its average, or a long document
about something else outranks a short one that says the thing. Width and model
are stored beside every vector and a mismatch is skipped, because vectors from
two spaces score against each other perfectly happily and mean nothing -- a
search that works and is wrong is the worst failure this can have, and a model
change now leaves stale rows ignored rather than trusted.

Indexing is fired and forgotten, and how a change is noticed is a session event
rather than a call in each of the ten library writers. That is a departure from
this codebase's taste for explicit seams, for the reason tool_label is a Jinja
global: a step every writer has to remember is one that gets forgotten, and here
forgetting is silent -- the record saves, keyword search still finds it, and only
its recall goes stale. Chunks are embedded before anything is deleted, so a
failure leaves the old index rather than half a new one.

Also: `embeddings` joins the model capabilities, and the three tool flags that
had shipped with no checkbox -- canvas, scheduling and helpers -- have one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/lembas/__init__.py

@@ -1,3 +1,3 @@
 """LLeMbas - a Middle-earth themed web UI for OpenAI-compatible LLM endpoints."""
 
-__version__ = "0.9.4"
+__version__ = "0.9.5"

src/lembas/api/admin_extraction.py

@@ -0,0 +1,200 @@
+"""What happens to a file between the upload and the model, and how it is found.
+
+Two halves on one page because they are two ends of the same pipeline: what gets
+extracted decides what there is to search, and the search settings decide what
+becomes of it. Splitting them would mean an administrator setting a 300-page PDF
+limit on one screen and wondering on another why half a book is missing from the
+index.
+
+Every save drops `files.forget()`, and this is the only module that calls it —
+the same discipline `admin_branding` has with the branding snapshot, and for the
+same reason: a process-level cache whose save does not drop it is a setting that
+takes effect at the next restart.
+"""
+
+from __future__ import annotations
+
+import logging
+
+from fastapi import APIRouter, Form, Request, Response, status
+from fastapi.responses import RedirectResponse
+from sqlalchemy import select
+
+from lembas.api.deps import AdminUser, Db
+from lembas.db.models import Connection, Model
+from lembas.services import files as files_service
+from lembas.services import settings_store
+from lembas.services.library import indexing
+from lembas.web.templating import render
+
+log = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/admin/extraction", tags=["admin-extraction"])
+
+
+def _embedding_models(db: Db) -> list[Model]:
+    """Models an administrator has marked as producing embeddings.
+
+    Filtered rather than listed in full, the same shape `/admin/images` uses for
+    its reviewer: a chat model in this picker is a setting that looks configured
+    and fails on the first request, which is the shape of failure this codebase
+    keeps cataloguing.
+    """
+    return [
+        model
+        for model in db.scalars(
+            select(Model).join(Connection).order_by(Model.position, Model.model_id)
+        )
+        if (model.capabilities_json or {}).get("embeddings")
+    ]
+
+
+def _lines(text: str) -> list[str]:
+    return [line.strip() for line in (text or "").splitlines() if line.strip()]
+
+
+@router.get("")
+async def extraction_page(request: Request, db: Db, user: AdminUser, saved: str = ""):
+    values = settings_store.extraction(db)
+    models = _embedding_models(db)
+    return render(
+        request,
+        "admin/extraction.html",
+        {
+            "values": values,
+            "extensions_text": "\n".join(values.get("extra_text_extensions") or []),
+            "models": models,
+            # A model that was chosen and has since lost its flag, or its
+            # connection. Named rather than silently dropped from the picker:
+            # a setting that vanishes is one nobody can tell from one that was
+            # never made.
+            "missing_model": (
+                values["embedding_model_id"]
+                if values["embedding_model_id"]
+                and values["embedding_model_id"] not in {m.model_id for m in models}
+                else ""
+            ),
+            "ready": indexing.enabled(db),
+            "counts": indexing.counts(db),
+            "progress": indexing.progress(),
+            "saved": saved,
+        },
+    )
+
+
+@router.post("")
+async def save_extraction(
+    db: Db,
+    user: AdminUser,
+    max_upload_mb: int = Form(20),
+    max_image_edge: int = Form(1400),
+    jpeg_quality: int = Form(85),
+    max_pdf_pages: int = Form(300),
+    max_extracted_chars: int = Form(120_000),
+    orphan_hours: int = Form(24),
+    extra_text_extensions: str = Form(""),
+    reject_unreadable_pdf: bool = Form(False),
+) -> Response:
+    settings_store.update(
+        db,
+        {
+            # Clamped here as well as on read, for the reason the agent settings
+            # give: a number with no bound is a way to break the instance from
+            # a form.
+            "max_upload_mb": min(max(max_upload_mb, 1), 512),
+            "max_image_edge": min(max(max_image_edge, 128), 8192),
+            "jpeg_quality": min(max(jpeg_quality, 30), 100),
+            "max_pdf_pages": min(max(max_pdf_pages, 1), 5000),
+            "max_extracted_chars": min(max(max_extracted_chars, 1000), 5_000_000),
+            "orphan_hours": min(max(orphan_hours, 1), 8760),
+            "extra_text_extensions": _lines(extra_text_extensions),
+            "reject_unreadable_pdf": reject_unreadable_pdf,
+        },
+        key=settings_store.EXTRACTION,
+    )
+    files_service.forget()
+    log.info("extraction settings changed by %s", user.email)
+    return RedirectResponse(
+        "/admin/extraction?saved=Extraction+saved.", status_code=status.HTTP_303_SEE_OTHER
+    )
+
+
+@router.post("/search")
+async def save_search(
+    db: Db,
+    user: AdminUser,
+    embedding_model_id: str = Form(""),
+    chunk_chars: int = Form(1200),
+    chunk_overlap: int = Form(150),
+    embed_batch: int = Form(16),
+) -> Response:
+    """The semantic half.
+
+    Its own form and its own route, because the two halves have different
+    consequences: changing a chunk size invalidates every vector already stored,
+    and changing an upload limit does not. Keeping them apart is what lets the
+    page say so beside the control that does it.
+    """
+    before = settings_store.extraction(db)
+    settings_store.update(
+        db,
+        {
+            "embedding_model_id": embedding_model_id.strip()[:300],
+            "chunk_chars": min(max(chunk_chars, 200), 8000),
+            "chunk_overlap": max(chunk_overlap, 0),
+            "embed_batch": min(max(embed_batch, 1), 256),
+        },
+        key=settings_store.EXTRACTION,
+    )
+    files_service.forget()
+
+    # Changing the model changes the vector space, so what is stored stops
+    # meaning anything against a new query. Nothing is deleted -- the scorer
+    # already skips a width that does not match the query's, so a stale index is
+    # ignored rather than trusted -- but a rebuild is what makes it useful
+    # again, and offering it here is cheaper than leaving somebody to notice.
+    changed = before["embedding_model_id"] != embedding_model_id.strip()
+    message = "Search+saved."
+    if changed and embedding_model_id.strip():
+        message = "Search+saved.+Rebuild+the+index+to+use+the+new+model."
+    log.info("embedding model set to %r by %s", embedding_model_id, user.email)
+    return RedirectResponse(
+        f"/admin/extraction?saved={message}", status_code=status.HTTP_303_SEE_OTHER
+    )
+
+
+@router.post("/rebuild")
+async def rebuild(request: Request, db: Db, user: AdminUser) -> Response:
+    """Start a rebuild, and answer with the progress card.
+
+    A background task rather than a request that waits: embedding a library of a
+    few thousand records is minutes of HTTP round trips, and a page that hangs
+    for that long is one somebody reloads, which starts a second one.
+    """
+    started = indexing.start_rebuild()
+    if started:
+        log.info("index rebuild started by %s", user.email)
+    return render(
+        request,
+        "admin/_index_progress.html",
+        {"progress": indexing.progress(), "counts": indexing.counts(db), "ready": True},
+    )
+
+
+@router.get("/progress")
+async def rebuild_progress(request: Request, db: Db, user: AdminUser) -> Response:
+    """Polled while a rebuild runs. Stops polling itself when it finishes.
+
+    Polled rather than streamed for the reason `/api/chats/unread` is: this is
+    one small fragment on one page, and an SSE stream for it would be a second
+    streaming path to keep correct.
+    """
+    return render(
+        request,
+        "admin/_index_progress.html",
+        {
+            "progress": indexing.progress(),
+            "counts": indexing.counts(db),
+            "ready": indexing.enabled(db),
+        },
+    )

src/lembas/api/admin_models.py

@@ -23,7 +23,10 @@ router = APIRouter(tags=["admin-models"])
 
 # What the endpoint can do. Endpoints do not advertise any of this reliably, so
 # these are an administrator's assertion.
-PROTOCOL_CAPABILITIES = ("reasoning", "vision", "tools")
+# `embeddings` is the odd one out and is worth naming as such: the other three
+# say what a model can do in a *chat*, and this one says it is not for chatting
+# at all. It is what /admin/extraction picks from, and nothing else reads it.
+PROTOCOL_CAPABILITIES = ("reasoning", "vision", "tools", "embeddings")
 
 # Which tools this model is given. Distinct from the above: `tools` is whether a
 # tools array may be sent at all, these are what goes in it. Every one of them is
@@ -46,6 +49,9 @@ TOOL_CAPABILITIES = (
     ("tool_ask", "Ask the reader"),
     ("tool_report", "Reports"),
     ("tool_image", "Image generation"),
+    ("tool_scratch", "Canvas"),
+    ("tool_schedule", "Scheduling"),
+    ("tool_subagent", "Helpers"),
     ("tool_agent", "Agent execution"),
 )
 

src/lembas/api/library.py

@@ -40,6 +40,7 @@ from lembas.services.fetch import FetchError, fetch
 from lembas.services.library import documents as documents_service
 from lembas.services.library import memories as memories_service
 from lembas.services.library import notes as notes_service
+from lembas.services.library import retrieval
 from lembas.services.library import skills as skills_service
 from lembas.services.markdown import render_markdown
 from lembas.web.templating import render
@@ -175,7 +176,12 @@ async def base_detail(
         raise HTTPException(status.HTTP_404_NOT_FOUND, "That knowledge base is not available.")
 
     if q.strip():
-        rows = documents_service.search(db, user, q, limit=PAGE_SIZE, base_ids=[base.id])
+        # The reader's search box gets the same recall a model's does. `None`
+        # when nothing is configured, which is the keyword search unchanged.
+        vector = await retrieval.embed_query(db, q)
+        rows = documents_service.search(
+            db, user, q, limit=PAGE_SIZE, base_ids=[base.id], vector=vector
+        )
         pager = {"page": 1, "pages": 1, "total": len(rows)}
     else:
         rows, pager = _page(
@@ -241,7 +247,7 @@ async def upload_document(
     if base is not None and not sharing.can_write(base, user):
         raise HTTPException(status.HTTP_403_FORBIDDEN, "That base is not yours to add to.")
 
-    payload = await file.read(files_service.MAX_UPLOAD_BYTES + 1)
+    payload = await file.read(files_service.limits().max_upload_bytes + 1)
     try:
         document = documents_service.store_upload(
             db,
@@ -341,7 +347,9 @@ async def document_content(db: Db, user: RequiredUser, document_id: str) -> Resp
 @router.get("/library/notes")
 async def notes_list(request: Request, db: Db, user: RequiredUser, q: str = "", page: int = 1):
     if q.strip():
-        rows = notes_service.search(db, user, q, limit=PAGE_SIZE)
+        rows = notes_service.search(
+            db, user, q, limit=PAGE_SIZE, vector=await retrieval.embed_query(db, q)
+        )
         pager = {"page": 1, "pages": 1, "total": len(rows)}
     else:
         rows, pager = _page(
@@ -422,7 +430,9 @@ async def delete_note(db: Db, user: RequiredUser, note_id: str) -> Response:
 @router.get("/library/skills")
 async def skills_list(request: Request, db: Db, user: RequiredUser, q: str = "", page: int = 1):
     if q.strip():
-        rows = skills_service.search(db, user, q, limit=PAGE_SIZE)
+        rows = skills_service.search(
+            db, user, q, limit=PAGE_SIZE, vector=await retrieval.embed_query(db, q)
+        )
         pager = {"page": 1, "pages": 1, "total": len(rows)}
     else:
         rows, pager = _page(db, skills_service.visible(db, user).order_by(Skill.name), page)

src/lembas/api/reports.py

@@ -24,6 +24,7 @@ from lembas.api.library import PAGE_SIZE, _page
 from lembas.api.pages import sidebar_context
 from lembas.db.models import Report
 from lembas.services import reports as reports_service
+from lembas.services.library import retrieval
 from lembas.services.markdown import render_markdown
 from lembas.web.templating import render
 
@@ -35,7 +36,9 @@ router = APIRouter(dependencies=[Depends(require_permission("reports.use"))], ta
 @router.get("/reports")
 async def reports_list(request: Request, db: Db, user: RequiredUser, q: str = "", page: int = 1):
     if q.strip():
-        rows = reports_service.search(db, user, q, limit=PAGE_SIZE)
+        rows = reports_service.search(
+            db, user, q, limit=PAGE_SIZE, vector=await retrieval.embed_query(db, q)
+        )
         pager = {"page": 1, "pages": 1, "total": len(rows)}
     else:
         rows, pager = _page(

src/lembas/db/models/__init__.py

@@ -39,6 +39,11 @@ from lembas.db.models.image import ImageWorkflow
 from lembas.db.models.library import (
     AUTHOR_MODEL,
     AUTHOR_USER,
+    CHUNK_DOCUMENT,
+    CHUNK_KINDS,
+    CHUNK_NOTE,
+    CHUNK_REPORT,
+    CHUNK_SKILL,
     PRINCIPAL_GROUP,
     PRINCIPAL_USER,
     RESOURCE_BASE,
@@ -46,6 +51,7 @@ from lembas.db.models.library import (
     RESOURCE_SKILL,
     SOURCE_LINK,
     SOURCE_UPLOAD,
+    Chunk,
     Document,
     KnowledgeBase,
     Memory,
@@ -155,6 +161,12 @@ __all__ = [
     "Job",
     "Connection",
     "CustomTool",
+    "CHUNK_DOCUMENT",
+    "CHUNK_KINDS",
+    "CHUNK_NOTE",
+    "CHUNK_REPORT",
+    "CHUNK_SKILL",
+    "Chunk",
     "Document",
     "Folder",
     "Group",

src/lembas/db/models/library.py

@@ -29,6 +29,7 @@ from sqlalchemy import (
     ForeignKey,
     Index,
     Integer,
+    LargeBinary,
     String,
     Table,
     Text,
@@ -286,3 +287,69 @@ class Share(UUIDPrimaryKey, Timestamps, Base):
 
 Index("ix_shares_resource", Share.resource_type, Share.resource_id)
 Index("ix_shares_principal", Share.principal_type, Share.principal_id)
+
+
+# --- Semantic index -----------------------------------------------------------
+# What a chunk belongs to. Strings rather than a foreign key per store, because
+# one table serving four of them is what stops the chunking, the scoring and the
+# rebuild being written four times and drifting three ways.
+CHUNK_DOCUMENT = "document"
+CHUNK_NOTE = "note"
+CHUNK_SKILL = "skill"
+CHUNK_REPORT = "report"
+
+CHUNK_KINDS = (CHUNK_DOCUMENT, CHUNK_NOTE, CHUNK_SKILL, CHUNK_REPORT)
+
+
+class Chunk(UUIDPrimaryKey, Timestamps, Base):
+    """A piece of one library record, and its embedding.
+
+    **Additive, so `sync_schema` creates it at startup with no manual step**, and
+    absent-means-nothing: an instance with no embedding model chosen never writes
+    a row here and the search behaves exactly as it always did.
+
+    `owner_id` is denormalised off the resource. It is not used for
+    authorisation -- `services/sharing.py` is still the only definition of who
+    may see what, and scoring happens before that filter exactly as the
+    full-text path does -- but it is what makes "rebuild this person's index"
+    and "drop everything of theirs" one indexed query rather than four joins.
+
+    No foreign key on `resource_id`, for the reason `Share.principal_id` has
+    none: the column points at one of four tables depending on `resource_type`,
+    which SQLite cannot express. `indexing.forget_resource` deletes the rows.
+    """
+
+    __tablename__ = "chunks"
+
+    owner_id: Mapped[str] = mapped_column(
+        String(32), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True
+    )
+    resource_type: Mapped[str] = mapped_column(String(16), nullable=False)
+    resource_id: Mapped[str] = mapped_column(String(32), nullable=False)
+    # Where in the record this piece came from, so a set can be rebuilt in order
+    # and a hit can say which part matched.
+    ordinal: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    text: Mapped[str] = mapped_column(Text, default="")
+
+    # float32, little-endian, packed. A BLOB rather than JSON because a 1024
+    # dimension vector is 4KB packed and about 20KB as text, and every one of
+    # them is read on every semantic search.
+    vector: Mapped[bytes] = mapped_column(LargeBinary, nullable=False)
+    # How many floats are in it. Stored rather than derived from the length so a
+    # mismatch is a comparison this code refuses rather than one it gets wrong:
+    # changing the embedding model changes the space, and vectors from two
+    # spaces score against each other perfectly happily and mean nothing.
+    dims: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    # Which model wrote it, for the same reason. A rebuild is what reconciles
+    # them; until then the odd ones out are ignored rather than trusted.
+    model_id: Mapped[str] = mapped_column(String(300), default="")
+    # A hash of the text this set was built from. What makes re-indexing an
+    # unchanged record free, and what makes "is this index current?" answerable
+    # without re-embedding anything.
+    source_hash: Mapped[str] = mapped_column(String(64), default="")
+
+    def __repr__(self) -> str:
+        return f"<Chunk {self.resource_type}:{self.resource_id}#{self.ordinal}>"
+
+
+Index("ix_chunks_resource", Chunk.resource_type, Chunk.resource_id)

src/lembas/main.py

@@ -17,6 +17,7 @@ from lembas.api import (
     admin_agents,
     admin_audio,
     admin_branding,
+    admin_extraction,
     admin_images,
     admin_models,
     admin_prompts,
@@ -45,6 +46,7 @@ from lembas.api import (
 from lembas.api.deps import RedirectToLogin, is_htmx, login_redirect
 from lembas.config import settings
 from lembas.db.session import init_db
+from lembas.services.library import indexing
 from lembas.web.templating import STATIC_DIR, render
 
 log = logging.getLogger("lembas")
@@ -79,6 +81,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
         from lembas.services.chat import sweep_temporary
         from lembas.services.files import sweep_orphans
         from lembas.services.library.documents import sweep_unfiled
+        from lembas.services.library.indexing import sweep_orphans as sweep_chunks
         from lembas.services.suggestions import seed_defaults as seed_suggestions
 
         with session_scope() as db:
@@ -89,6 +92,10 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
             # Temporary chats older than a day. Startup only, like the sweeps
             # above it -- see services/chat.py:sweep_temporary.
             sweep_temporary(db)
+            # Chunks whose record has gone. A backstop for a delete that
+            # happened with no event loop to schedule the tidy-up -- a CLI
+            # command, or a cascade from removing an account.
+            sweep_chunks(db)
             # Three starting points on the empty screen, written once ever.
             seed_suggestions(db)
     except Exception:  # noqa: BLE001 - housekeeping must never block startup
@@ -150,6 +157,9 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
     # detached remote job, which keeps running and is rehydrated on the next
     # start. Only the watching stops here.
     await stop_jobs()
+    # A chunk set is written whole or not at all, so cancelling loses nothing
+    # a rebuild does not pick up again.
+    await indexing.shutdown()
     log.info("LLeMbas stopped")
 
 
@@ -165,6 +175,11 @@ def create_app() -> FastAPI:
 
     app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
 
+    # One place that notices a library record changing, rather than a call in
+    # each of the ten writers that touch those tables. Idempotent, because the
+    # factory is called per test. See services/library/indexing.py:install.
+    indexing.install()
+
     app.include_router(pages.router)
     app.include_router(auth.router)
     app.include_router(preferences.router)
@@ -184,6 +199,7 @@ def create_app() -> FastAPI:
     app.include_router(admin_models.router)
     app.include_router(admin_audio.router)
     app.include_router(admin_branding.router)
+    app.include_router(admin_extraction.router)
     app.include_router(admin_search.router)
     app.include_router(admin_schedules.router)
     app.include_router(admin_images.router)

src/lembas/services/files.py

@@ -34,6 +34,16 @@ from lembas.db.models import KIND_DOCUMENT, KIND_IMAGE, KIND_TEXT, Attachment
 log = logging.getLogger(__name__)
 
 # --- Limits ------------------------------------------------------------------
+# These are the *defaults*, and an administrator can move every one of them on
+# /admin/extraction. They stay here because a default belongs beside the code
+# that depends on it, and because `prepare` is called from places with no
+# database session at all.
+#
+# The values are read through `limits()`, a process-level snapshot with the same
+# shape and the same reasoning as `services/branding.py`: one query per process,
+# dropped when the page saves. Threading a session through `prepare`,
+# `_process_image`, `_process_pdf` and `_process_text` would have meant six
+# signatures changed to carry a number.
 MAX_UPLOAD_BYTES = 20 * 1024 * 1024
 
 # Longest edge after downscaling. Large enough for a model to read a screenshot
@@ -43,6 +53,9 @@ JPEG_QUALITY = 85
 
 # Pillow's own guard against decompression bombs: a 60,000x60,000 PNG is a few
 # KB on disk and hundreds of GB decoded.
+#
+# Deliberately NOT a setting. It is a guard, not a preference, and nothing good
+# comes of being able to raise it from a form.
 Image.MAX_IMAGE_PIXELS = 64_000_000
 
 MAX_PDF_PAGES = 300
@@ -77,6 +90,84 @@ TEXT_EXTENSIONS = {
 }
 
 
+@dataclass(frozen=True)
+class Limits:
+    """What extraction is allowed to spend, for one process.
+
+    A snapshot rather than a lookup per call: `prepare` and everything under it
+    are called from routes, from tool runners and from the startup sweep, and
+    several of them have no session in hand. The pattern and the cost are the
+    same as `services/branding.py` -- one query per process, dropped when the
+    admin page saves, and stale across workers until each next reads.
+    """
+
+    max_upload_bytes: int = MAX_UPLOAD_BYTES
+    max_image_edge: int = MAX_IMAGE_EDGE
+    jpeg_quality: int = JPEG_QUALITY
+    max_pdf_pages: int = MAX_PDF_PAGES
+    max_extracted_chars: int = MAX_EXTRACTED_CHARS
+    orphan_hours: int = 24
+    extra_text_extensions: tuple[str, ...] = ()
+    reject_unreadable_pdf: bool = False
+
+    def media_type_for(self, extension: str) -> str | None:
+        """The media type for a text extension, or None if it is not one.
+
+        The built-in table first, then the administrator's additions as plain
+        text. Additions are extensions and not a mapping, because the mapping is
+        a thing somebody would have to get right twice and the media type of a
+        `.env` is `text/plain` whatever anybody types.
+        """
+        if extension in TEXT_EXTENSIONS:
+            return TEXT_EXTENSIONS[extension]
+        return "text/plain" if extension in self.extra_text_extensions else None
+
+
+_LIMITS: Limits | None = None
+
+
+def limits() -> Limits:
+    """The current extraction limits. Never raises -- see `branding.snapshot`."""
+    global _LIMITS
+    if _LIMITS is not None:
+        return _LIMITS
+    try:
+        from lembas.db.session import session_scope
+        from lembas.services import settings_store
+
+        with session_scope() as db:
+            values = settings_store.extraction(db)
+        _LIMITS = Limits(
+            max_upload_bytes=int(values["max_upload_mb"]) * 1024 * 1024,
+            max_image_edge=int(values["max_image_edge"]),
+            jpeg_quality=int(values["jpeg_quality"]),
+            max_pdf_pages=int(values["max_pdf_pages"]),
+            max_extracted_chars=int(values["max_extracted_chars"]),
+            orphan_hours=int(values["orphan_hours"]),
+            extra_text_extensions=tuple(
+                _clean_extension(item) for item in values["extra_text_extensions"]
+            ),
+            reject_unreadable_pdf=bool(values.get("reject_unreadable_pdf")),
+        )
+    except Exception:  # noqa: BLE001 - the shipped defaults are a usable answer
+        log.debug("could not read extraction settings; using defaults", exc_info=True)
+        return Limits()
+    return _LIMITS
+
+
+def _clean_extension(raw: str) -> str:
+    value = str(raw or "").strip().lower()
+    if not value:
+        return ""
+    return value if value.startswith(".") else f".{value}"
+
+
+def forget() -> None:
+    """Drop the snapshot. Called by the admin page's save, and by tests."""
+    global _LIMITS
+    _LIMITS = None
+
+
 class FileError(Exception):
     """A rejected upload, with a message fit to show the user."""
 
@@ -135,6 +226,7 @@ def _looks_like_pdf(payload: bytes) -> bool:
 
 # --- Processing --------------------------------------------------------------
 def _process_image(payload: bytes) -> Prepared:
+    bounds = limits()
     try:
         with Image.open(io.BytesIO(payload)) as image:
             image.load()
@@ -145,8 +237,8 @@ def _process_image(payload: bytes) -> Prepared:
 
             width, height = frame.size
             longest = max(width, height)
-            if longest > MAX_IMAGE_EDGE:
-                scale = MAX_IMAGE_EDGE / longest
+            if longest > bounds.max_image_edge:
+                scale = bounds.max_image_edge / longest
                 frame = frame.resize(
                     (max(1, int(width * scale)), max(1, int(height * scale))),
                     Image.LANCZOS,
@@ -157,7 +249,7 @@ def _process_image(payload: bytes) -> Prepared:
                 frame.save(buffer, format="PNG", optimize=True)
                 media_type, extension = "image/png", ".png"
             else:
-                frame.save(buffer, format="JPEG", quality=JPEG_QUALITY, optimize=True)
+                frame.save(buffer, format="JPEG", quality=bounds.jpeg_quality, optimize=True)
                 media_type, extension = "image/jpeg", ".jpg"
 
             return Prepared(
@@ -175,6 +267,7 @@ def _process_image(payload: bytes) -> Prepared:
 
 
 def _process_pdf(payload: bytes) -> Prepared:
+    bounds = limits()
     from pypdf import PdfReader
     from pypdf.errors import PdfReadError
 
@@ -198,7 +291,7 @@ def _process_pdf(payload: bytes) -> Prepared:
         chunks: list[str] = []
         total = 0
 
-        for index, page in enumerate(reader.pages[:MAX_PDF_PAGES]):
+        for index, page in enumerate(reader.pages[:bounds.max_pdf_pages]):
             try:
                 text = page.extract_text() or ""
             except Exception as exc:  # noqa: BLE001 - one bad page is not fatal
@@ -208,14 +301,14 @@ def _process_pdf(payload: bytes) -> Prepared:
                 continue
             chunks.append(f"[page {index + 1}]\n{text.strip()}")
             total += len(text)
-            if total >= MAX_EXTRACTED_CHARS:
+            if total >= bounds.max_extracted_chars:
                 prepared.truncated = True
                 break
 
-        if prepared.pages > MAX_PDF_PAGES:
+        if prepared.pages > bounds.max_pdf_pages:
             prepared.truncated = True
 
-        prepared.extracted_text = "\n\n".join(chunks)[:MAX_EXTRACTED_CHARS]
+        prepared.extracted_text = "\n\n".join(chunks)[:bounds.max_extracted_chars]
 
         if not prepared.extracted_text.strip():
             # Almost always a scan. Saying so beats the model silently ignoring
@@ -236,6 +329,7 @@ def _process_pdf(payload: bytes) -> Prepared:
 
 
 def _process_text(payload: bytes, filename: str) -> Prepared:
+    bounds = limits()
     for encoding in ("utf-8", "utf-16", "latin-1"):
         try:
             text = payload.decode(encoding)
@@ -250,15 +344,15 @@ def _process_text(payload: bytes, filename: str) -> Prepared:
     if "\x00" in text[:4096]:
         raise FileError("That file is not text, and is not a format LLeMbas can read.")
 
-    truncated = len(text) > MAX_EXTRACTED_CHARS
+    truncated = len(text) > bounds.max_extracted_chars
     extension = Path(filename).suffix.lower()
 
     return Prepared(
         payload=payload,
         kind=KIND_TEXT,
-        media_type=TEXT_EXTENSIONS.get(extension, "text/plain"),
-        extension=extension if extension in TEXT_EXTENSIONS else ".txt",
-        extracted_text=text[:MAX_EXTRACTED_CHARS],
+        media_type=bounds.media_type_for(extension) or "text/plain",
+        extension=extension if bounds.media_type_for(extension) else ".txt",
+        extracted_text=text[:bounds.max_extracted_chars],
         truncated=truncated,
     )
 
@@ -311,8 +405,9 @@ def prepare(payload: bytes, filename: str, *, keep_original: bool = False) -> Pr
     """
     if not payload:
         raise FileError("That file is empty.")
-    if len(payload) > MAX_UPLOAD_BYTES:
-        raise FileError(f"Files must be under {MAX_UPLOAD_BYTES // (1024 * 1024)} MB.")
+    ceiling = limits().max_upload_bytes
+    if len(payload) > ceiling:
+        raise FileError(f"Files must be under {ceiling // (1024 * 1024)} MB.")
 
     if _detect_image(payload) is not None:
         return _keep_image(payload) if keep_original else _process_image(payload)
@@ -406,7 +501,7 @@ def store_text(
     on the tag around it, which is what a reader sees on the chip and what
     survives if the text is later truncated away from its own first line.
     """
-    body = text[:MAX_EXTRACTED_CHARS]
+    body = text[:limits().max_extracted_chars]
     payload = body.encode("utf-8")
 
     stored_name = f"{secrets.token_hex(16)}.txt"
@@ -565,12 +660,19 @@ def remove_files_for_chats(db: DBSession, chat_ids: list[str]) -> int:
     return removed
 
 
-def sweep_orphans(db: DBSession, older_than: timedelta = ORPHAN_AGE) -> int:
+def sweep_orphans(db: DBSession, older_than: timedelta | None = None) -> int:
     """Delete uploads that were never attached to a message.
 
     A file picked in the composer and then abandoned would otherwise sit on
     disk forever.
+
+    `older_than` defaults to the configured age rather than to a constant, and
+    it is resolved *here* rather than in the signature: a default argument is
+    evaluated at import, so a module-level `ORPHAN_AGE` in the signature would
+    pin the shipped 24 hours whatever an administrator later set.
     """
+    if older_than is None:
+        older_than = timedelta(hours=limits().orphan_hours)
     cutoff = datetime.now(UTC) - older_than
     orphans = list(db.scalars(select(Attachment).where(Attachment.message_id.is_(None))))
 
@@ -608,7 +710,7 @@ def data_uri(attachment: Attachment) -> str | None:
     return f"data:{attachment.media_type};base64,{encoded}"
 
 
-def preview_data_uri(payload: bytes, *, max_edge: int = MAX_IMAGE_EDGE) -> str | None:
+def preview_data_uri(payload: bytes, *, max_edge: int = 0) -> str | None:
     """The same thing for bytes in hand, downscaled, for a model to look at.
 
     Fidelity and weight are two different jobs. What is stored is what ComfyUI
@@ -619,9 +721,15 @@ def preview_data_uri(payload: bytes, *, max_edge: int = MAX_IMAGE_EDGE) -> str |
     Takes bytes rather than an Attachment: the reviewer looks at an image that
     may be about to be thrown away, and writing a row for something rejected
     seconds later is work with nothing to show for it.
+
+    `max_edge` of 0 means the configured one. Zero rather than None because the
+    caller that passes a number passes a number, and a sentinel that is also a
+    plausible value would be worse -- an edge of zero is not a picture.
     """
     import base64
 
+    max_edge = max_edge or limits().max_image_edge
+
     try:
         with Image.open(io.BytesIO(payload)) as image:
             image.load()
@@ -634,7 +742,7 @@ def preview_data_uri(payload: bytes, *, max_edge: int = MAX_IMAGE_EDGE) -> str |
                     Image.LANCZOS,
                 )
             buffer = io.BytesIO()
-            frame.save(buffer, format="JPEG", quality=JPEG_QUALITY, optimize=True)
+            frame.save(buffer, format="JPEG", quality=limits().jpeg_quality, optimize=True)
     except (Image.DecompressionBombError, UnidentifiedImageError, OSError, ValueError):
         log.warning("could not build a preview of a generated image", exc_info=True)
         return None

src/lembas/services/library/chunks.py

@@ -0,0 +1,125 @@
+"""Splitting a record into pieces small enough to embed, and packing vectors.
+
+One implementation, used by documents, notes, skills and reports. Three would
+drift, and drift here is invisible: a splitter that behaves differently for
+notes than for documents produces a search that works and ranks wrongly.
+
+## How it splits
+
+On **paragraph boundaries first**, falling back to lines and then to a hard cut,
+because a chunk that ends mid-sentence is one whose embedding is about half a
+thought. The overlap carries the tail of the previous chunk into the next, so a
+sentence that straddles a boundary is whole in one of them.
+
+Characters rather than tokens throughout. The count has to be made without
+asking the endpoint -- `services/tokens.py` already establishes four characters
+to a token as this codebase's estimate, and being 20% out about a chunk size is
+a slightly different chunk, not a wrong one.
+
+## Packing
+
+float32, little-endian. A 1024-dimension vector is 4KB packed and about 20KB as
+JSON text, and every one of them is read on every semantic search.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import struct
+
+# Below this a piece is not worth a row: the embedding of six words is mostly
+# noise, and a search that returns "and the following:" as its best hit is worse
+# than one that returns nothing.
+MIN_CHUNK_CHARS = 40
+
+
+def split(text: str, *, size: int = 1200, overlap: int = 150) -> list[str]:
+    """A record's text as pieces of roughly `size` characters.
+
+    `overlap` is how much of the previous piece rides along with the next. It is
+    clamped to half the size here as well as in the settings accessor, because
+    an overlap at or past the size means every piece starts where the last one
+    did and the loop never advances -- a hang rather than a bad index, so it is
+    refused in both places rather than in the more convenient one.
+    """
+    body = (text or "").strip()
+    if not body:
+        return []
+    size = max(200, int(size))
+    overlap = max(0, min(int(overlap), size // 2))
+    if len(body) <= size:
+        return [body]
+
+    pieces: list[str] = []
+    start = 0
+    while start < len(body):
+        end = min(start + size, len(body))
+        if end < len(body):
+            end = _boundary(body, start, end)
+        piece = body[start:end].strip()
+        if len(piece) >= MIN_CHUNK_CHARS:
+            pieces.append(piece)
+        if end >= len(body):
+            break
+        start = max(end - overlap, start + 1)
+    return pieces
+
+
+def _boundary(body: str, start: int, end: int) -> int:
+    """Where to cut, preferring a paragraph break and then a line break.
+
+    Searched backwards from the hard limit, and only within the last third of
+    the piece: a paragraph break near the *start* would produce a chunk a
+    fraction of the size, which is how a long document turns into hundreds of
+    tiny rows that each match nothing.
+    """
+    floor = start + (end - start) * 2 // 3
+    for marker in ("\n\n", "\n", ". "):
+        found = body.rfind(marker, floor, end)
+        if found > floor:
+            return found + len(marker)
+    return end
+
+
+def digest(text: str) -> str:
+    """A hash of what a chunk set was built from.
+
+    What makes re-indexing an unchanged record free, and what makes "is this
+    index current?" answerable without embedding anything. sha256 rather than
+    md5 for no reason beyond having no reason to prefer md5; both are being used
+    as a change detector rather than against an adversary.
+    """
+    return hashlib.sha256((text or "").encode("utf-8")).hexdigest()
+
+
+def pack(vector: list[float]) -> bytes:
+    return struct.pack(f"<{len(vector)}f", *vector)
+
+
+def unpack(blob: bytes, dims: int) -> list[float]:
+    """A stored vector, or an empty list if the row does not add up.
+
+    Length is checked against the declared width rather than inferred from it: a
+    truncated BLOB would otherwise unpack into a shorter vector and score
+    against a query happily, which is a wrong answer rather than a missing one.
+    """
+    if dims <= 0 or len(blob) != dims * 4:
+        return []
+    return list(struct.unpack(f"<{dims}f", blob))
+
+
+def dot(left: list[float], right: list[float]) -> float:
+    """Cosine similarity, given that both sides are already unit vectors.
+
+    Normalisation happens once, at write time, in `llm/embeddings.py` -- so
+    every comparison here is a multiply-and-add rather than two square roots per
+    pair. A width mismatch scores zero rather than raising: it means the vectors
+    came from two different models, and the honest answer to "how similar are
+    these?" across two spaces is "this tells you nothing".
+    """
+    if len(left) != len(right) or not left:
+        return 0.0
+    return sum(a * b for a, b in zip(left, right, strict=True))
+
+
+__all__ = ["MIN_CHUNK_CHARS", "digest", "dot", "pack", "split", "unpack"]

src/lembas/services/library/documents.py

@@ -18,11 +18,18 @@ from sqlalchemy import select
 from sqlalchemy.orm import Session as DBSession
 
 from lembas.config import settings
-from lembas.db.models import SOURCE_LINK, SOURCE_UPLOAD, Document, KnowledgeBase, User
+from lembas.db.models import (
+    CHUNK_DOCUMENT,
+    SOURCE_LINK,
+    SOURCE_UPLOAD,
+    Document,
+    KnowledgeBase,
+    User,
+)
 from lembas.services import files as files_service
 from lembas.services import sharing
 from lembas.services.fetch import Fetched
-from lembas.services.library.fts import search_ids
+from lembas.services.library import retrieval
 
 log = logging.getLogger(__name__)
 
@@ -287,8 +294,9 @@ def set_text(db: DBSession, document: Document, text: str) -> Document:
     The commit fires the `documents_fts` UPDATE trigger, so search stays correct
     with nothing else to do. See `db/migrations.py:ensure_fts`.
     """
-    document.extracted_text = text[:files_service.MAX_EXTRACTED_CHARS]
-    document.truncated = len(text) > files_service.MAX_EXTRACTED_CHARS
+    ceiling = files_service.limits().max_extracted_chars
+    document.extracted_text = text[:ceiling]
+    document.truncated = len(text) > ceiling
     document.extraction_error = ""
     db.commit()
     return document
@@ -301,14 +309,22 @@ def search(
     *,
     limit: int = 10,
     base_ids: list[str] | None = None,
+    vector: list[float] | None = None,
 ) -> list[Document]:
     """Documents matching `needle` that this user may see, best match first.
 
     The index is searched first and the visibility filter applied to the rows
     it returned. That order matters: filtering afterwards is what makes it
     impossible for a hit on somebody else's document to leak, even as a count.
+
+    `vector` is the query already embedded, or None. It comes from the caller
+    rather than being worked out here because this is synchronous and embedding
+    is an HTTP request -- see `services/library/retrieval.py`. None means the
+    keyword search exactly as it always was.
     """
-    hits = search_ids(db, INDEX, needle, limit=limit * 4)
+    hits = retrieval.search(
+        db, INDEX, needle, kind=CHUNK_DOCUMENT, vector=vector, limit=limit * 4
+    )
     if not hits:
         return []
 

src/lembas/services/library/indexing.py

@@ -0,0 +1,529 @@
+"""Keeping the semantic index current, and rebuilding it when it is not.
+
+## The shape, and why it is a background task
+
+Embedding is an HTTP request. Every writer in the library -- `documents.create`,
+`notes.edit`, `skills.save`, `reports.create` -- is synchronous and is called
+from a route or a tool runner that has just committed a row, and none of them
+should wait on a model server to answer before saying "saved".
+
+So indexing is **fired and forgotten**: `schedule(kind, id)` starts a task and
+returns immediately. A save that cannot be indexed is a save; the row is written
+either way and the search falls back to keywords for that record until the next
+rebuild. That is the whole degradation story, and it is the same one that covers
+having no embedding model at all.
+
+## Nothing is written when no model is chosen
+
+`embedding_model_id` empty means the FTS path exactly as it has always been --
+no chunk rows, no requests, no cost. That is what makes this safe to add to an
+instance that never asked for it, and it is asserted rather than assumed.
+
+## Staleness is a hash, not a timestamp
+
+Every chunk carries `source_hash` (of the text it was built from), `model_id`
+and `dims`. Re-indexing an unchanged record is free; a record whose text moved
+is rebuilt; a record embedded by a *different* model is rebuilt on the next pass
+and, until then, ignored by the scorer rather than trusted. Vectors from two
+spaces score against each other perfectly happily and mean nothing, which is a
+search that works and is wrong -- the worst failure this feature can have.
+
+## The rebuild is restartable and reports itself
+
+A half-finished index has to be usable rather than empty, so the rebuild walks
+records one at a time and commits each. `progress()` is what the admin page
+polls; it is in-process, because a rebuild does not survive a restart and
+pretending otherwise would mean a progress bar that never moves.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import logging
+from dataclasses import dataclass, field
+
+from sqlalchemy import delete, func, select
+from sqlalchemy.orm import Session as DBSession
+
+from lembas.db.models import (
+    CHUNK_DOCUMENT,
+    CHUNK_KINDS,
+    CHUNK_NOTE,
+    CHUNK_REPORT,
+    CHUNK_SKILL,
+    Chunk,
+    Connection,
+    Document,
+    Model,
+    Note,
+    Report,
+    Skill,
+)
+from lembas.db.session import session_scope
+from lembas.services import settings_store
+from lembas.services.library import chunks as chunk_service
+from lembas.services.llm.openai_client import Endpoint, LLMError
+
+log = logging.getLogger(__name__)
+
+# What each kind is, and how to get its text. One table rather than four
+# branches, for the reason `tool_labels` is one table: four copies of "which
+# columns make up the searchable text" is three chances to disagree.
+SOURCES: dict[str, tuple[type, tuple[str, ...]]] = {
+    CHUNK_DOCUMENT: (Document, ("title", "description", "extracted_text")),
+    CHUNK_NOTE: (Note, ("title", "body")),
+    CHUNK_SKILL: (Skill, ("name", "description", "body")),
+    CHUNK_REPORT: (Report, ("title", "summary", "body")),
+}
+
+# Tasks in flight, so a record saved twice in quick succession is indexed once
+# more rather than twice at the same time. Keyed on kind and id.
+_TASKS: dict[tuple[str, str], asyncio.Task] = {}
+
+
+# --- What the model is ----------------------------------------------------------
+@dataclass(frozen=True)
+class Embedder:
+    """Which model turns text into vectors, resolved while a session is open."""
+
+    endpoint: Endpoint
+    model_id: str
+    batch: int = 16
+
+
+def embedder(db: DBSession) -> Embedder | None:
+    """The configured embedding model, or None.
+
+    None is the answer to every "no" -- none chosen, the model row deleted, its
+    connection disabled -- and every caller reads it the same way: do nothing,
+    and let the keyword search stand. That is deliberately not an error. An
+    instance that never configured this is the common case, not a broken one.
+    """
+    values = settings_store.extraction(db)
+    wanted = str(values.get("embedding_model_id") or "").strip()
+    if not wanted:
+        return None
+    model = db.scalar(
+        select(Model)
+        .join(Connection)
+        .where(
+            Model.model_id == wanted,
+            Model.enabled.is_(True),
+            Connection.enabled.is_(True),
+        )
+        .order_by(Connection.position)
+    )
+    if model is None:
+        log.info("embedding model %r is configured but not available", wanted)
+        return None
+    connection = db.get(Connection, model.connection_id)
+    if connection is None:
+        return None
+    return Embedder(
+        endpoint=Endpoint.from_connection(connection),
+        model_id=model.model_id,
+        batch=int(values.get("embed_batch") or 16),
+    )
+
+
+def enabled(db: DBSession) -> bool:
+    return embedder(db) is not None
+
+
+# --- Reading a record -----------------------------------------------------------
+def text_of(row) -> str:
+    """The searchable text of one record, in the same order the FTS index uses.
+
+    Blank fields are dropped rather than joined as empty lines, so a note with
+    no body hashes the same before and after somebody clears its body twice.
+    """
+    kind = kind_of(row)
+    if kind is None:
+        return ""
+    _, columns = SOURCES[kind]
+    parts = [str(getattr(row, name, "") or "").strip() for name in columns]
+    return "\n\n".join(part for part in parts if part)
+
+
+def kind_of(row) -> str | None:
+    for kind, (model, _) in SOURCES.items():
+        if isinstance(row, model):
+            return kind
+    return None
+
+
+def owner_of(row) -> str:
+    return str(getattr(row, "owner_id", "") or "")
+
+
+# --- Writing the index ----------------------------------------------------------
+def forget_resource(db: DBSession, kind: str, resource_id: str) -> int:
+    """Drop every chunk of one record. Called when it is deleted.
+
+    A plain DELETE rather than a cascade, because `resource_id` has no foreign
+    key -- it points at one of four tables depending on `resource_type`, which
+    SQLite cannot express. Same reasoning as `Share.principal_id`.
+    """
+    result = db.execute(
+        delete(Chunk).where(Chunk.resource_type == kind, Chunk.resource_id == resource_id)
+    )
+    db.commit()
+    return int(result.rowcount or 0)
+
+
+def current_hash(db: DBSession, kind: str, resource_id: str) -> tuple[str, str]:
+    """The hash and model of the chunks already stored for a record."""
+    row = db.execute(
+        select(Chunk.source_hash, Chunk.model_id)
+        .where(Chunk.resource_type == kind, Chunk.resource_id == resource_id)
+        .limit(1)
+    ).first()
+    return (str(row[0] or ""), str(row[1] or "")) if row else ("", "")
+
+
+async def index_resource(kind: str, resource_id: str, *, force: bool = False) -> int:
+    """Rebuild one record's chunks. Returns how many were written.
+
+    Opens its own session, for the reason every background worker here does: it
+    outlives the request that scheduled it. Never raises -- a failure leaves the
+    old chunks in place, which is a slightly stale index rather than a hole, and
+    is strictly better than deleting first and failing to write.
+    """
+    if kind not in SOURCES:
+        return 0
+    try:
+        with session_scope() as db:
+            model, _ = SOURCES[kind]
+            row = db.get(model, resource_id)
+            if row is None:
+                forget_resource(db, kind, resource_id)
+                return 0
+            worker = embedder(db)
+            if worker is None:
+                return 0
+            body = text_of(row)
+            owner = owner_of(row)
+            values = settings_store.extraction(db)
+            digest = chunk_service.digest(body)
+            stored_hash, stored_model = current_hash(db, kind, resource_id)
+
+        if not body.strip():
+            with session_scope() as db:
+                forget_resource(db, kind, resource_id)
+            return 0
+        if not force and digest == stored_hash and stored_model == worker.model_id:
+            return 0
+
+        pieces = chunk_service.split(
+            body, size=int(values["chunk_chars"]), overlap=int(values["chunk_overlap"])
+        )
+        if not pieces:
+            with session_scope() as db:
+                forget_resource(db, kind, resource_id)
+            return 0
+
+        vectors = await _embed_all(worker, pieces)
+
+        # Written only once every vector is in hand. Deleting first and failing
+        # half way through would leave a record indexed by half of itself, which
+        # ranks worse than not being indexed at all and looks like nothing.
+        with session_scope() as db:
+            db.execute(
+                delete(Chunk).where(
+                    Chunk.resource_type == kind, Chunk.resource_id == resource_id
+                )
+            )
+            for ordinal, (piece, vector) in enumerate(zip(pieces, vectors, strict=True)):
+                db.add(
+                    Chunk(
+                        owner_id=owner,
+                        resource_type=kind,
+                        resource_id=resource_id,
+                        ordinal=ordinal,
+                        text=piece,
+                        vector=chunk_service.pack(vector),
+                        dims=len(vector),
+                        model_id=worker.model_id,
+                        source_hash=digest,
+                    )
+                )
+            db.commit()
+        return len(pieces)
+    except LLMError as exc:
+        log.info("could not index %s %s: %s", kind, resource_id, exc)
+        return 0
+    except asyncio.CancelledError:
+        raise
+    except Exception:  # noqa: BLE001 - one bad record must not stop a rebuild
+        log.exception("indexing %s %s failed", kind, resource_id)
+        return 0
+
+
+async def _embed_all(worker: Embedder, pieces: list[str]) -> list[list[float]]:
+    from lembas.services.llm import embeddings as embeddings_service
+
+    vectors: list[list[float]] = []
+    for start in range(0, len(pieces), worker.batch):
+        batch = pieces[start : start + worker.batch]
+        vectors.extend(await embeddings_service.embed(worker.endpoint, worker.model_id, batch))
+    return vectors
+
+
+# --- Scheduling -----------------------------------------------------------------
+def schedule(kind: str, resource_id: str) -> None:
+    """Index a record soon, without making its writer wait.
+
+    Called from synchronous writers that have just committed. Two things it is
+    careful about:
+
+    - **No running loop means do nothing.** A CLI command, a test, or the
+      startup sweep has no event loop to attach to, and building a coroutine
+      there produces "never awaited" at the caller's own line. The check is
+      before the coroutine, the same trap `push.announce_later` documents.
+    - **A record already being indexed is left alone.** Saving twice in a second
+      would otherwise embed the same text twice at once; the second call is
+      dropped and the record is picked up by the *next* save or rebuild, which
+      is why `index_resource` re-reads the row rather than taking text passed in.
+    """
+    if kind not in SOURCES or not resource_id:
+        return
+    try:
+        asyncio.get_running_loop()
+    except RuntimeError:
+        return
+    key = (kind, resource_id)
+    existing = _TASKS.get(key)
+    if existing is not None and not existing.done():
+        return
+    task = asyncio.create_task(index_resource(kind, resource_id))
+    _TASKS[key] = task
+    task.add_done_callback(lambda _t, k=key: _TASKS.pop(k, None))
+
+
+def schedule_for(row) -> None:
+    """The same, given a record rather than its kind and id."""
+    kind = kind_of(row)
+    if kind is not None:
+        schedule(kind, str(getattr(row, "id", "") or ""))
+
+
+# --- Noticing a change ----------------------------------------------------------
+# Two SQLAlchemy session events rather than a call in each of the ten writers
+# that touch these four tables. That is a departure from this codebase's taste
+# for explicit seams, and the reason is the one `tool_label` gives for being a
+# Jinja global: a step every writer has to remember is a step one of them will
+# forget, and here forgetting is *silent* -- the record saves, the keyword search
+# still finds it, and only its semantic recall is quietly stale.
+#
+# `after_flush` collects and `after_commit` acts, in that order and never
+# merged. Inside a flush the transaction has not landed yet, so a task started
+# there could read the row before it exists; and `session.deleted` is empty by
+# the time the commit fires, so the collecting has to happen while it is not.
+_PENDING = "lembas_index_pending"
+
+
+def _collect(session, _flush_context) -> None:
+    seen: set[tuple[str, str]] = session.info.setdefault(_PENDING, set())
+    for row in (*session.new, *session.dirty, *session.deleted):
+        kind = kind_of(row)
+        if kind is None:
+            continue
+        resource_id = str(getattr(row, "id", "") or "")
+        if resource_id:
+            seen.add((kind, resource_id))
+
+
+def _fire(session) -> None:
+    # A deletion is scheduled exactly like a change: `index_resource` finds no
+    # row and drops the chunks. One path rather than two, and the one that runs
+    # is the one that has to be right anyway.
+    for kind, resource_id in session.info.pop(_PENDING, set()):
+        schedule(kind, resource_id)
+
+
+def _forget(session) -> None:
+    session.info.pop(_PENDING, None)
+
+
+def install() -> None:
+    """Listen for library records changing. Called once, from the app factory.
+
+    Idempotent: `event.contains` is checked, because the app factory is called
+    per test in the suite and registering the same listener a hundred times
+    would index every record a hundred times over.
+    """
+    from sqlalchemy import event
+    from sqlalchemy.orm import Session
+
+    for name, handler in (
+        ("after_flush", _collect),
+        ("after_commit", _fire),
+        ("after_rollback", _forget),
+    ):
+        if not event.contains(Session, name, handler):
+            event.listen(Session, name, handler)
+
+
+def sweep_orphans(db: DBSession) -> int:
+    """Drop chunks whose record has gone.
+
+    A backstop for the one case the listeners cannot cover: a delete that
+    happened with no event loop running -- a CLI command, a test, a cascade from
+    deleting a user -- where `schedule` had nowhere to put its task. Cheap
+    enough to run at startup and at the end of every rebuild: one NOT IN per
+    kind, against an indexed column.
+    """
+    removed = 0
+    for kind, (model, _) in SOURCES.items():
+        result = db.execute(
+            delete(Chunk).where(
+                Chunk.resource_type == kind,
+                Chunk.resource_id.not_in(select(model.id)),
+            )
+        )
+        removed += int(result.rowcount or 0)
+    if removed:
+        db.commit()
+        log.info("dropped %d orphaned chunk(s)", removed)
+    return removed
+
+
+# --- Rebuilding everything ------------------------------------------------------
+@dataclass
+class Progress:
+    """What a rebuild has done so far.
+
+    In-process, because a rebuild does not survive a restart. Persisting it
+    would mean a progress bar that stops moving and never finishes, which is
+    worse than one that admits it is gone.
+    """
+
+    running: bool = False
+    total: int = 0
+    done: int = 0
+    written: int = 0
+    error: str = ""
+    kinds: dict[str, int] = field(default_factory=dict)
+
+    @property
+    def percent(self) -> int:
+        return int(self.done * 100 / self.total) if self.total else 0
+
+
+_PROGRESS = Progress()
+_REBUILD: asyncio.Task | None = None
+
+
+def progress() -> Progress:
+    return _PROGRESS
+
+
+def counts(db: DBSession) -> dict[str, int]:
+    """How many chunks exist per kind. What the page shows when nothing is running."""
+    rows = db.execute(
+        select(Chunk.resource_type, func.count()).group_by(Chunk.resource_type)
+    ).all()
+    return {str(kind): int(count) for kind, count in rows}
+
+
+async def rebuild_all(*, force: bool = True) -> None:
+    """Walk every record and index it, committing as it goes.
+
+    One at a time and never gathered. The far side is usually one local model
+    server, and twenty concurrent embedding requests against it is slower than
+    twenty sequential ones as well as being ruder.
+    """
+    global _PROGRESS
+    _PROGRESS = Progress(running=True)
+    try:
+        with session_scope() as db:
+            if embedder(db) is None:
+                _PROGRESS.error = "No embedding model is configured."
+                return
+            work: list[tuple[str, str]] = []
+            for kind, (model, _) in SOURCES.items():
+                ids = [row[0] for row in db.execute(select(model.id)).all()]
+                work.extend((kind, str(row_id)) for row_id in ids)
+        _PROGRESS.total = len(work)
+
+        for kind, resource_id in work:
+            written = await index_resource(kind, resource_id, force=force)
+            _PROGRESS.done += 1
+            _PROGRESS.written += written
+            _PROGRESS.kinds[kind] = _PROGRESS.kinds.get(kind, 0) + written
+
+        # After the walk, not before: a record deleted while this was running
+        # would otherwise be swept and then re-indexed from a row that no longer
+        # exists. `index_resource` handles that case too, and doing it in this
+        # order means one pass reconciles both directions.
+        with session_scope() as db:
+            sweep_orphans(db)
+    except asyncio.CancelledError:
+        _PROGRESS.error = "Stopped."
+        raise
+    except Exception as exc:  # noqa: BLE001 - a rebuild failing must be reportable
+        log.exception("rebuilding the index failed")
+        _PROGRESS.error = str(exc)
+    finally:
+        _PROGRESS.running = False
+
+
+def start_rebuild(*, force: bool = True) -> bool:
+    """Start a rebuild if one is not already going. True if this call started it."""
+    global _REBUILD
+    if _REBUILD is not None and not _REBUILD.done():
+        return False
+    try:
+        asyncio.get_running_loop()
+    except RuntimeError:
+        return False
+    _REBUILD = asyncio.create_task(rebuild_all(force=force))
+    return True
+
+
+async def shutdown() -> None:
+    """Cancel the rebuild and any in-flight indexing.
+
+    Nothing here is lost that matters: a chunk set is either written whole or
+    not at all, and the next rebuild picks up whatever was missed.
+    """
+    global _REBUILD
+    tasks = [task for task in (_REBUILD, *_TASKS.values()) if task is not None]
+    _TASKS.clear()
+    _REBUILD = None
+    for task in tasks:
+        task.cancel()
+    for task in tasks:
+        with contextlib.suppress(asyncio.CancelledError, Exception):
+            await task
+
+
+def clear() -> None:
+    """For tests: forget the in-process state without touching the database."""
+    global _REBUILD, _PROGRESS
+    _TASKS.clear()
+    _REBUILD = None
+    _PROGRESS = Progress()
+
+
+__all__ = [
+    "CHUNK_KINDS",
+    "SOURCES",
+    "Embedder",
+    "Progress",
+    "clear",
+    "counts",
+    "embedder",
+    "enabled",
+    "forget_resource",
+    "index_resource",
+    "kind_of",
+    "progress",
+    "rebuild_all",
+    "schedule",
+    "schedule_for",
+    "shutdown",
+    "start_rebuild",
+    "text_of",
+]

src/lembas/services/library/notes.py

@@ -13,9 +13,9 @@ import logging
 from sqlalchemy import select
 from sqlalchemy.orm import Session as DBSession
 
-from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, Note, User
+from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, CHUNK_NOTE, Note, User
 from lembas.services import sharing
-from lembas.services.library.fts import search_ids
+from lembas.services.library import retrieval
 
 log = logging.getLogger(__name__)
 
@@ -43,9 +43,22 @@ def recent(db: DBSession, user: User | None, *, limit: int = 20) -> list[Note]:
     )
 
 
-def search(db: DBSession, user: User | None, needle: str, *, limit: int = 10) -> list[Note]:
-    """Notes matching `needle` that this user may see, best match first."""
-    hits = search_ids(db, INDEX, needle, limit=limit * 4)
+def search(
+    db: DBSession,
+    user: User | None,
+    needle: str,
+    *,
+    limit: int = 10,
+    vector: list[float] | None = None,
+) -> list[Note]:
+    """Notes matching `needle` that this user may see, best match first.
+
+    `vector` is the query already embedded, or None. It comes from the caller
+    rather than being worked out here because this is synchronous and embedding
+    is an HTTP request -- see `services/library/retrieval.py`. None means the
+    keyword search exactly as it always was.
+    """
+    hits = retrieval.search(db, INDEX, needle, kind=CHUNK_NOTE, vector=vector, limit=limit * 4)
     if not hits:
         return []
     order = {hit.id: position for position, hit in enumerate(hits)}

src/lembas/services/library/retrieval.py

@@ -0,0 +1,207 @@
+"""Finding things: keywords, meaning, and the two fused.
+
+`fts.search_ids` was already the one seam every store searches through. This
+sits beside it and keeps that true — the four stores still call one function and
+still get ids back, and what changed is what is behind it.
+
+## Reciprocal rank fusion, and why not a weight
+
+Two rankings have to become one, and their scores are not comparable: bm25 is a
+negative number whose scale depends on the corpus, cosine is 0..1. Normalising
+them onto a common scale means picking a constant, and that constant is a knob
+nobody can tune without a labelled test set they do not have.
+
+RRF uses the **ranks** and not the scores: `1 / (K + rank)`, summed. It has one
+constant, `K`, it is famously insensitive to it, and it degrades to exactly one
+of the two lists when the other is empty — which is what makes "no embedding
+model configured" mean the keyword search, unchanged, with no branch anywhere
+that says so.
+
+## The query is embedded by the caller, not here
+
+`search` is synchronous, because every store's `search()` is and every one of
+them is called from both a route and a tool runner. Embedding is an HTTP request.
+So a caller that can await gets the query vector first and passes it in; one that
+cannot passes nothing and gets keywords. `embed_query` is the async half, and
+being able to answer `None` for every "no" is what keeps that from being a branch
+at each call site.
+
+## Visibility is still somebody else's job
+
+Both halves return ids, and both are scored across *everything* — the filter is
+applied to the row query afterwards, in each store, through
+`services/sharing.py`. That order is deliberate and is the same one the
+full-text path has always used: filtering afterwards is what makes it impossible
+for a hit on somebody else's record to leak, even as a count.
+"""
+
+from __future__ import annotations
+
+import logging
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session as DBSession
+
+from lembas.db.models import Chunk
+from lembas.services.library import chunks as chunk_service
+from lembas.services.library.fts import SearchHit, fts_query, search_ids
+
+log = logging.getLogger(__name__)
+
+# The one constant in reciprocal rank fusion. 60 is what the original paper used
+# and what everything since has copied; the method's whole appeal is that the
+# result barely moves for anything in the tens. It is not a tuning knob and is
+# deliberately not a setting -- a number nobody can evaluate is a number nobody
+# should be asked about.
+RRF_K = 60
+
+# How many chunks are scored before they are collapsed to records. Larger than
+# the number of records wanted, because one long document can own several of the
+# best chunks and would otherwise crowd everything else out of the answer.
+CHUNK_MULTIPLIER = 6
+
+
+def embeddable(db: DBSession) -> bool:
+    from lembas.services.library import indexing
+
+    return indexing.enabled(db)
+
+
+def worker_for(db: DBSession):
+    """The configured embedder, resolved while a session is open.
+
+    Split from the awaiting half deliberately. A caller that must not hold a
+    database session across an HTTP request -- a tool runner, which is about to
+    open its own -- resolves here, closes, and awaits `embed_with`. One that
+    already holds a request's session and is content to keep it can use
+    `embed_query` instead.
+    """
+    from lembas.services.library import indexing
+
+    return indexing.embedder(db)
+
+
+async def embed_with(worker, needle: str) -> list[float] | None:
+    """The query as a vector, or None.
+
+    None for every "no": no model configured, an empty query, an endpoint that
+    is down. Each of them means the same thing to the caller — search by
+    keywords — so none of them is an error, and a search that quietly stops
+    being semantic is far better than one that 500s because a model server was
+    restarting.
+    """
+    from lembas.services.llm import embeddings as embeddings_service
+    from lembas.services.llm.openai_client import LLMError
+
+    if worker is None or not (needle or "").strip():
+        return None
+    try:
+        vectors = await embeddings_service.embed(worker.endpoint, worker.model_id, [needle])
+    except LLMError as exc:
+        log.info("could not embed a query: %s", exc)
+        return None
+    return vectors[0] if vectors else None
+
+
+async def embed_query(db: DBSession, needle: str) -> list[float] | None:
+    """`worker_for` and `embed_with`, for a caller happy to hold its session."""
+    return await embed_with(worker_for(db), needle)
+
+
+def semantic_ids(
+    db: DBSession, kind: str, vector: list[float], *, limit: int = 20
+) -> list[SearchHit]:
+    """Record ids whose best chunk is closest to `vector`, best first.
+
+    A brute-force scan, and that is the right answer at this scale: a library of
+    ten thousand chunks is forty megabytes of float32 and a few million
+    multiply-adds, which is milliseconds. A real index is a later change behind
+    this same call, which is why the signature says nothing about how.
+
+    **A record scores as its best chunk, not its average.** One paragraph that
+    answers the question is what makes a document worth returning; averaging
+    would rank a long document about something else above a short one that says
+    exactly the thing, because most of the long one is not about anything.
+
+    Chunks whose width does not match the query's are skipped. That is a change
+    of embedding model with a rebuild still pending, and scoring across two
+    spaces produces a confident wrong answer rather than a missing one.
+    """
+    if not vector:
+        return []
+    width = len(vector)
+    rows = db.execute(
+        select(Chunk.resource_id, Chunk.vector, Chunk.dims).where(Chunk.resource_type == kind)
+    ).all()
+
+    best: dict[str, float] = {}
+    for resource_id, blob, dims in rows:
+        if int(dims or 0) != width:
+            continue
+        stored = chunk_service.unpack(blob, int(dims))
+        if not stored:
+            continue
+        score = chunk_service.dot(vector, stored)
+        key = str(resource_id)
+        if score > best.get(key, -2.0):
+            best[key] = score
+
+    ordered = sorted(best.items(), key=lambda pair: pair[1], reverse=True)
+    return [SearchHit(id=key, rank=score) for key, score in ordered[: max(1, limit)]]
+
+
+def fuse(*rankings: list[SearchHit], limit: int = 20) -> list[SearchHit]:
+    """Reciprocal rank fusion of any number of rankings.
+
+    The returned `rank` is the fused score, and it is **larger for better**,
+    which is the opposite of bm25's convention. Nothing downstream reads it --
+    every caller uses the order — but it is worth saying out loud rather than
+    leaving somebody to infer it from a negative number that is no longer there.
+    """
+    scores: dict[str, float] = {}
+    for ranking in rankings:
+        for position, hit in enumerate(ranking):
+            scores[hit.id] = scores.get(hit.id, 0.0) + 1.0 / (RRF_K + position + 1)
+    ordered = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
+    return [SearchHit(id=key, rank=score) for key, score in ordered[: max(1, limit)]]
+
+
+def search(
+    db: DBSession,
+    index: str,
+    needle: str,
+    *,
+    kind: str = "",
+    vector: list[float] | None = None,
+    limit: int = 20,
+) -> list[SearchHit]:
+    """Ids matching `needle`, keywords and meaning fused.
+
+    With no `vector` this is `fts.search_ids` and nothing else — the same call,
+    the same results, in the same order. That is what makes an instance with no
+    embedding model byte-for-byte what it always was, and it is asserted by a
+    test rather than left as a claim.
+    """
+    keyword = search_ids(db, index, needle, limit=limit)
+    if not vector or not kind:
+        return keyword
+    meaning = semantic_ids(db, kind, vector, limit=limit * CHUNK_MULTIPLIER)
+    if not meaning:
+        return keyword
+    if not keyword and not fts_query(needle):
+        # Nothing typed that FTS could match — a query of pure punctuation, or
+        # one whose every word is a separator. The semantic side still has an
+        # answer, and fusing a list with nothing is that list.
+        return meaning[:limit]
+    return fuse(keyword, meaning, limit=limit)
+
+
+__all__ = [
+    "CHUNK_MULTIPLIER",
+    "RRF_K",
+    "embed_query",
+    "embeddable",
+    "fuse",
+    "search",
+    "semantic_ids",
+]

src/lembas/services/library/skills.py

@@ -28,9 +28,9 @@ from collections.abc import Iterable
 from sqlalchemy import select
 from sqlalchemy.orm import Session as DBSession
 
-from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, Skill, SkillRevision, User
+from lembas.db.models import AUTHOR_MODEL, AUTHOR_USER, CHUNK_SKILL, Skill, SkillRevision, User
 from lembas.services import sharing
-from lembas.services.library.fts import search_ids
+from lembas.services.library import retrieval
 
 log = logging.getLogger(__name__)
 
@@ -103,8 +103,22 @@ def count_enabled(db: DBSession, user: User | None, *, exclude: Iterable[str] =
     return len(enabled_for(db, user, exclude=exclude))
 
 
-def search(db: DBSession, user: User | None, needle: str, *, limit: int = 10) -> list[Skill]:
-    hits = search_ids(db, INDEX, needle, limit=limit * 4)
+def search(
+    db: DBSession,
+    user: User | None,
+    needle: str,
+    *,
+    limit: int = 10,
+    vector: list[float] | None = None,
+) -> list[Skill]:
+    """Skills matching `needle` that this user may see, best match first.
+
+    `vector` is the query already embedded, or None. It comes from the caller
+    rather than being worked out here because this is synchronous and embedding
+    is an HTTP request -- see `services/library/retrieval.py`. None means the
+    keyword search exactly as it always was.
+    """
+    hits = retrieval.search(db, INDEX, needle, kind=CHUNK_SKILL, vector=vector, limit=limit * 4)
     if not hits:
         return []
     order = {hit.id: position for position, hit in enumerate(hits)}

src/lembas/services/llm/embeddings.py

@@ -0,0 +1,144 @@
+"""Turning text into vectors, against an OpenAI-shaped `/v1/embeddings`.
+
+The same reasoning as the chat and audio clients: plain httpx rather than an
+SDK, because the target is llama.cpp, Ollama, LM Studio, Infinity or vLLM at
+least as often as it is api.openai.com. They agree about the request and
+disagree politely about the response, so this is tolerant about what comes back
+and strict about what it hands on.
+
+**Batched, because the cost is the round trip.** A hundred chunks one at a time
+against a local endpoint is a hundred model loads' worth of latency for work
+that fits in six requests. The batch size is a setting, because "how many at
+once" is a property of the far side rather than of this code.
+
+**Dimensions are discovered, never declared.** Nobody should have to look up
+that bge-m3 is 1024 and nomic-embed-text is 768, and an instance that changes
+model must not silently compare vectors from two different spaces --
+`services/library/indexing.py` records the width beside every vector and
+refuses to score across a mismatch.
+
+Normalisation happens here, once, on the way out. Cosine similarity between two
+unit vectors is their dot product, so normalising at write time turns every
+later comparison into a multiply-and-add instead of two square roots per pair.
+"""
+
+from __future__ import annotations
+
+import logging
+import math
+from typing import Any
+
+import httpx
+
+from lembas.services.llm.openai_client import (
+    Endpoint,
+    LLMError,
+    describe_http_error,
+    wrap_transport_error,
+)
+
+log = logging.getLogger(__name__)
+
+# Longer than a chat request's, because a batch of sixteen chunks against a
+# cold local endpoint includes loading the model.
+TIMEOUT = 120.0
+
+
+def normalise(vector: list[float]) -> list[float]:
+    """A unit vector, or the input unchanged when it has no length.
+
+    A zero vector is what an endpoint returns for empty input, and dividing by
+    its norm is the one arithmetic error this path can make. It is left as it
+    is: scoring it against anything gives zero, which is the honest answer.
+    """
+    length = math.sqrt(sum(value * value for value in vector))
+    if length <= 0:
+        return vector
+    return [value / length for value in vector]
+
+
+def _vectors_in(payload: Any) -> list[list[float]]:
+    """The embeddings out of a response, whatever shape it arrived in.
+
+    OpenAI's own answer is `{"data": [{"embedding": [...], "index": 0}]}`, and
+    the index is honoured rather than assumed: nothing in the specification
+    promises the order, and a provider that sorts differently would silently
+    pair every chunk with somebody else's vector — which produces a search that
+    works and is wrong, the worst failure this whole feature can have.
+    """
+    if not isinstance(payload, dict):
+        raise LLMError("The embedding endpoint returned something unreadable.")
+    data = payload.get("data")
+    if not isinstance(data, list) or not data:
+        raise LLMError("The embedding endpoint returned no vectors.")
+
+    ordered: list[tuple[int, list[float]]] = []
+    for position, entry in enumerate(data):
+        if not isinstance(entry, dict):
+            raise LLMError("The embedding endpoint returned no vectors.")
+        raw = entry.get("embedding")
+        if not isinstance(raw, list) or not raw:
+            raise LLMError("The embedding endpoint returned an empty vector.")
+        index = entry.get("index")
+        at = int(index) if isinstance(index, int) else position
+        ordered.append((at, [float(value) for value in raw]))
+    ordered.sort(key=lambda pair: pair[0])
+    return [vector for _, vector in ordered]
+
+
+async def embed(
+    endpoint: Endpoint, model_id: str, texts: list[str], *, timeout: float = TIMEOUT
+) -> list[list[float]]:
+    """One request. Returns a unit vector per input, in the order given.
+
+    Raises `LLMError` for everything -- a missing model, an endpoint that does
+    not implement embeddings at all, a transport failure -- because every caller
+    treats them the same way: the index is left as it was and the search falls
+    back to keywords. Nothing here is worth a partial answer.
+    """
+    if not texts:
+        return []
+    body = {"model": model_id, "input": texts}
+    try:
+        async with httpx.AsyncClient(timeout=timeout) as client:
+            response = await client.post(
+                endpoint.url("/embeddings"), headers=endpoint.headers(), json=body
+            )
+            # raise_for_status, then translate. `describe_http_error` takes the
+            # exception rather than the response, which is what every other
+            # client here hands it.
+            response.raise_for_status()
+            payload = response.json()
+    except httpx.HTTPStatusError as exc:
+        raise LLMError(describe_http_error(exc)) from exc
+    except httpx.HTTPError as exc:
+        raise wrap_transport_error(exc, endpoint) from exc
+    except ValueError as exc:
+        raise LLMError("The embedding endpoint did not return JSON.") from exc
+
+    vectors = _vectors_in(payload)
+    if len(vectors) != len(texts):
+        # Not recoverable by guessing. A response with fewer vectors than inputs
+        # would pair chunk three's text with chunk four's vector from there on,
+        # for the life of the index.
+        raise LLMError(
+            f"Asked for {len(texts)} embeddings and got {len(vectors)}."
+        )
+    widths = {len(vector) for vector in vectors}
+    if len(widths) != 1:
+        raise LLMError("The embedding endpoint returned vectors of different widths.")
+    return [normalise(vector) for vector in vectors]
+
+
+async def probe(endpoint: Endpoint, model_id: str) -> int:
+    """How wide this model's vectors are, by asking for one.
+
+    Used by the admin page's Test button and by nothing on the request path.
+    There is no endpoint that reports it, so the only honest way to find out is
+    to embed something.
+    """
+    vectors = await embed(endpoint, model_id, ["lembas"], timeout=60.0)
+    return len(vectors[0])
+
+
+__all__ = ["TIMEOUT", "embed", "normalise", "probe"]

src/lembas/services/reports.py

@@ -19,8 +19,8 @@ import logging
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session as DBSession
 
-from lembas.db.models import SOURCE_MANUAL, SOURCES, Report, User
-from lembas.services.library.fts import search_ids
+from lembas.db.models import CHUNK_REPORT, SOURCE_MANUAL, SOURCES, Report, User
+from lembas.services.library import retrieval
 
 log = logging.getLogger(__name__)
 
@@ -56,14 +56,26 @@ def recent(db: DBSession, user: User | None, *, limit: int = 20) -> list[Report]
     return list(db.scalars(visible(user).order_by(Report.created_at.desc()).limit(limit)))
 
 
-def search(db: DBSession, user: User | None, needle: str, *, limit: int = 20) -> list[Report]:
+def search(
+    db: DBSession,
+    user: User | None,
+    needle: str,
+    *,
+    limit: int = 20,
+    vector: list[float] | None = None,
+) -> list[Report]:
     """Reports matching `needle`, best match first.
 
     Ids come back from FTS and the rows are re-ordered by hit position, exactly
     as the library stores do -- the index knows about ranking and the ORM query
     knows about ownership, and neither is asked to do the other's job.
+
+    `vector` is the query already embedded, or None. It comes from the caller
+    rather than being worked out here because this is synchronous and embedding
+    is an HTTP request -- see `services/library/retrieval.py`. None means the
+    keyword search exactly as it always was.
     """
-    hits = search_ids(db, INDEX, needle, limit=limit * 4)
+    hits = retrieval.search(db, INDEX, needle, kind=CHUNK_REPORT, vector=vector, limit=limit * 4)
     if not hits:
         return []
     order = {hit.id: position for position, hit in enumerate(hits)}

src/lembas/services/settings_store.py

@@ -33,6 +33,7 @@ IMAGES = "images"
 SCHEDULES = "schedules"
 SUBAGENTS = "subagents"
 BRANDING = "branding"
+EXTRACTION = "extraction"
 
 
 def _general_defaults() -> dict[str, Any]:
@@ -393,9 +394,93 @@ _DEFAULTS: dict[str, Any] = {
     # a label and a hint for the admin page and splitting the three across two
     # modules is how one of them goes stale.
     BRANDING: lambda: _branding_defaults(),
+    # A lambda for the same reason BRANDING is one: both factories are
+    # defined below this table, which is where the accessor that reads each
+    # group lives.
+    EXTRACTION: lambda: _extraction_defaults(),
 }
 
 
+def _extraction_defaults() -> dict[str, Any]:
+    """What happens to a file between the upload and the model.
+
+    The numbers were constants in `services/files.py` and every one of them is a
+    trade somebody with a different corpus makes differently: a 20 MB ceiling is
+    generous for notes and small for scans, and 120,000 characters is thirty
+    thousand tokens, which is most of a small window and a rounding error in a
+    large one. The defaults here are exactly the constants they replace, so an
+    instance that changes nothing behaves as it always did.
+
+    `Image.MAX_IMAGE_PIXELS` is deliberately **not** here. It is a
+    decompression-bomb guard, not a preference: a 60,000x60,000 PNG is a few KB
+    on disk and hundreds of gigabytes decoded, and nobody should be able to
+    raise that from a form.
+    """
+    return {
+        "max_upload_mb": 20,
+        "max_image_edge": 1400,
+        "jpeg_quality": 85,
+        "max_pdf_pages": 300,
+        "max_extracted_chars": 120_000,
+        "orphan_hours": 24,
+        # Extensions treated as text beyond the built-in list. Decodability is
+        # what actually decides, so this only picks a media type -- which is why
+        # it is a list of extensions rather than a mapping somebody has to get
+        # right twice.
+        "extra_text_extensions": [],
+        # Whether a PDF nothing could read is stored with its error, or refused.
+        # Keeping it is the default and the honest one: a scanned page is a file
+        # somebody still wants attached, and the error says why it contributes
+        # nothing rather than leaving them to wonder.
+        "reject_unreadable_pdf": False,
+        # --- Semantic search ---------------------------------------------------
+        # Which model turns text into vectors. Empty means none, and none means
+        # the keyword search that has always been here, byte for byte -- which
+        # is what makes this safe to add to an instance that never asked for it.
+        "embedding_model_id": "",
+        # How long a chunk is, in characters, and how much of the previous one
+        # rides along with it. Characters rather than tokens because the count
+        # has to be made without asking the endpoint, and the estimate is the
+        # same four-to-one this codebase already uses.
+        "chunk_chars": 1200,
+        "chunk_overlap": 150,
+        # How many chunks one embedding request carries. Small enough that a
+        # local endpoint is not asked for a megabyte at once.
+        "embed_batch": 16,
+    }
+
+
+def extraction(db: DBSession) -> dict[str, Any]:
+    """Extraction settings, clamped on read for the reason `agents` gives.
+
+    Every floor here is a number that means something bad at zero: a zero-page
+    PDF limit extracts nothing from every PDF and reports success, and a
+    zero-character chunk is an infinite loop in the splitter.
+    """
+    values = get_group(db, EXTRACTION)
+    values["max_upload_mb"] = min(max(int(values.get("max_upload_mb") or 1), 1), 512)
+    values["max_image_edge"] = min(max(int(values.get("max_image_edge") or 1), 128), 8192)
+    values["jpeg_quality"] = min(max(int(values.get("jpeg_quality") or 1), 30), 100)
+    values["max_pdf_pages"] = min(max(int(values.get("max_pdf_pages") or 1), 1), 5000)
+    values["max_extracted_chars"] = min(
+        max(int(values.get("max_extracted_chars") or 1), 1000), 5_000_000
+    )
+    values["orphan_hours"] = min(max(int(values.get("orphan_hours") or 1), 1), 8760)
+    values["chunk_chars"] = min(max(int(values.get("chunk_chars") or 1), 200), 8000)
+    # Bounded *against the chunk*, not absolutely: an overlap at or past the
+    # chunk size means every chunk starts where the last one did, which is a
+    # splitter that never advances.
+    values["chunk_overlap"] = min(
+        max(int(values.get("chunk_overlap") or 0), 0), values["chunk_chars"] // 2
+    )
+    values["embed_batch"] = min(max(int(values.get("embed_batch") or 1), 1), 256)
+    stored = values.get("extra_text_extensions")
+    values["extra_text_extensions"] = (
+        [str(item) for item in stored] if isinstance(stored, list) else []
+    )
+    return values
+
+
 def _branding_defaults() -> dict[str, Any]:
     """Imported inside the call: `services/branding.py` imports this module for
     the group key, so a top-level import back is a cycle."""

src/lembas/services/tools.py

@@ -417,10 +417,16 @@ async def _run_knowledge_search(context: ToolContext, args: dict[str, Any]) -> T
             {"name": "knowledge_search", "status": "error", "error": "No query."},
         )
 
+    # Embedded before the session opens, because it is an HTTP request and a
+    # session held across one is the trade `_maybe_compact` already refuses.
+    # None for every "no" -- no model configured, endpoint down -- and the
+    # search is then exactly the keyword one it has always been.
+    vector = await _query_vector(query)
+
     with session_scope() as db:
         user = db.get(User, context.owner_id)
         found = documents_service.search(
-            db, user, query, limit=6, base_ids=context.base_ids
+            db, user, query, limit=6, base_ids=context.base_ids, vector=vector
         )
         event = {
             "name": "knowledge_search",
@@ -468,13 +474,30 @@ async def _run_knowledge_get(context: ToolContext, args: dict[str, Any]) -> Tool
         return ToolOutcome(f"{document.title}\n\n{body}", event)
 
 
+async def _query_vector(query: str) -> list[float] | None:
+    """The query as a vector, for the stores that can use one.
+
+    Its own session, opened and closed before the caller opens theirs: this is
+    an HTTP request, and holding a database session across one is the trade
+    compaction and the project listing both already refuse.
+    """
+    if not query:
+        return None
+    from lembas.services.library import retrieval
+
+    with session_scope() as db:
+        worker = retrieval.worker_for(db)
+    return await retrieval.embed_with(worker, query)
+
+
 # --- Notes -------------------------------------------------------------------
 async def _run_notes_search(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
     query = str(args.get("query") or "").strip()
+    vector = await _query_vector(query)
     with session_scope() as db:
         user = db.get(User, context.owner_id)
         found = (
-            notes_service.search(db, user, query, limit=8)
+            notes_service.search(db, user, query, limit=8, vector=vector)
             if query
             else notes_service.recent(db, user, limit=8)
         )
@@ -747,10 +770,11 @@ async def _run_report_write(context: ToolContext, args: dict[str, Any]) -> ToolO
 
 async def _run_report_search(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
     query = str(args.get("query") or "").strip()
+    vector = await _query_vector(query)
     with session_scope() as db:
         user = db.get(User, context.owner_id)
         found = (
-            reports_service.search(db, user, query, limit=8)
+            reports_service.search(db, user, query, limit=8, vector=vector)
             if query
             else reports_service.recent(db, user, limit=8)
         )

src/lembas/web/templates/admin/_index_progress.html

@@ -0,0 +1,66 @@
+{% from "_macros.html" import icon %}
+{#
+  The rebuild's state, swapped into itself.
+
+  It polls while a rebuild is running and stops when it is not: `hx-trigger` is
+  only emitted in the running branch, so the last frame is a plain fragment with
+  nothing attached to it. A poller that kept going after the work finished would
+  be a request every two seconds, forever, on a page somebody left open.
+
+  `hx-target="this"` and `hx-swap="outerHTML"` are both spelled out. This lives
+  inside a form on the page, and htmx resolves `hx-target` by walking up the DOM
+  -- an element in there that fetches and names no target aims at whatever an
+  ancestor said, which is the bug the jobs chip had.
+#}
+<div id="index-progress"
+     {% if progress.running %}
+     hx-get="/admin/extraction/progress"
+     hx-trigger="every 2s"
+     hx-target="this"
+     hx-swap="outerHTML"
+     {% endif %}>
+
+  {% if progress.running %}
+  <div class="alert">
+    {{ icon("clock", "icon--sm") }}
+    <span>
+      Rebuilding: {{ progress.done }} of {{ progress.total }} records
+      ({{ progress.percent }}%), {{ progress.written }} pieces written.
+      You can leave this page — it carries on.
+    </span>
+  </div>
+  {% elif progress.error %}
+  <div class="alert alert--error">
+    {{ icon("warning", "icon--sm") }} <span>{{ progress.error }}</span>
+  </div>
+  {% elif progress.total %}
+  <div class="alert alert--success">
+    {{ icon("check", "icon--sm") }}
+    <span>
+      Finished: {{ progress.done }} records, {{ progress.written }} pieces indexed.
+    </span>
+  </div>
+  {% endif %}
+
+  <p class="field__hint">
+    {% if counts %}
+    Indexed now:
+    {% for kind, count in counts | dictsort %}
+    <strong>{{ count }}</strong> from {{ kind }}s{{ "," if not loop.last }}
+    {% endfor %}.
+    {% else %}
+    Nothing is indexed yet.
+    {% endif %}
+  </p>
+
+  {% if ready %}
+  <button class="btn" type="button"
+          hx-post="/admin/extraction/rebuild"
+          hx-target="#index-progress"
+          hx-swap="outerHTML"
+          {{ 'disabled' if progress.running }}>
+    {{ icon("sparkle", "icon--sm") }}
+    {{ "Rebuilding…" if progress.running else "Rebuild the index" }}
+  </button>
+  {% endif %}
+</div>

src/lembas/web/templates/admin/_layout.html

@@ -52,6 +52,11 @@
           {{ icon("image", "icon--sm") }}
           <span class="nav-item__label">Image generation</span>
         </a>
+        <a class="nav-item {{ 'is-active' if section == 'extraction' }}"
+           href="/admin/extraction">
+          {{ icon("file-text", "icon--sm") }}
+          <span class="nav-item__label">Extraction</span>
+        </a>
         <a class="nav-item {{ 'is-active' if section == 'tools' }}" href="/admin/tools">
           {{ icon("link", "icon--sm") }}
           <span class="nav-item__label">Tools</span>

src/lembas/web/templates/admin/extraction.html

@@ -0,0 +1,226 @@
+{% extends "admin/_layout.html" %}
+{% from "_macros.html" import icon %}
+{% set section = "extraction" %}
+
+{% block title %}Extraction - {{ brand.name }}{% endblock %}
+{% block heading %}Extraction and search{% endblock %}
+
+{% block admin_content %}
+<p class="admin-lede">
+  What happens to a file between the upload and the model, and how anything is
+  found again afterwards. The two are the same pipeline: what is extracted
+  decides what there is to search.
+</p>
+
+{% if saved %}
+<div class="alert alert--success">{{ icon("check", "icon--sm") }} <span>{{ saved }}</span></div>
+{% endif %}
+
+{# --- Extraction ----------------------------------------------------------- #}
+<form method="post" action="/admin/extraction" class="form-grid">
+  <section class="card">
+    <h2 class="card__title">What a file may cost</h2>
+    <p class="field__hint">
+      Every number here is a trade, and a large one usually breaks a request
+      rather than being slow — the text of a whole book does not fit in a
+      context window, and a model handed it fails the request outright rather
+      than reading the first half.
+    </p>
+
+    <div class="field-row">
+      <div class="field">
+        <label class="field__label" for="max_upload_mb">Largest upload</label>
+        <input class="input" id="max_upload_mb" name="max_upload_mb"
+               type="number" min="1" max="512" step="1" value="{{ values.max_upload_mb }}">
+        <p class="field__hint">Megabytes, before anything is done to it.</p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="max_extracted_chars">Text kept per file</label>
+        <input class="input" id="max_extracted_chars" name="max_extracted_chars"
+               type="number" min="1000" max="5000000" step="1000"
+               value="{{ values.max_extracted_chars }}">
+        <p class="field__hint">
+          Characters. Roughly four to a token, so 120,000 is about 30,000 tokens
+          — already most of a small context window. The rest is cut and the
+          model is told so.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="max_pdf_pages">Pages read from a PDF</label>
+        <input class="input" id="max_pdf_pages" name="max_pdf_pages"
+               type="number" min="1" max="5000" step="1" value="{{ values.max_pdf_pages }}">
+        <p class="field__hint">
+          Extraction is slow and happens once, at upload. Beyond this the file
+          is still stored; only its text stops.
+        </p>
+      </div>
+    </div>
+
+    <div class="field-row">
+      <div class="field">
+        <label class="field__label" for="max_image_edge">Longest image edge</label>
+        <input class="input" id="max_image_edge" name="max_image_edge"
+               type="number" min="128" max="8192" step="16"
+               value="{{ values.max_image_edge }}">
+        <p class="field__hint">
+          Pixels. Images are re-encoded before they are sent, because a phone
+          photo is several megabytes of base64.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="jpeg_quality">JPEG quality</label>
+        <input class="input" id="jpeg_quality" name="jpeg_quality"
+               type="number" min="30" max="100" step="1" value="{{ values.jpeg_quality }}">
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="orphan_hours">Keep abandoned uploads for</label>
+        <input class="input" id="orphan_hours" name="orphan_hours"
+               type="number" min="1" max="8760" step="1" value="{{ values.orphan_hours }}">
+        <p class="field__hint">
+          Hours. A file chosen in the composer and never sent. Swept at startup.
+        </p>
+      </div>
+    </div>
+
+    <div class="field">
+      <label class="field__label" for="extra_text_extensions">Also treat as text</label>
+      <textarea class="textarea input--mono" id="extra_text_extensions"
+                name="extra_text_extensions" rows="4" spellcheck="false"
+      >{{ extensions_text }}</textarea>
+      <p class="field__hint">
+        One extension per line, <code>.env</code> or <code>env</code>. Only
+        needed to pick a media type — whether the bytes decode as text is what
+        actually decides, so an unlisted extension already works.
+      </p>
+    </div>
+
+    <div class="field">
+      <label class="checkbox">
+        <input type="checkbox" name="reject_unreadable_pdf" value="true"
+               {{ 'checked' if values.reject_unreadable_pdf }}>
+        <span>Refuse a PDF whose text cannot be read</span>
+      </label>
+      <p class="field__hint">
+        Off, a scanned PDF is stored with an explanation saying why it
+        contributes nothing — there is no OCR here. That is usually what
+        somebody wants: the file is still attached and still downloadable.
+      </p>
+    </div>
+  </section>
+
+  <div class="btn-row">
+    <button class="btn btn--primary" type="submit">Save extraction</button>
+  </div>
+</form>
+
+{# --- Semantic search ------------------------------------------------------ #}
+<form method="post" action="/admin/extraction/search" class="form-grid">
+  <section class="card">
+    <h2 class="card__title">Searching by meaning</h2>
+    <p class="field__hint">
+      Keyword search finds a document that uses your words. This finds one that
+      means the same thing — <em>"how do I get in"</em> reaching a note about
+      passwords. Both run and the two rankings are fused, so nothing that
+      keyword search found is lost.
+    </p>
+
+    {% if not models %}
+    <div class="alert">
+      {{ icon("warning", "icon--sm") }}
+      <span>
+        No model is marked as producing embeddings. Tick
+        <strong>embeddings</strong> on one under
+        <a href="/admin/models">Models</a> — usually a small dedicated model
+        such as <code>nomic-embed-text</code> or <code>bge-m3</code>, not a chat
+        model.
+      </span>
+    </div>
+    {% endif %}
+
+    {% if missing_model %}
+    <div class="alert alert--error">
+      {{ icon("warning", "icon--sm") }}
+      <span>
+        <code>{{ missing_model }}</code> is selected but is not available — its
+        model or its connection has gone, or it is no longer marked for
+        embeddings. Search has fallen back to keywords.
+      </span>
+    </div>
+    {% endif %}
+
+    <div class="field">
+      <label class="field__label" for="embedding_model_id">Embedding model</label>
+      <select class="input" id="embedding_model_id" name="embedding_model_id">
+        <option value="">None — keyword search only</option>
+        {% for model in models %}
+        <option value="{{ model.model_id }}"
+                {{ 'selected' if values.embedding_model_id == model.model_id }}>
+          {{ model.label }}
+        </option>
+        {% endfor %}
+      </select>
+      <p class="field__hint">
+        <strong>None</strong> is not a degraded mode: it is the keyword search
+        this has always had, with nothing stored and nothing sent anywhere.
+        Changing the model changes what a vector means, so anything already
+        indexed is ignored until it is rebuilt.
+      </p>
+    </div>
+
+    <div class="field-row">
+      <div class="field">
+        <label class="field__label" for="chunk_chars">Piece size</label>
+        <input class="input" id="chunk_chars" name="chunk_chars"
+               type="number" min="200" max="8000" step="50" value="{{ values.chunk_chars }}">
+        <p class="field__hint">
+          Characters. A record is split on paragraph boundaries into pieces of
+          about this size, and each is embedded separately — a document is found
+          by its best piece, not by its average.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="chunk_overlap">Overlap</label>
+        <input class="input" id="chunk_overlap" name="chunk_overlap"
+               type="number" min="0" max="4000" step="10" value="{{ values.chunk_overlap }}">
+        <p class="field__hint">
+          How much of each piece is repeated at the start of the next, so a
+          sentence across a boundary is whole somewhere. Capped at half the
+          piece size.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="embed_batch">Pieces per request</label>
+        <input class="input" id="embed_batch" name="embed_batch"
+               type="number" min="1" max="256" step="1" value="{{ values.embed_batch }}">
+        <p class="field__hint">
+          Lower this if the endpoint refuses large requests; raise it if a
+          rebuild is slow and the far side has room.
+        </p>
+      </div>
+    </div>
+  </section>
+
+  <div class="btn-row">
+    <button class="btn btn--primary" type="submit">Save search</button>
+  </div>
+</form>
+
+{# --- The index ------------------------------------------------------------ #}
+<section class="card">
+  <h2 class="card__title">The index</h2>
+  <p class="field__hint">
+    Documents, notes, skills and reports are indexed as they are written. A
+    rebuild is for everything that already existed — or for after changing the
+    model or the piece size, both of which make what is stored stop meaning
+    anything. It runs in the background and can be left.
+  </p>
+
+  {% include "admin/_index_progress.html" %}
+</section>
+{% endblock %}

src/lembas/web/templates/admin/model_detail.html

@@ -172,6 +172,9 @@
       <strong>vision</strong> lets images be sent, and <strong>tools</strong> is
       whether a tool list may be sent at all — turn it on for a model that does
       not support tool calling and every one of its replies fails.
+      <strong>embeddings</strong> is the odd one out: it says this is not a chat
+      model at all, but one that turns text into vectors, and it is what
+      <a href="/admin/extraction">Extraction</a> picks from.
     </p>
     <div class="checkbox-row">
       {% for name in capabilities %}