Finding a thing that does not use your words

Three pieces, and the first one is that they are all optional.

Extraction stops being constants. Upload size, image edge, JPEG quality, PDF
pages, extracted characters, orphan age and the text-extension list are settings
now, read through a process-level snapshot rather than a session -- `prepare` and
everything under it are called from routes, tool runners and the startup sweep,
and several of those have no session in hand. Two things deliberately stayed
constants: the decompression-bomb guard, which is a guard and not a preference,
and ORPHAN_AGE, which would have been evaluated at import if it stayed in the
signature and pinned the shipped 24 hours whatever anybody set.

An embedding model is picked from the models an administrator flagged for it, and
one that has since lost its flag is *named* rather than dropped from the picker:
a setting that vanishes is one nobody can tell from a setting never made. Nothing
here is required. Choosing none means no chunk rows, no requests, and
retrieval.search returning exactly what fts.search_ids returns in exactly that
order -- asserted, because it is what makes this safe to land on an instance that
never asked for it.

The two rankings are fused by reciprocal rank fusion: ranks and not scores,
because bm25 is a corpus-dependent negative and cosine is 0..1, and normalising
them onto one scale means picking a constant nobody can tune without a labelled
set they do not have. RRF's one constant is famously insensitive and degrades to
whichever list is non-empty -- which is what turns "no embedding model" into a
branch that does not exist.

A record scores as its best chunk rather than its average, or a long document
about something else outranks a short one that says the thing. Width and model
are stored beside every vector and a mismatch is skipped, because vectors from
two spaces score against each other perfectly happily and mean nothing -- a
search that works and is wrong is the worst failure this can have, and a model
change now leaves stale rows ignored rather than trusted.

Indexing is fired and forgotten, and how a change is noticed is a session event
rather than a call in each of the ten library writers. That is a departure from
this codebase's taste for explicit seams, for the reason tool_label is a Jinja
global: a step every writer has to remember is one that gets forgotten, and here
forgetting is silent -- the record saves, keyword search still finds it, and only
its recall goes stale. Chunks are embedded before anything is deleted, so a
failure leaves the old index rather than half a new one.

Also: `embeddings` joins the model capabilities, and the three tool flags that
had shipped with no checkbox -- canvas, scheduling and helpers -- have one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/notes/search-and-extraction.md

@@ -0,0 +1,142 @@
+# Extraction, embeddings and hybrid search
+
+Read this before touching `services/files.py:limits`, `services/library/`'s new
+three modules, or the `Chunk` table.
+
+## Extraction is a snapshot, not a session
+
+The constants in `services/files.py` are **defaults** now; what `prepare` reads
+is `limits()`, a process-level snapshot with the same shape and the same
+reasoning as `services/branding.py`. Threading a session through `prepare`,
+`_process_image`, `_process_pdf` and `_process_text` would have meant six
+signatures changed to carry a number, and several of their callers — the startup
+sweep, a tool runner — have no session in hand.
+
+`files.forget()` is called by `api/admin_extraction.py` and by nothing else. The
+tests drop it between cases in `conftest.py` beside the branding one, for the
+same reason.
+
+Two things stayed constants on purpose:
+
+- **`Image.MAX_IMAGE_PIXELS`** — a decompression-bomb guard, not a preference. A
+  60,000×60,000 PNG is a few KB on disk and hundreds of gigabytes decoded, and
+  nothing good comes of being able to raise that from a form.
+- **`ORPHAN_AGE` in a signature.** `sweep_orphans(older_than=None)` resolves the
+  default inside the body, because a default argument is evaluated at import and
+  a module constant there would pin the shipped 24 hours whatever anybody set.
+
+## Nothing changes for an instance that configures nothing
+
+`embedding_model_id` empty means: no chunk rows written, no requests made,
+`retrieval.search` returning exactly what `fts.search_ids` returns, in exactly
+that order. That is asserted rather than claimed
+(`test_with_no_model_search_is_exactly_the_keyword_search`), and it is what makes
+this safe to land on an existing instance.
+
+## Reciprocal rank fusion, and why not a weight
+
+bm25 is a negative number whose scale depends on the corpus; cosine is 0..1. They
+are not comparable, and normalising them onto a common scale means picking a
+constant nobody can tune without a labelled test set they do not have.
+
+RRF uses the **ranks**: `1 / (K + rank)`, summed. One constant, famously
+insensitive to it, and it degrades to exactly one list when the other is empty —
+which is what makes "no embedding model" a *branch that does not exist* rather
+than a special case. `RRF_K` is deliberately not a setting: a number nobody can
+evaluate is a number nobody should be asked about.
+
+The fused `rank` is **larger for better**, the opposite of bm25's convention.
+Nothing downstream reads it, but it is worth knowing.
+
+## The query is embedded by the caller
+
+`search()` is synchronous because every store's `search()` is, and every one of
+those is called from both a route and a tool runner. Embedding is an HTTP
+request. So the caller embeds first and passes a vector in; one that cannot
+passes nothing and gets keywords.
+
+`retrieval.worker_for(db)` and `retrieval.embed_with(worker, needle)` are split
+for a specific reason: a **tool runner must not hold a database session across
+an HTTP request**, so it resolves, closes, and awaits. A route that already holds
+the request's session uses `embed_query(db, needle)`, which is the two together.
+
+## A record scores as its best chunk
+
+Not its average. One paragraph that answers the question is what makes a document
+worth returning; averaging ranks a long document about something else above a
+short one that says exactly the thing, because most of the long one is not about
+anything.
+
+`CHUNK_MULTIPLIER` is why the semantic side asks for more rows than are wanted:
+one long document can own several of the best chunks and would otherwise crowd
+everything else out.
+
+## Vectors from two models never meet
+
+`Chunk` stores `dims` and `model_id` beside every vector, and
+`retrieval.semantic_ids` **skips a chunk whose width is not the query's**.
+Changing the embedding model changes the space, and vectors from two spaces score
+against each other perfectly happily and mean nothing — a search that works and
+is wrong, which is the worst failure this feature can have. Nothing is deleted on
+a model change; the stale rows are ignored until a rebuild replaces them, and the
+save says so.
+
+`unpack` checks the BLOB's length against the declared width for the same reason:
+inferring the width would let a truncated row unpack into a shorter vector and
+score happily.
+
+## Indexing is fired and forgotten, and noticed by an event
+
+Every library writer is synchronous and has just committed a row. None should
+wait on a model server before saying "saved". So `schedule(kind, id)` starts a
+task and returns; a save that cannot be indexed is still a save, and that record
+falls back to keywords until the next rebuild.
+
+**How a change is noticed is a SQLAlchemy session event, not a call in each of
+the ten writers.** That is a departure from this codebase's taste for explicit
+seams, and the reason is the one `tool_label` gives for being a Jinja global: a
+step every writer has to remember is a step one of them will forget, and here
+forgetting is silent — the record saves, keyword search still finds it, and only
+its semantic recall is quietly stale.
+
+`after_flush` collects and `after_commit` fires, in that order and never merged:
+inside a flush the transaction has not landed, so a task started there could read
+a row that does not exist yet — and `session.deleted` is empty by the time the
+commit fires, so the collecting has to happen while it is not. `install()` is
+idempotent because the app factory runs once per test.
+
+A **deletion is scheduled like a change**: `index_resource` finds no row and drops
+the chunks. One path rather than two, and the one that runs is the one that has
+to be right anyway. `sweep_orphans` is the backstop for a delete with no event
+loop to schedule anything — a CLI command, or a cascade from removing an account
+— and runs at startup and at the end of every rebuild.
+
+## Writing is all-or-nothing
+
+`index_resource` embeds everything **before** it deletes anything. Deleting first
+and failing half way through would leave a record indexed by half of itself,
+which ranks worse than not being indexed at all and looks like nothing.
+
+Staleness is a hash (`source_hash`) rather than a timestamp, so re-indexing an
+unchanged record is free and "is this current?" is answerable without embedding
+anything.
+
+## The rebuild
+
+One record at a time, never gathered: the far side is usually one local model
+server, and twenty concurrent embedding requests against it is slower than twenty
+sequential ones as well as being ruder. Each record commits, so a half-finished
+index is usable.
+
+`Progress` is in-process, because a rebuild does not survive a restart —
+persisting it would mean a progress bar that stops moving and never finishes.
+`admin/_index_progress.html` emits its `hx-trigger` **only while running**, so the
+last frame has nothing attached and the polling stops by itself.
+
+## The response order is trusted only as far as `index`
+
+`_vectors_in` sorts on the declared `index` rather than on arrival order, and
+refuses a response with a different number of vectors than inputs. Nothing in the
+specification promises the order, and a provider that sorts differently would
+pair every chunk with somebody else's vector — silently, for the life of the
+index.