# Extraction, embeddings and hybrid search

Read this before touching `services/files.py:limits`, `services/library/`'s new
three modules, or the `Chunk` table.

## Extraction is a snapshot, not a session

The constants in `services/files.py` are **defaults** now; what `prepare` reads
is `limits()`, a process-level snapshot with the same shape and the same
reasoning as `services/branding.py`. Threading a session through `prepare`,
`_process_image`, `_process_pdf` and `_process_text` would have meant six
signatures changed to carry a number, and several of their callers — the startup
sweep, a tool runner — have no session in hand.

`files.forget()` is called by `api/admin_extraction.py` and by nothing else. The
tests drop it between cases in `conftest.py` beside the branding one, for the
same reason.

Two things stayed constants on purpose:

- **`Image.MAX_IMAGE_PIXELS`** — a decompression-bomb guard, not a preference. A
  60,000×60,000 PNG is a few KB on disk and hundreds of gigabytes decoded, and
  nothing good comes of being able to raise that from a form.
- **`ORPHAN_AGE` in a signature.** `sweep_orphans(older_than=None)` resolves the
  default inside the body, because a default argument is evaluated at import and
  a module constant there would pin the shipped 24 hours whatever anybody set.

## Nothing changes for an instance that configures nothing

`embedding_model_id` empty means: no chunk rows written, no requests made,
`retrieval.search` returning exactly what `fts.search_ids` returns, in exactly
that order. That is asserted rather than claimed
(`test_with_no_model_search_is_exactly_the_keyword_search`), and it is what makes
this safe to land on an existing instance.

## Reciprocal rank fusion, and why not a weight

bm25 is a negative number whose scale depends on the corpus; cosine is 0..1. They
are not comparable, and normalising them onto a common scale means picking a
constant nobody can tune without a labelled test set they do not have.

RRF uses the **ranks**: `1 / (K + rank)`, summed. One constant, famously
insensitive to it, and it degrades to exactly one list when the other is empty —
which is what makes "no embedding model" a *branch that does not exist* rather
than a special case. `RRF_K` is deliberately not a setting: a number nobody can
evaluate is a number nobody should be asked about.

The fused `rank` is **larger for better**, the opposite of bm25's convention.
Nothing downstream reads it, but it is worth knowing.

## The query is embedded by the caller

`search()` is synchronous because every store's `search()` is, and every one of
those is called from both a route and a tool runner. Embedding is an HTTP
request. So the caller embeds first and passes a vector in; one that cannot
passes nothing and gets keywords.

`retrieval.worker_for(db)` and `retrieval.embed_with(worker, needle)` are split
for a specific reason: a **tool runner must not hold a database session across
an HTTP request**, so it resolves, closes, and awaits. A route that already holds
the request's session uses `embed_query(db, needle)`, which is the two together.

## A record scores as its best chunk

Not its average. One paragraph that answers the question is what makes a document
worth returning; averaging ranks a long document about something else above a
short one that says exactly the thing, because most of the long one is not about
anything.

`CHUNK_MULTIPLIER` is why the semantic side asks for more rows than are wanted:
one long document can own several of the best chunks and would otherwise crowd
everything else out.

## Vectors from two models never meet

`Chunk` stores `dims` and `model_id` beside every vector, and
`retrieval.semantic_ids` **skips a chunk whose width is not the query's**.
Changing the embedding model changes the space, and vectors from two spaces score
against each other perfectly happily and mean nothing — a search that works and
is wrong, which is the worst failure this feature can have. Nothing is deleted on
a model change; the stale rows are ignored until a rebuild replaces them, and the
save says so.

`unpack` checks the BLOB's length against the declared width for the same reason:
inferring the width would let a truncated row unpack into a shorter vector and
score happily.

## Indexing is fired and forgotten, and noticed by an event

Every library writer is synchronous and has just committed a row. None should
wait on a model server before saying "saved". So `schedule(kind, id)` starts a
task and returns; a save that cannot be indexed is still a save, and that record
falls back to keywords until the next rebuild.

**How a change is noticed is a SQLAlchemy session event, not a call in each of
the ten writers.** That is a departure from this codebase's taste for explicit
seams, and the reason is the one `tool_label` gives for being a Jinja global: a
step every writer has to remember is a step one of them will forget, and here
forgetting is silent — the record saves, keyword search still finds it, and only
its semantic recall is quietly stale.

`after_flush` collects and `after_commit` fires, in that order and never merged:
inside a flush the transaction has not landed, so a task started there could read
a row that does not exist yet — and `session.deleted` is empty by the time the
commit fires, so the collecting has to happen while it is not. `install()` is
idempotent because the app factory runs once per test.

A **deletion is scheduled like a change**: `index_resource` finds no row and drops
the chunks. One path rather than two, and the one that runs is the one that has
to be right anyway. `sweep_orphans` is the backstop for a delete with no event
loop to schedule anything — a CLI command, or a cascade from removing an account
— and runs at startup and at the end of every rebuild.

## Writing is all-or-nothing

`index_resource` embeds everything **before** it deletes anything. Deleting first
and failing half way through would leave a record indexed by half of itself,
which ranks worse than not being indexed at all and looks like nothing.

Staleness is a hash (`source_hash`) rather than a timestamp, so re-indexing an
unchanged record is free and "is this current?" is answerable without embedding
anything.

## The rebuild

One record at a time, never gathered: the far side is usually one local model
server, and twenty concurrent embedding requests against it is slower than twenty
sequential ones as well as being ruder. Each record commits, so a half-finished
index is usable.

`Progress` is in-process, because a rebuild does not survive a restart —
persisting it would mean a progress bar that stops moving and never finishes.
`admin/_index_progress.html` emits its `hx-trigger` **only while running**, so the
last frame has nothing attached and the polling stops by itself.

## The response order is trusted only as far as `index`

`_vectors_in` sorts on the declared `index` rather than on arrival order, and
refuses a response with a different number of vectors than inputs. Nothing in the
specification promises the order, and a provider that sorts differently would
pair every chunk with somebody else's vector — silently, for the life of the
index.