Finding a thing that does not use your words

Three pieces, and the first one is that they are all optional.

Extraction stops being constants. Upload size, image edge, JPEG quality, PDF
pages, extracted characters, orphan age and the text-extension list are settings
now, read through a process-level snapshot rather than a session -- `prepare` and
everything under it are called from routes, tool runners and the startup sweep,
and several of those have no session in hand. Two things deliberately stayed
constants: the decompression-bomb guard, which is a guard and not a preference,
and ORPHAN_AGE, which would have been evaluated at import if it stayed in the
signature and pinned the shipped 24 hours whatever anybody set.

An embedding model is picked from the models an administrator flagged for it, and
one that has since lost its flag is *named* rather than dropped from the picker:
a setting that vanishes is one nobody can tell from a setting never made. Nothing
here is required. Choosing none means no chunk rows, no requests, and
retrieval.search returning exactly what fts.search_ids returns in exactly that
order -- asserted, because it is what makes this safe to land on an instance that
never asked for it.

The two rankings are fused by reciprocal rank fusion: ranks and not scores,
because bm25 is a corpus-dependent negative and cosine is 0..1, and normalising
them onto one scale means picking a constant nobody can tune without a labelled
set they do not have. RRF's one constant is famously insensitive and degrades to
whichever list is non-empty -- which is what turns "no embedding model" into a
branch that does not exist.

A record scores as its best chunk rather than its average, or a long document
about something else outranks a short one that says the thing. Width and model
are stored beside every vector and a mismatch is skipped, because vectors from
two spaces score against each other perfectly happily and mean nothing -- a
search that works and is wrong is the worst failure this can have, and a model
change now leaves stale rows ignored rather than trusted.

Indexing is fired and forgotten, and how a change is noticed is a session event
rather than a call in each of the ten library writers. That is a departure from
this codebase's taste for explicit seams, for the reason tool_label is a Jinja
global: a step every writer has to remember is one that gets forgotten, and here
forgetting is silent -- the record saves, keyword search still finds it, and only
its recall goes stale. Chunks are embedded before anything is deleted, so a
failure leaves the old index rather than half a new one.

Also: `embeddings` joins the model capabilities, and the three tool flags that
had shipped with no checkbox -- canvas, scheduling and helpers -- have one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/lembas/web/templates/admin/_index_progress.html

@@ -0,0 +1,66 @@
+{% from "_macros.html" import icon %}
+{#
+  The rebuild's state, swapped into itself.
+
+  It polls while a rebuild is running and stops when it is not: `hx-trigger` is
+  only emitted in the running branch, so the last frame is a plain fragment with
+  nothing attached to it. A poller that kept going after the work finished would
+  be a request every two seconds, forever, on a page somebody left open.
+
+  `hx-target="this"` and `hx-swap="outerHTML"` are both spelled out. This lives
+  inside a form on the page, and htmx resolves `hx-target` by walking up the DOM
+  -- an element in there that fetches and names no target aims at whatever an
+  ancestor said, which is the bug the jobs chip had.
+#}
+<div id="index-progress"
+     {% if progress.running %}
+     hx-get="/admin/extraction/progress"
+     hx-trigger="every 2s"
+     hx-target="this"
+     hx-swap="outerHTML"
+     {% endif %}>
+
+  {% if progress.running %}
+  <div class="alert">
+    {{ icon("clock", "icon--sm") }}
+    <span>
+      Rebuilding: {{ progress.done }} of {{ progress.total }} records
+      ({{ progress.percent }}%), {{ progress.written }} pieces written.
+      You can leave this page — it carries on.
+    </span>
+  </div>
+  {% elif progress.error %}
+  <div class="alert alert--error">
+    {{ icon("warning", "icon--sm") }} <span>{{ progress.error }}</span>
+  </div>
+  {% elif progress.total %}
+  <div class="alert alert--success">
+    {{ icon("check", "icon--sm") }}
+    <span>
+      Finished: {{ progress.done }} records, {{ progress.written }} pieces indexed.
+    </span>
+  </div>
+  {% endif %}
+
+  <p class="field__hint">
+    {% if counts %}
+    Indexed now:
+    {% for kind, count in counts | dictsort %}
+    <strong>{{ count }}</strong> from {{ kind }}s{{ "," if not loop.last }}
+    {% endfor %}.
+    {% else %}
+    Nothing is indexed yet.
+    {% endif %}
+  </p>
+
+  {% if ready %}
+  <button class="btn" type="button"
+          hx-post="/admin/extraction/rebuild"
+          hx-target="#index-progress"
+          hx-swap="outerHTML"
+          {{ 'disabled' if progress.running }}>
+    {{ icon("sparkle", "icon--sm") }}
+    {{ "Rebuilding…" if progress.running else "Rebuild the index" }}
+  </button>
+  {% endif %}
+</div>

src/lembas/web/templates/admin/_layout.html

@@ -52,6 +52,11 @@
           {{ icon("image", "icon--sm") }}
           <span class="nav-item__label">Image generation</span>
         </a>
+        <a class="nav-item {{ 'is-active' if section == 'extraction' }}"
+           href="/admin/extraction">
+          {{ icon("file-text", "icon--sm") }}
+          <span class="nav-item__label">Extraction</span>
+        </a>
         <a class="nav-item {{ 'is-active' if section == 'tools' }}" href="/admin/tools">
           {{ icon("link", "icon--sm") }}
           <span class="nav-item__label">Tools</span>

src/lembas/web/templates/admin/extraction.html

@@ -0,0 +1,226 @@
+{% extends "admin/_layout.html" %}
+{% from "_macros.html" import icon %}
+{% set section = "extraction" %}
+
+{% block title %}Extraction - {{ brand.name }}{% endblock %}
+{% block heading %}Extraction and search{% endblock %}
+
+{% block admin_content %}
+<p class="admin-lede">
+  What happens to a file between the upload and the model, and how anything is
+  found again afterwards. The two are the same pipeline: what is extracted
+  decides what there is to search.
+</p>
+
+{% if saved %}
+<div class="alert alert--success">{{ icon("check", "icon--sm") }} <span>{{ saved }}</span></div>
+{% endif %}
+
+{# --- Extraction ----------------------------------------------------------- #}
+<form method="post" action="/admin/extraction" class="form-grid">
+  <section class="card">
+    <h2 class="card__title">What a file may cost</h2>
+    <p class="field__hint">
+      Every number here is a trade, and a large one usually breaks a request
+      rather than being slow — the text of a whole book does not fit in a
+      context window, and a model handed it fails the request outright rather
+      than reading the first half.
+    </p>
+
+    <div class="field-row">
+      <div class="field">
+        <label class="field__label" for="max_upload_mb">Largest upload</label>
+        <input class="input" id="max_upload_mb" name="max_upload_mb"
+               type="number" min="1" max="512" step="1" value="{{ values.max_upload_mb }}">
+        <p class="field__hint">Megabytes, before anything is done to it.</p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="max_extracted_chars">Text kept per file</label>
+        <input class="input" id="max_extracted_chars" name="max_extracted_chars"
+               type="number" min="1000" max="5000000" step="1000"
+               value="{{ values.max_extracted_chars }}">
+        <p class="field__hint">
+          Characters. Roughly four to a token, so 120,000 is about 30,000 tokens
+          — already most of a small context window. The rest is cut and the
+          model is told so.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="max_pdf_pages">Pages read from a PDF</label>
+        <input class="input" id="max_pdf_pages" name="max_pdf_pages"
+               type="number" min="1" max="5000" step="1" value="{{ values.max_pdf_pages }}">
+        <p class="field__hint">
+          Extraction is slow and happens once, at upload. Beyond this the file
+          is still stored; only its text stops.
+        </p>
+      </div>
+    </div>
+
+    <div class="field-row">
+      <div class="field">
+        <label class="field__label" for="max_image_edge">Longest image edge</label>
+        <input class="input" id="max_image_edge" name="max_image_edge"
+               type="number" min="128" max="8192" step="16"
+               value="{{ values.max_image_edge }}">
+        <p class="field__hint">
+          Pixels. Images are re-encoded before they are sent, because a phone
+          photo is several megabytes of base64.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="jpeg_quality">JPEG quality</label>
+        <input class="input" id="jpeg_quality" name="jpeg_quality"
+               type="number" min="30" max="100" step="1" value="{{ values.jpeg_quality }}">
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="orphan_hours">Keep abandoned uploads for</label>
+        <input class="input" id="orphan_hours" name="orphan_hours"
+               type="number" min="1" max="8760" step="1" value="{{ values.orphan_hours }}">
+        <p class="field__hint">
+          Hours. A file chosen in the composer and never sent. Swept at startup.
+        </p>
+      </div>
+    </div>
+
+    <div class="field">
+      <label class="field__label" for="extra_text_extensions">Also treat as text</label>
+      <textarea class="textarea input--mono" id="extra_text_extensions"
+                name="extra_text_extensions" rows="4" spellcheck="false"
+      >{{ extensions_text }}</textarea>
+      <p class="field__hint">
+        One extension per line, <code>.env</code> or <code>env</code>. Only
+        needed to pick a media type — whether the bytes decode as text is what
+        actually decides, so an unlisted extension already works.
+      </p>
+    </div>
+
+    <div class="field">
+      <label class="checkbox">
+        <input type="checkbox" name="reject_unreadable_pdf" value="true"
+               {{ 'checked' if values.reject_unreadable_pdf }}>
+        <span>Refuse a PDF whose text cannot be read</span>
+      </label>
+      <p class="field__hint">
+        Off, a scanned PDF is stored with an explanation saying why it
+        contributes nothing — there is no OCR here. That is usually what
+        somebody wants: the file is still attached and still downloadable.
+      </p>
+    </div>
+  </section>
+
+  <div class="btn-row">
+    <button class="btn btn--primary" type="submit">Save extraction</button>
+  </div>
+</form>
+
+{# --- Semantic search ------------------------------------------------------ #}
+<form method="post" action="/admin/extraction/search" class="form-grid">
+  <section class="card">
+    <h2 class="card__title">Searching by meaning</h2>
+    <p class="field__hint">
+      Keyword search finds a document that uses your words. This finds one that
+      means the same thing — <em>"how do I get in"</em> reaching a note about
+      passwords. Both run and the two rankings are fused, so nothing that
+      keyword search found is lost.
+    </p>
+
+    {% if not models %}
+    <div class="alert">
+      {{ icon("warning", "icon--sm") }}
+      <span>
+        No model is marked as producing embeddings. Tick
+        <strong>embeddings</strong> on one under
+        <a href="/admin/models">Models</a> — usually a small dedicated model
+        such as <code>nomic-embed-text</code> or <code>bge-m3</code>, not a chat
+        model.
+      </span>
+    </div>
+    {% endif %}
+
+    {% if missing_model %}
+    <div class="alert alert--error">
+      {{ icon("warning", "icon--sm") }}
+      <span>
+        <code>{{ missing_model }}</code> is selected but is not available — its
+        model or its connection has gone, or it is no longer marked for
+        embeddings. Search has fallen back to keywords.
+      </span>
+    </div>
+    {% endif %}
+
+    <div class="field">
+      <label class="field__label" for="embedding_model_id">Embedding model</label>
+      <select class="input" id="embedding_model_id" name="embedding_model_id">
+        <option value="">None — keyword search only</option>
+        {% for model in models %}
+        <option value="{{ model.model_id }}"
+                {{ 'selected' if values.embedding_model_id == model.model_id }}>
+          {{ model.label }}
+        </option>
+        {% endfor %}
+      </select>
+      <p class="field__hint">
+        <strong>None</strong> is not a degraded mode: it is the keyword search
+        this has always had, with nothing stored and nothing sent anywhere.
+        Changing the model changes what a vector means, so anything already
+        indexed is ignored until it is rebuilt.
+      </p>
+    </div>
+
+    <div class="field-row">
+      <div class="field">
+        <label class="field__label" for="chunk_chars">Piece size</label>
+        <input class="input" id="chunk_chars" name="chunk_chars"
+               type="number" min="200" max="8000" step="50" value="{{ values.chunk_chars }}">
+        <p class="field__hint">
+          Characters. A record is split on paragraph boundaries into pieces of
+          about this size, and each is embedded separately — a document is found
+          by its best piece, not by its average.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="chunk_overlap">Overlap</label>
+        <input class="input" id="chunk_overlap" name="chunk_overlap"
+               type="number" min="0" max="4000" step="10" value="{{ values.chunk_overlap }}">
+        <p class="field__hint">
+          How much of each piece is repeated at the start of the next, so a
+          sentence across a boundary is whole somewhere. Capped at half the
+          piece size.
+        </p>
+      </div>
+
+      <div class="field">
+        <label class="field__label" for="embed_batch">Pieces per request</label>
+        <input class="input" id="embed_batch" name="embed_batch"
+               type="number" min="1" max="256" step="1" value="{{ values.embed_batch }}">
+        <p class="field__hint">
+          Lower this if the endpoint refuses large requests; raise it if a
+          rebuild is slow and the far side has room.
+        </p>
+      </div>
+    </div>
+  </section>
+
+  <div class="btn-row">
+    <button class="btn btn--primary" type="submit">Save search</button>
+  </div>
+</form>
+
+{# --- The index ------------------------------------------------------------ #}
+<section class="card">
+  <h2 class="card__title">The index</h2>
+  <p class="field__hint">
+    Documents, notes, skills and reports are indexed as they are written. A
+    rebuild is for everything that already existed — or for after changing the
+    model or the piece size, both of which make what is stored stop meaning
+    anything. It runs in the background and can be left.
+  </p>
+
+  {% include "admin/_index_progress.html" %}
+</section>
+{% endblock %}

src/lembas/web/templates/admin/model_detail.html

@@ -172,6 +172,9 @@
       <strong>vision</strong> lets images be sent, and <strong>tools</strong> is
       whether a tool list may be sent at all — turn it on for a model that does
       not support tool calling and every one of its replies fails.
+      <strong>embeddings</strong> is the odd one out: it says this is not a chat
+      model at all, but one that turns text into vectors, and it is what
+      <a href="/admin/extraction">Extraction</a> picks from.
     </p>
     <div class="checkbox-row">
       {% for name in capabilities %}