Finding a thing that does not use your words

Three pieces, and the first one is that they are all optional.

Extraction stops being constants. Upload size, image edge, JPEG quality, PDF
pages, extracted characters, orphan age and the text-extension list are settings
now, read through a process-level snapshot rather than a session -- `prepare` and
everything under it are called from routes, tool runners and the startup sweep,
and several of those have no session in hand. Two things deliberately stayed
constants: the decompression-bomb guard, which is a guard and not a preference,
and ORPHAN_AGE, which would have been evaluated at import if it stayed in the
signature and pinned the shipped 24 hours whatever anybody set.

An embedding model is picked from the models an administrator flagged for it, and
one that has since lost its flag is *named* rather than dropped from the picker:
a setting that vanishes is one nobody can tell from a setting never made. Nothing
here is required. Choosing none means no chunk rows, no requests, and
retrieval.search returning exactly what fts.search_ids returns in exactly that
order -- asserted, because it is what makes this safe to land on an instance that
never asked for it.

The two rankings are fused by reciprocal rank fusion: ranks and not scores,
because bm25 is a corpus-dependent negative and cosine is 0..1, and normalising
them onto one scale means picking a constant nobody can tune without a labelled
set they do not have. RRF's one constant is famously insensitive and degrades to
whichever list is non-empty -- which is what turns "no embedding model" into a
branch that does not exist.

A record scores as its best chunk rather than its average, or a long document
about something else outranks a short one that says the thing. Width and model
are stored beside every vector and a mismatch is skipped, because vectors from
two spaces score against each other perfectly happily and mean nothing -- a
search that works and is wrong is the worst failure this can have, and a model
change now leaves stale rows ignored rather than trusted.

Indexing is fired and forgotten, and how a change is noticed is a session event
rather than a call in each of the ten library writers. That is a departure from
this codebase's taste for explicit seams, for the reason tool_label is a Jinja
global: a step every writer has to remember is one that gets forgotten, and here
forgetting is silent -- the record saves, keyword search still finds it, and only
its recall goes stale. Chunks are embedded before anything is deleted, so a
failure leaves the old index rather than half a new one.

Also: `embeddings` joins the model capabilities, and the three tool flags that
had shipped with no checkbox -- canvas, scheduling and helpers -- have one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/lembas/services/settings_store.py

@@ -33,6 +33,7 @@ IMAGES = "images"
 SCHEDULES = "schedules"
 SUBAGENTS = "subagents"
 BRANDING = "branding"
+EXTRACTION = "extraction"
 
 
 def _general_defaults() -> dict[str, Any]:
@@ -393,9 +394,93 @@ _DEFAULTS: dict[str, Any] = {
     # a label and a hint for the admin page and splitting the three across two
     # modules is how one of them goes stale.
     BRANDING: lambda: _branding_defaults(),
+    # A lambda for the same reason BRANDING is one: both factories are
+    # defined below this table, which is where the accessor that reads each
+    # group lives.
+    EXTRACTION: lambda: _extraction_defaults(),
 }
 
 
+def _extraction_defaults() -> dict[str, Any]:
+    """What happens to a file between the upload and the model.
+
+    The numbers were constants in `services/files.py` and every one of them is a
+    trade somebody with a different corpus makes differently: a 20 MB ceiling is
+    generous for notes and small for scans, and 120,000 characters is thirty
+    thousand tokens, which is most of a small window and a rounding error in a
+    large one. The defaults here are exactly the constants they replace, so an
+    instance that changes nothing behaves as it always did.
+
+    `Image.MAX_IMAGE_PIXELS` is deliberately **not** here. It is a
+    decompression-bomb guard, not a preference: a 60,000x60,000 PNG is a few KB
+    on disk and hundreds of gigabytes decoded, and nobody should be able to
+    raise that from a form.
+    """
+    return {
+        "max_upload_mb": 20,
+        "max_image_edge": 1400,
+        "jpeg_quality": 85,
+        "max_pdf_pages": 300,
+        "max_extracted_chars": 120_000,
+        "orphan_hours": 24,
+        # Extensions treated as text beyond the built-in list. Decodability is
+        # what actually decides, so this only picks a media type -- which is why
+        # it is a list of extensions rather than a mapping somebody has to get
+        # right twice.
+        "extra_text_extensions": [],
+        # Whether a PDF nothing could read is stored with its error, or refused.
+        # Keeping it is the default and the honest one: a scanned page is a file
+        # somebody still wants attached, and the error says why it contributes
+        # nothing rather than leaving them to wonder.
+        "reject_unreadable_pdf": False,
+        # --- Semantic search ---------------------------------------------------
+        # Which model turns text into vectors. Empty means none, and none means
+        # the keyword search that has always been here, byte for byte -- which
+        # is what makes this safe to add to an instance that never asked for it.
+        "embedding_model_id": "",
+        # How long a chunk is, in characters, and how much of the previous one
+        # rides along with it. Characters rather than tokens because the count
+        # has to be made without asking the endpoint, and the estimate is the
+        # same four-to-one this codebase already uses.
+        "chunk_chars": 1200,
+        "chunk_overlap": 150,
+        # How many chunks one embedding request carries. Small enough that a
+        # local endpoint is not asked for a megabyte at once.
+        "embed_batch": 16,
+    }
+
+
+def extraction(db: DBSession) -> dict[str, Any]:
+    """Extraction settings, clamped on read for the reason `agents` gives.
+
+    Every floor here is a number that means something bad at zero: a zero-page
+    PDF limit extracts nothing from every PDF and reports success, and a
+    zero-character chunk is an infinite loop in the splitter.
+    """
+    values = get_group(db, EXTRACTION)
+    values["max_upload_mb"] = min(max(int(values.get("max_upload_mb") or 1), 1), 512)
+    values["max_image_edge"] = min(max(int(values.get("max_image_edge") or 1), 128), 8192)
+    values["jpeg_quality"] = min(max(int(values.get("jpeg_quality") or 1), 30), 100)
+    values["max_pdf_pages"] = min(max(int(values.get("max_pdf_pages") or 1), 1), 5000)
+    values["max_extracted_chars"] = min(
+        max(int(values.get("max_extracted_chars") or 1), 1000), 5_000_000
+    )
+    values["orphan_hours"] = min(max(int(values.get("orphan_hours") or 1), 1), 8760)
+    values["chunk_chars"] = min(max(int(values.get("chunk_chars") or 1), 200), 8000)
+    # Bounded *against the chunk*, not absolutely: an overlap at or past the
+    # chunk size means every chunk starts where the last one did, which is a
+    # splitter that never advances.
+    values["chunk_overlap"] = min(
+        max(int(values.get("chunk_overlap") or 0), 0), values["chunk_chars"] // 2
+    )
+    values["embed_batch"] = min(max(int(values.get("embed_batch") or 1), 1), 256)
+    stored = values.get("extra_text_extensions")
+    values["extra_text_extensions"] = (
+        [str(item) for item in stored] if isinstance(stored, list) else []
+    )
+    return values
+
+
 def _branding_defaults() -> dict[str, Any]:
     """Imported inside the call: `services/branding.py` imports this module for
     the group key, so a top-level import back is a cycle."""