Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Compaction: a button, and automatically when the window fills

Browse Source 
A long conversation eventually just stops working. Compaction summarises the
earlier turns and sends the summary in their place.

The messages are kept. They stay in the transcript behind a collapsed
divider and simply stop being part of the request, which is what makes the
button safe to press and automatic compaction safe to have at all: a summary
that came out badly is a bad turn, not a lost conversation.

Stored on the Chat, not as a synthetic Message. A synthetic row needs a
role -- `system` breaks the one-system-message rule the moment build_messages
emits it beside the harness, and user/assistant makes it a turn people can
edit, regenerate from and copy, indistinguishable from a real one in all
four places a bubble is rendered. Worse, "editing rewinds, it does not
branch" would silently delete it and leave no marker that compaction had
happened at all.

The summary goes out as a user turn and an assistant turn, not one. A
leading assistant breaks templates requiring the first non-system message to
be user; a lone leading user produces user, user whenever the kept history
starts on a user turn -- which it always does, because the cutoff lands on a
finished reply.

compacted_through_id is a plain id rather than a foreign key: migrations.py
compiles only the column type, so a REFERENCES clause would exist on a fresh
database and not on an upgraded one, and a constraint half the fleet has is
worse than none. cutoff_message validates it on every read instead, and a
rewind past the boundary clears it.

Compacting again summarises only the delta, with the previous summary
supplied to be subsumed. Re-summarising the whole chat each time grows
quadratically and eventually exceeds the window it is protecting.

Automatically at the top of _run, not in post_message: that route's contract
is to return immediately and leave the slow part to a resumable connection,
and it also means build_request is called once, after compaction, with no
second assembly path. The trigger is the last reply's recorded usage plus an
estimate of the new turn -- retrospective because true prompt_tokens are only
knowable after a response, plus the delta because otherwise fifty thousand
characters pasted into the composer overflow a window that read 90% last
turn. It never fires when the context length is unknown. It does fire on
estimated counts, which is safe here precisely because nothing is lost.

_maybe_compact never raises: a failure logs and sends the uncompacted
request. A `status` event says "Summarising earlier messages…" in the
meantime, because a silent multi-second pause before the first token is what
a hang looks like.

The wording is three fragments under Admin - Prompts. Clearing task.compact
turns compaction off entirely.

Also adds compaction.moment(): SQLite does not store the offset, so a row
loaded from disk is naive while one in the session's identity map keeps its
tzinfo, and comparing the two raises. Every comparison here is between
exactly those.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 01:02:02 +02:00
parent 314cc946d7
commit 17f3fa1946
17 changed files with 1068 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+49 -3
View File
@@ -19,7 +19,7 @@ lembas info # paths + counts, useful when confused
lembas secret-key # generate LEMBAS_SECRET_KEY
lembas create-admin # create or promote an admin
pytest # 488 tests, ~29s
pytest # 590 tests, ~35s
# PLAN.md tracks what is and is not built
ruff check . # lint (line length 100)
python scripts/build_artwork.py # regenerate artwork (SVG + PWA icons;
@@ -81,6 +81,7 @@ src/lembas/
admin_audio.py speech-to-text and text-to-speech endpoints
admin_search.py web search provider and credentials
admin_prompts.py the prompt fragment editor and its preview
admin_suggestions.py the cards offered on the new-chat screen
audio.py transcribe, speak, voice discovery
library.py knowledge, notes, skills pages; memory CRUD
files.py upload, serve, remove attachments
@@ -99,6 +100,10 @@ src/lembas/
fetch.py URL retrieval, HTML to text, the SSRF guard
sharing.py one visibility rule for every library store
prompts.py every injected prompt fragment, and {{variables}}
metrics.py tokens, context percentage and tokens/second
tokens.py the chars/4 estimate, for endpoints that report none
compaction.py summarising the earlier turns of a long chat
suggestions.py new-chat starting points, seeded once
harness.py the operational prompt built from what a model has
tools.py tool registry, schemas, streamed-call reassembly
chat.py request building, endpoint resolution, titles
@@ -213,8 +218,25 @@ NOT stop the reply -- that was the old behaviour and it cut answers off when
the reader navigated away. Any route that creates an assistant placeholder must
also call `generation.ensure()`.
**Stream frames carry whole blocks, not deltas.** Both `render` and `reasoning`
send the complete text each time. That is what makes reattaching mid-reply
**`ensure` attaches, `restart` replaces.** The registry is keyed on message id
and finished generations linger `KEEP_FINISHED` so a follower arriving at the
last moment still gets the final frames. `ensure` is idempotent because a page
load finding an unfinished reply must attach rather than start a second one.
Regeneration is the only caller that reuses a `Message` row, and therefore the
only one for which idempotence is wrong -- it got the finished generation back,
made no request, and left the browser reconnecting to a stream with nothing to
say. It calls `restart`. `_persist` refuses to write when another generation
owns the message, because a cancelled predecessor's `finally:` still runs.
**The row is written before `done` is set.** `_follow` breaks out the instant it
sees that flag and re-renders the bubble from the database, so the row has to be
authoritative first. The other order silently showed the previous turn's stored
metrics.
**Stream frames carry whole blocks, not deltas.** `render`, `reasoning`,
`metrics` and `status` all send the complete value each time, and every one of
them is swapped with `innerHTML`. `reasoning` used `beforeend` and so repeated
everything already shown on every frame. That is what makes reattaching mid-reply
work: a follower arriving late has no earlier fragments to append to. It also
means Markdown is re-rendered whole, which is required anyway -- a list or code
fence is only correct once its context exists.
@@ -490,5 +512,29 @@ flag, not a new code path. Its guidance is the same shape: a
harness, on the admin page and in the preview without touching the assembler,
the save handler or a template.
**Unknown is not zero.** `Model.context_length` of 0 means nobody has said how
big the window is, which is different from "small". The context percentage is
omitted rather than computed, and automatic compaction never fires. Token counts
fall back to `services/tokens.py` -- four characters to a token -- and anything
derived from an estimate is shown with a `~`. Compaction *does* act on an
estimate, because a premature compaction costs one turn of answer quality rather
than data: the messages are kept.
**Compaction hides turns, it does not delete them.** `Chat.compact_summary` plus
`compacted_through_id` say how far it reached; the messages stay in the
transcript behind a `<details>` divider and simply stop being part of the
request. The summary is carried by a **user turn and an assistant turn**, not
one: a leading `assistant` breaks templates that require the first non-system
message to be `user`, and a lone leading `user` produces `user, user` whenever
the kept history starts on a user turn -- which it always does, because the
cutoff lands on a finished reply. `compacted_through_id` is a plain id, not a
foreign key, because `migrations.py` compiles only the column type and a
`REFERENCES` clause would exist on a fresh database and not on an upgraded one;
`compaction.cutoff_message` validates it on every read instead.
**Compare message timestamps through `compaction.moment()`.** SQLite does not
store the offset, so a row loaded from disk is naive while one still in the
session's identity map keeps its tzinfo. Comparing the two raises.
No OCR: a scanned PDF is stored with an explanatory `extraction_error` rather
than silently contributing nothing.