Compaction: a button, and automatically when the window fills

A long conversation eventually just stops working. Compaction summarises the
earlier turns and sends the summary in their place.

The messages are kept. They stay in the transcript behind a collapsed
divider and simply stop being part of the request, which is what makes the
button safe to press and automatic compaction safe to have at all: a summary
that came out badly is a bad turn, not a lost conversation.

Stored on the Chat, not as a synthetic Message. A synthetic row needs a
role -- `system` breaks the one-system-message rule the moment build_messages
emits it beside the harness, and user/assistant makes it a turn people can
edit, regenerate from and copy, indistinguishable from a real one in all
four places a bubble is rendered. Worse, "editing rewinds, it does not
branch" would silently delete it and leave no marker that compaction had
happened at all.

The summary goes out as a user turn and an assistant turn, not one. A
leading assistant breaks templates requiring the first non-system message to
be user; a lone leading user produces user, user whenever the kept history
starts on a user turn -- which it always does, because the cutoff lands on a
finished reply.

compacted_through_id is a plain id rather than a foreign key: migrations.py
compiles only the column type, so a REFERENCES clause would exist on a fresh
database and not on an upgraded one, and a constraint half the fleet has is
worse than none. cutoff_message validates it on every read instead, and a
rewind past the boundary clears it.

Compacting again summarises only the delta, with the previous summary
supplied to be subsumed. Re-summarising the whole chat each time grows
quadratically and eventually exceeds the window it is protecting.

Automatically at the top of _run, not in post_message: that route's contract
is to return immediately and leave the slow part to a resumable connection,
and it also means build_request is called once, after compaction, with no
second assembly path. The trigger is the last reply's recorded usage plus an
estimate of the new turn -- retrospective because true prompt_tokens are only
knowable after a response, plus the delta because otherwise fifty thousand
characters pasted into the composer overflow a window that read 90% last
turn. It never fires when the context length is unknown. It does fire on
estimated counts, which is safe here precisely because nothing is lost.

_maybe_compact never raises: a failure logs and sends the uncompacted
request. A `status` event says "Summarising earlier messages…" in the
meantime, because a silent multi-second pause before the first token is what
a hang looks like.

The wording is three fragments under Admin - Prompts. Clearing task.compact
turns compaction off entirely.

Also adds compaction.moment(): SQLite does not store the offset, so a row
loaded from disk is naive while one in the session's identity map keeps its
tzinfo, and comparing the two raises. Every comparison here is between
exactly those.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 01:02:02 +02:00
parent aa0bbe524a
commit 3b1632069c
15 changed files with 1009 additions and 9 deletions
+81
View File
@@ -156,6 +156,16 @@ VARIABLES: tuple[Variable, ...] = (
),
Variable("question", "Question", "The first message. Chat title task only."),
Variable("answer", "Answer", "The first reply. Chat title task only."),
Variable(
"transcript",
"Transcript",
"The turns being summarised, oldest first. Compaction task only.",
),
Variable(
"previous_summary",
"Earlier summary",
"The summary from a previous compaction, if there was one. Compaction task only.",
),
)
VARIABLE_NAMES = frozenset(variable.name for variable in VARIABLES)
@@ -764,6 +774,77 @@ BUILTIN: tuple[Fragment, ...] = (
"Assistant: {{answer}}"
),
),
Fragment(
key="task.compact",
label="Compaction summary",
group=GROUP_TASKS,
order=410,
variables=("transcript", "previous_summary"),
hint="A separate one-message request, not part of any chat. Clear it to "
"turn compaction off entirely: the button says so and nothing is "
"summarised automatically.",
default=(
"Summarise the conversation below so it can be carried forward after the "
"earlier turns are dropped from your context. This is a working record, "
"not a report for a reader.\n"
"\n"
"Keep, under these headings and in this order:\n"
"\n"
"## What we are doing\n"
"The goal, and where we have got to.\n"
"\n"
"## Decisions\n"
"Anything settled, and why. A decision without its reason gets argued "
"again.\n"
"\n"
"## Facts established\n"
"Names, numbers, versions, file paths, URLs and identifiers, copied "
"exactly. Do not round them, paraphrase them or reconstruct one from "
"memory — if it is not in the transcript, leave it out.\n"
"\n"
"## Open threads\n"
"What is unfinished, and what was about to happen next.\n"
"\n"
"Leave out pleasantries, retracted ideas and anything already superseded. "
"Do not answer the conversation: you are recording it. Write in the "
"language of the conversation, and stay under 500 words.\n"
"\n"
"{{previous_summary}}\n"
"\n"
"## Transcript\n"
"\n"
"{{transcript}}"
),
),
Fragment(
key="task.compact_lead",
label="How a summary is introduced",
group=GROUP_TASKS,
order=420,
hint="Sits in front of the summary, in the turn that replaces the "
"messages no longer being sent. Without it a model reads the summary as "
"something the person has just typed.",
default=(
"Here is a summary of the earlier part of this conversation. Those "
"messages are no longer in your context. Treat this summary as an "
"accurate record of them and rely on it rather than on what you can no "
"longer see; if it does not cover something you need, say so instead of "
"filling the gap."
),
),
Fragment(
key="task.compact_ack",
label="The model's acknowledgement",
group=GROUP_TASKS,
order=430,
hint="One assistant turn after the summary, so the conversation still "
"alternates user, assistant, user. Several chat templates reject a "
"history that does not.",
default=(
"Understood. I have the summary of the earlier turns and will carry on "
"from there."
),
),
)
register_source(_builtin_source)