One round for a chat, as many as it takes for an agent

Two different jobs were sharing one number. A plain conversation asking a
question is one round of looking things up and then an answer; the rounds after
that were a small model that had decided searching was the answer searching
until the context ran out, at a full request each. MAX_ROUNDS is 1 now. Several
tools can still be called within that round, which is the thing worth telling
the model.

The trade is real and worth naming: a plain chat can no longer search and then
read one of the results, because reading is a second round. That is what an
agent chat is for.

An agent chat is sized by Limits instead, where steps is now a runaway backstop
and not a working budget. It was 40 and it was reached -- a step count low
enough to be the thing that ends a reply is a count that ends it halfway. What
bounds one now is the wall clock and a new completion-token ceiling, with zero
meaning no ceiling, the same convention index_chars already uses.

That ceiling would have been decorative. generation.completion_tokens is only
populated when the endpoint sends a usage block, and llama.cpp, Ollama and
friends never do; the fallback estimate is computed once, in _run's finally,
long after the loop that needs it. So _written takes the larger of reported and
estimated, and there is a test that runs the whole thing against a stream
reporting no usage at all. A limit that works on OpenAI and silently does
nothing everywhere else is the worst kind: one that looks configured.

core.rounds could not stay one fragment. "You get at most N rounds" is not the
same sentence with a different number in it -- a model told it has a budget
rations it and stops early to report progress, which is exactly the behaviour
that strands a long piece of work. So it splits: core.rounds keeps the
one-round case and gates on a new round_budget variable that _agent_values
blanks, and core.keep_working says the other thing to an agent chat.

A queued message during a one-round reply is now never taken mid-reply -- there
is no work under way to steer -- and falls through to _drain, which gives it a
reply of its own. No code change went with that; it falls out of the guard, and
there is a test so that "it happens to work" and "it is meant to work" stop
looking the same.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-03 11:11:05 +02:00
parent 3345df5b38
commit 7977d4ef25
13 changed files with 322 additions and 30 deletions
+82
View File
@@ -741,6 +741,88 @@ async def test_an_agent_chat_gets_the_rounds_it_was_promised(db, user_id, machin
assert "after 5 rounds" in generation.tool_events[-1]["error"]
async def test_a_reply_stops_when_it_has_written_too_much(db, user_id, machine, monkeypatch):
"""The bound that is meant to end a long piece of work.
Steps are a runaway backstop now (200), so something has to say when enough
has been written. Asserted on the loop, not on the wording: the count of
requests must be far short of the step budget.
"""
settings_store.update(
db, {"max_steps": 200, "max_completion_tokens": 40}, key=settings_store.AGENTS
)
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
message_id = _pending_reply(db, chat)
payloads: list[dict] = []
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream(
[[_text("x" * 400), _chunk("file_list", '{"path": "."}')]],
payloads,
),
)
async def _no_title(*_args, **_kwargs):
return ""
monkeypatch.setattr("lembas.services.chat.generate_title", _no_title)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await generation_service._run(generation)
assert len(payloads) < 5, "it should have stopped long before the step backstop"
assert "tokens" in generation.tool_events[-1]["error"]
async def test_the_token_ceiling_fires_on_an_endpoint_that_reports_no_usage(
db, user_id, machine, monkeypatch
):
"""The half that would otherwise be silently broken.
`generation.completion_tokens` is only populated when the endpoint sends a
usage block, and llama.cpp, Ollama and friends never do -- the fallback
estimate is computed once, in `_run`'s `finally:`, long after the loop that
needs it. A ceiling reading only the reported figure would work on OpenAI
and do nothing at all everywhere else. The stub above sends no usage, so
this asserts the estimate path directly.
"""
settings_store.update(
db, {"max_steps": 200, "max_completion_tokens": 40}, key=settings_store.AGENTS
)
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
message_id = _pending_reply(db, chat)
payloads: list[dict] = []
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream([[_text("y" * 400), _chunk("file_list", '{"path": "."}')]], payloads),
)
async def _no_title(*_args, **_kwargs):
return ""
monkeypatch.setattr("lembas.services.chat.generate_title", _no_title)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await generation_service._run(generation)
assert not any("usage" in str(p) for p in payloads), "the stub reports no usage"
assert "tokens" in generation.tool_events[-1]["error"]
async def test_a_zero_ceiling_means_no_ceiling(db, user_id, machine, monkeypatch):
"""Zero is how an administrator says "no limit", the same as index_chars.
Read with `or 0` on the wrong side it would silently become 200_000."""
settings_store.update(db, {"max_completion_tokens": 0}, key=settings_store.AGENTS)
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
user = db.get(User, user_id)
context = session.resolve(db, chat, user)
assert context.limits.completion_tokens == 0
# --- Interjecting while it works --------------------------------------------------
async def test_a_queued_prompt_is_taken_in_between_rounds(db, user_id, machine, monkeypatch):
"""The point of queueing in an agent chat: steering work already under way.