One round for a chat, as many as it takes for an agent

Two different jobs were sharing one number. A plain conversation asking a
question is one round of looking things up and then an answer; the rounds after
that were a small model that had decided searching was the answer searching
until the context ran out, at a full request each. MAX_ROUNDS is 1 now. Several
tools can still be called within that round, which is the thing worth telling
the model.

The trade is real and worth naming: a plain chat can no longer search and then
read one of the results, because reading is a second round. That is what an
agent chat is for.

An agent chat is sized by Limits instead, where steps is now a runaway backstop
and not a working budget. It was 40 and it was reached -- a step count low
enough to be the thing that ends a reply is a count that ends it halfway. What
bounds one now is the wall clock and a new completion-token ceiling, with zero
meaning no ceiling, the same convention index_chars already uses.

That ceiling would have been decorative. generation.completion_tokens is only
populated when the endpoint sends a usage block, and llama.cpp, Ollama and
friends never do; the fallback estimate is computed once, in _run's finally,
long after the loop that needs it. So _written takes the larger of reported and
estimated, and there is a test that runs the whole thing against a stream
reporting no usage at all. A limit that works on OpenAI and silently does
nothing everywhere else is the worst kind: one that looks configured.

core.rounds could not stay one fragment. "You get at most N rounds" is not the
same sentence with a different number in it -- a model told it has a budget
rations it and stops early to report progress, which is exactly the behaviour
that strands a long piece of work. So it splits: core.rounds keeps the
one-round case and gates on a new round_budget variable that _agent_values
blanks, and core.keep_working says the other thing to an agent chat.

A queued message during a one-round reply is now never taken mid-reply -- there
is no work under way to steer -- and falls through to _drain, which gives it a
reply of its own. No code change went with that; it falls out of the guard, and
there is a test so that "it happens to work" and "it is meant to work" stop
looking the same.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-03 11:11:05 +02:00
parent 3345df5b38
commit 7977d4ef25
13 changed files with 322 additions and 30 deletions
+58 -6
View File
@@ -128,6 +128,13 @@ VARIABLES: tuple[Variable, ...] = (
Variable("user_name", "User's name", "The name of the person in the conversation."),
Variable("model_name", "Model", "The display name of the model answering."),
Variable("max_rounds", "Tool rounds", "How many rounds of tool calls one reply may take."),
Variable(
"round_budget",
"Round budget applies",
"Set in an ordinary chat and blank in an agent chat. Nothing renders it; "
"it exists so a fragment can say `requires=('round_budget',)` and appear "
"for one and not the other.",
),
Variable(
"memory_limit",
"Memory length",
@@ -572,19 +579,64 @@ BUILTIN: tuple[Fragment, ...] = (
"permission first."
),
),
Fragment(
key="core.tool_list",
label="What you have",
group=GROUP_CORE,
order=105,
when_tools=True,
variables=("tool_names",),
requires=("tool_names",),
hint="The names of the tools offered on THIS request, which is not the "
"same as the tools that exist -- a chat can narrow them, a model's "
"capabilities can, a permission can. A model that has to discover its "
"own list by calling something and being told it does not exist spends "
"a round finding out, and in an ordinary chat that round is the whole "
"reply. It is also what stops a model hunting for a skill when there "
"are none.",
default=(
"The tools you have on this request are: {{tool_names}}. That is the whole "
"list. Anything not named there does not exist here — calling it costs a "
"round and returns nothing."
),
),
Fragment(
key="core.rounds",
label="The round budget",
group=GROUP_CORE,
order=110,
when_tools=True,
variables=("max_rounds",),
hint="A model that plans six searches gets cut off after three. Better it "
"knows the budget than discovers it.",
requires=("round_budget",),
hint="An ordinary chat only. It gets ONE round of tool calls, and the "
"thing worth saying about one round is 'ask for everything at once'"
"which is different in kind from what is true of an agent chat's two "
"hundred, not a different number in the same sentence. So this is "
"gated on `round_budget`, which `_agent_values` blanks, and the agent "
"case is its own fragment below.",
default=(
"You get at most {{max_rounds}} rounds of tool calls before you have to "
"answer with what you have. Several tools can be called in one round. Plan "
"within that budget: two careful searches beat six that run out halfway."
"You get one round of tool calls, and then you have to answer with what "
"came back. Ask for everything you need at once — several tools can be "
"called in the same round. If what comes back is not enough, say what you "
"would look up next rather than answering as though it were."
),
),
Fragment(
key="core.keep_working",
label="Working until it is done",
group=GROUP_CORE,
order=111,
families=("agent",),
hint="An agent chat only, and the counterpart to the round budget above. "
"A model told it has a budget rations it and stops early to report "
"progress; the step count here is a runaway backstop, not an "
"allowance, and saying so is what makes a long piece of work run.",
default=(
"Keep working until the task is actually done. You are not rationing a "
"round budget: call tools as many times as the work needs, one step "
"informing the next. What ends a reply is finishing it, being stopped, or "
"running past the time and output an administrator allowed — and if that "
"happens you are told so and can be asked to carry on. Do not stop halfway "
"to report progress and wait to be told to continue."
),
),
Fragment(