Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

A ceiling that was a schedule, and a reply that ended in silence

Browse Source 
Reported: an ordinary chat with a small local model researching a question
well -- six searches, each one informed by the last -- stopped at the round
limit and produced no answer at all. Two separate faults, and the second is
the serious one.

The limit was 5 and it should not have been a working number. It was 1 once,
and the note beside it already said why that was wrong: a count low enough to
be reached by ordinary work is a schedule, not a ceiling, and it overrides the
model's judgement on every turn instead of catching a runaway. Five was the
same mistake with a larger number. It is 0 now -- no ceiling, falling back to
MAX_TOOL_ROUNDS as a runaway backstop, which is the shape `Limits.steps`
already had for an agent chat. What bounds an ordinary chat is the context
window, which is a real limit rather than a guess at how much looking-up a
question deserves. An administrator who wants a ceiling can still set one.

The worse fault: *every* budget ended the reply where it was noticed. That is
survivable for a model that narrates as it works and produces nothing at all
for one that goes straight to tool calls -- an empty bubble with a red line
under it, and everything it had gathered thrown away. `_wrap_up` withdraws the
tools and asks once more instead. What it found is in the transcript either
way; one request turns it into an answer. Same move `plan_submit` makes, and
the reason the loop now runs to `budget + 2`: the round at the budget notices,
the one after it answers. The event stays, because an answer the model chose to
give and one it gave because it ran out of room read identically otherwise.

`_too_big` is the one exception and stays a hard stop. It *is* the finding that
there is no room for another request, so a wrap-up round would be the same
overflow with an upstream error in place of an explanation.

`core.keep_working` was gated on the agent family and is now gated on
`unbounded`, the exact complement of `round_budget` -- so an ordinary chat with
no ceiling is told to work until the job is done rather than being told nothing,
and is never told it has a budget of two hundred, which it would ration.

The regression test asserts the reply is not empty, and fails with `'' ==
'Here is what I found.'` against the old code -- which is exactly what was seen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-04 11:11:57 +02:00
parent a56ee16ee3
commit 9b65adf388
8 changed files with 285 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/generation.py
+87 -31
View File
@@ -450,7 +450,14 @@ async def _run(generation: Generation) -> None:
# tokens and the clock instead.
budget = limits.steps if limits else (chat_rounds or MAX_TOOL_ROUNDS)
for round_number in range(budget + 1):
# Set once a budget has run out, holding the last round open with the
# tools withdrawn so the reply ends in an answer rather than in silence.
# See `_wrap_up`. `budget + 2` rather than `+ 1` is that extra round:
# the iteration at `budget` is where the overrun is noticed, and the one
# after it is where the model gets to say what it found.
wrapping_up = False
for round_number in range(budget + 2):
generation.rounds = round_number + 1
# Recomputed every round, against once before the loop. The request
@@ -467,6 +474,11 @@ async def _run(generation: Generation) -> None:
# branch below on purpose: an ordinary chat with a round budget can
# fill a small window too, and `context_limit` is what decides,
# not what kind of chat it is.
# The one budget that still stops dead rather than asking for a final
# answer. Every other one can afford one more request; this one is
# the finding that there is no room for a request, and a wrap-up
# round would be the same overflow with an upstream error instead of
# an explanation.
if round_number and _too_big(generation):
_gave_up(generation, "with no room left in the context window")
break
@@ -476,18 +488,20 @@ async def _run(generation: Generation) -> None:
# Stop already covers the mid-stream case. Time spent waiting for a
# person is subtracted -- somebody who thinks for ten minutes about
# one command should not thereby spend the whole allowance.
if limits is not None and round_number:
if limits is not None and round_number and not wrapping_up:
spent = (time.monotonic() - started) - generation.waited
ran_out = ""
if spent > limits.wall_seconds:
_gave_up(generation, f"after {spent / 60:.0f} minutes")
break
if generation.output_bytes > limits.output_bytes:
_gave_up(generation, "with too much output to read")
break
written = _written(generation)
if limits.completion_tokens and written > limits.completion_tokens:
_gave_up(generation, f"after writing about {written:,} tokens")
break
ran_out = f"after {spent / 60:.0f} minutes"
elif generation.output_bytes > limits.output_bytes:
ran_out = "with too much output to read"
else:
written = _written(generation)
if limits.completion_tokens and written > limits.completion_tokens:
ran_out = f"after writing about {written:,} tokens"
if ran_out:
offered, payload = _wrap_up(generation, ran_out, payload)
wrapping_up = True
accumulator = tools_service.ToolCallAccumulator()
# Text the model produced in *this* round, needed separately from
# generation.content when echoing the assistant turn back.
@@ -582,29 +596,37 @@ async def _run(generation: Generation) -> None:
# not. The count is of *consecutive* stops.
generation.nudges = 0
if round_number == budget:
# Out of rounds with the model still asking for tools. Recorded
# rather than silently dropped: an answer that stops here needs
# to be explicable.
if round_number >= budget:
# Out of rounds with the model still asking for tools.
#
# `budget`, not `MAX_ROUNDS`. The loop is sized by the budget on
# the line above and the message below has always reported it,
# but the comparison was against the global 3 -- so an agent
# chat allowed forty steps stopped after three and said it had
# taken forty. Two numbers, one of them wrong, in code whose
# whole job is to say what happened.
# The tools are withdrawn and it is asked once more, rather than
# the reply simply ending here. A model that goes straight to
# tool calls has written no prose at all by this point, so
# breaking produced an empty bubble with an error line under it
# -- somebody watching a good piece of research get to its sixth
# search saw the whole thing thrown away. What it has gathered is
# in the transcript either way; one more request turns it into an
# answer.
#
# `budget`, not `MAX_ROUNDS`. The loop is sized by the budget
# above and the message below has always reported it, but the
# comparison was against the global 3 -- so an agent chat allowed
# forty steps stopped after three and said it had taken forty.
# Two numbers, one of them wrong, in code whose whole job is to
# say what happened.
howmany = "one round" if budget == 1 else f"{budget} rounds"
generation.tool_events.append(
{
"name": calls[0]["name"],
"status": "error",
"error": (
f"Stopped after {howmany} of tool calls without an answer."
),
}
offered, payload = _wrap_up(
generation,
f"after {howmany} of tool calls",
payload,
name=calls[0]["name"],
)
generation.touch()
break
if wrapping_up:
# Already asked, and it called a tool anyway -- which it
# cannot do, since none were offered. A backstop, not a path.
break
wrapping_up = True
continue
# Parsed once, here, and shared by everything below: the approval
# card, `policy.decide`, and the runner. See `_arguments_for`.
@@ -930,6 +952,40 @@ def _gave_up(generation, why: str) -> None:
generation.touch()
def _wrap_up(generation, why: str, payload: dict, *, name: str = "budget") -> tuple[list, dict]:
"""A budget has run out. Withdraw the tools and ask for an answer.
Returns the empty tool list and the payload without its `tools` array, so
the next request is one the model can only answer.
Every budget used to end the reply where it was noticed, which is fine for a
model that narrates as it works and produces nothing at all for one that goes
straight to tool calls: an empty bubble with a red line under it, and a good
piece of research thrown away at its sixth search. What it has gathered is
already in the transcript, so one more request without tools turns it into
an answer. That is the same move `plan_submit` makes -- a turn should not end
mid-sentence -- and it is why the loop runs to `budget + 2`.
The event still goes in the transcript. The reader has to be able to tell an
answer the model chose to give from one it gave because it ran out of room,
and those read identically otherwise.
"""
generation.tool_events.append(
{
"name": name,
"kind": "agent",
"status": "error",
"results": [],
"error": (
f"Stopped {why}. What follows is an answer from what had been "
"gathered by then; ask again to carry on."
),
}
)
generation.touch()
return [], {key: value for key, value in payload.items() if key != "tools"}
def _nudge(
generation: Generation,
context,