A time in no particular zone, and a preview missing what it previews

The first audit pass: everything from 0.8.1 to 0.9.8 read as a whole rather
than one feature at a time, starting with what a model is actually told.

Four of these had shipped as correct. The date line carried a timezone
variable that resolves to nothing until somebody chooses one -- so every
default account was told times were "in  unless they say otherwise", while
two comments asserted the line disappeared instead. The prompt preview
built its variables without a chat, which is what eleven fragments are
gated on, so the whole agent surface was absent from it whatever was
ticked. Plan mode was instructed to keep its plan current with a tool that
mode withdraws. And knowledge_get returned a document whole where every
sibling reader caps and says so, its description promising exactly that.

The subagent guidance was wrong in both directions at once: it denied a
documented parameter and named seven of twenty-three allowed commands.
Both halves are pinned by tests against the real list and the real schema
now, because prose and a constant drift the moment one is edited alone.

docs/notes/audit-0.9.md carries the findings that are not fixed here, with
why -- the ones whose fix would change what a feature does are the user's
call, not this pass's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-07 09:13:54 +02:00
parent 0ce8026bd2
commit e970f10cca
13 changed files with 521 additions and 51 deletions
+38
View File
@@ -672,3 +672,41 @@ def test_a_helpers_chat_is_sized_by_its_own_numbers(db):
assert limits.steps == 7
assert limits.wall_seconds == 111.0
assert agent_session.resolve(db, parent, _user(db)).limits.steps == 200
# --- What the model is told it may do -----------------------------------------
def test_the_guidance_names_every_command_a_helper_may_run():
"""`SAFE_COMMANDS` holds twenty-three entries; the fragment named seven of
them and said "and nothing else", so a model avoided commands it had --
which costs nothing visible and is therefore never reported. Pinned rather
than proof-read, because the two drift the moment one is edited alone.
"""
from lembas.services import prompts
text = next(f for f in prompts.BUILTIN if f.key == "tool.subagent_agent").default
for entry in subagent_service.SAFE_COMMANDS:
if entry.startswith("file_"):
continue # a tool name, not a command -- see SAFE_COMMANDS' comment
# The word a model would look for: the command itself, or for git the
# subcommand, since the prose shares one "git" across all six.
parts = entry.removesuffix("*").strip().split()
word = parts[1] if parts[0] == "git" else parts[0]
assert word in text, f"{entry!r} is allowed and unmentioned"
def test_the_guidance_does_not_deny_a_parameter_the_tool_offers():
"""It said a helper "reads and reports ... and nothing else, in every mode",
beside a `write` parameter that makes one write files. A model reads the
prose, not the schema, so the parameter was effectively unreachable.
"""
from lembas.services import prompts
text = next(f for f in prompts.BUILTIN if f.key == "tool.subagent_agent").default
assert "write" in text
assert "nothing else, in every mode" not in text
schema = subagent_service.tool_defs()[0]
assert "write" in schema.parameters["properties"]
# And the command list really is fixed whatever `write` says, which is the
# distinction the wording now has to carry.
assert "in every mode" in text