A time in no particular zone, and a preview missing what it previews
The first audit pass: everything from 0.8.1 to 0.9.8 read as a whole rather than one feature at a time, starting with what a model is actually told. Four of these had shipped as correct. The date line carried a timezone variable that resolves to nothing until somebody chooses one -- so every default account was told times were "in unless they say otherwise", while two comments asserted the line disappeared instead. The prompt preview built its variables without a chat, which is what eleven fragments are gated on, so the whole agent surface was absent from it whatever was ticked. Plan mode was instructed to keep its plan current with a tool that mode withdraws. And knowledge_get returned a document whole where every sibling reader caps and says so, its description promising exactly that. The subagent guidance was wrong in both directions at once: it denied a documented parameter and named seven of twenty-three allowed commands. Both halves are pinned by tests against the real list and the real schema now, because prose and a constant drift the moment one is edited alone. docs/notes/audit-0.9.md carries the findings that are not fixed here, with why -- the ones whose fix would change what a feature does are the user's call, not this pass's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -672,3 +672,41 @@ def test_a_helpers_chat_is_sized_by_its_own_numbers(db):
|
||||
assert limits.steps == 7
|
||||
assert limits.wall_seconds == 111.0
|
||||
assert agent_session.resolve(db, parent, _user(db)).limits.steps == 200
|
||||
|
||||
|
||||
# --- What the model is told it may do -----------------------------------------
|
||||
def test_the_guidance_names_every_command_a_helper_may_run():
|
||||
"""`SAFE_COMMANDS` holds twenty-three entries; the fragment named seven of
|
||||
them and said "and nothing else", so a model avoided commands it had --
|
||||
which costs nothing visible and is therefore never reported. Pinned rather
|
||||
than proof-read, because the two drift the moment one is edited alone.
|
||||
"""
|
||||
from lembas.services import prompts
|
||||
|
||||
text = next(f for f in prompts.BUILTIN if f.key == "tool.subagent_agent").default
|
||||
for entry in subagent_service.SAFE_COMMANDS:
|
||||
if entry.startswith("file_"):
|
||||
continue # a tool name, not a command -- see SAFE_COMMANDS' comment
|
||||
# The word a model would look for: the command itself, or for git the
|
||||
# subcommand, since the prose shares one "git" across all six.
|
||||
parts = entry.removesuffix("*").strip().split()
|
||||
word = parts[1] if parts[0] == "git" else parts[0]
|
||||
assert word in text, f"{entry!r} is allowed and unmentioned"
|
||||
|
||||
|
||||
def test_the_guidance_does_not_deny_a_parameter_the_tool_offers():
|
||||
"""It said a helper "reads and reports ... and nothing else, in every mode",
|
||||
beside a `write` parameter that makes one write files. A model reads the
|
||||
prose, not the schema, so the parameter was effectively unreachable.
|
||||
"""
|
||||
from lembas.services import prompts
|
||||
|
||||
text = next(f for f in prompts.BUILTIN if f.key == "tool.subagent_agent").default
|
||||
assert "write" in text
|
||||
assert "nothing else, in every mode" not in text
|
||||
|
||||
schema = subagent_service.tool_defs()[0]
|
||||
assert "write" in schema.parameters["properties"]
|
||||
# And the command list really is fixed whatever `write` says, which is the
|
||||
# distinction the wording now has to carry.
|
||||
assert "in every mode" in text
|
||||
|
||||
Reference in New Issue
Block a user