A time in no particular zone, and a preview missing what it previews

The first audit pass: everything from 0.8.1 to 0.9.8 read as a whole rather
than one feature at a time, starting with what a model is actually told.

Four of these had shipped as correct. The date line carried a timezone
variable that resolves to nothing until somebody chooses one -- so every
default account was told times were "in  unless they say otherwise", while
two comments asserted the line disappeared instead. The prompt preview
built its variables without a chat, which is what eleven fragments are
gated on, so the whole agent surface was absent from it whatever was
ticked. Plan mode was instructed to keep its plan current with a tool that
mode withdraws. And knowledge_get returned a document whole where every
sibling reader caps and says so, its description promising exactly that.

The subagent guidance was wrong in both directions at once: it denied a
documented parameter and named seven of twenty-three allowed commands.
Both halves are pinned by tests against the real list and the real schema
now, because prose and a constant drift the moment one is edited alone.

docs/notes/audit-0.9.md carries the findings that are not fixed here, with
why -- the ones whose fix would change what a feature does are the user's
call, not this pass's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-07 09:13:54 +02:00
parent 0ce8026bd2
commit e970f10cca
13 changed files with 521 additions and 51 deletions
+60
View File
@@ -34,6 +34,21 @@ def test_the_page_lists_every_fragment(client: TestClient, registered):
assert f'name="prompt.{fragment.key}"' in page, fragment.key
def test_the_preview_controls_are_inside_what_the_preview_includes(
client: TestClient, registered
):
"""`hx-include="#prompt-form, #preview-controls"` is the whole wiring, so a
control placed outside that container is submitted by nothing and changes
nothing -- with no error, which is this codebase's recurring failure. Assert
the containment rather than the markup of any one field.
"""
page = client.get("/admin/prompts").text
controls = page.split('id="preview-controls"', 1)[1].split("</div>\n </div>", 1)[0]
for name in ("preview_model", "preview_bases", "preview_documents",
"preview_situation", "preview_mode", "preview_family"):
assert f'name="{name}"' in controls, name
def test_the_page_is_refused_to_a_plain_user(client: TestClient, plain_user):
assert client.get("/admin/prompts").status_code == 403
assert client.post("/admin/prompts", data={}).status_code == 403
@@ -135,6 +150,51 @@ def test_the_preview_escapes_what_a_model_wrote(client: TestClient, db, register
assert "&lt;img src=x" in body
def test_the_preview_can_reach_the_fragments_that_need_a_chat(client: TestClient, registered):
"""Eleven fragments are gated on variables `context_variables` fills only
when it is handed a real `Chat`, and the preview hands it `None`. So the
entire agent surface, both scheduling fragments and the helper warning were
absent from every preview whatever was ticked -- an administrator editing
`tool.agent` previewed a system message with `tool.agent` missing from it,
and nothing said so. The samples are what close that.
"""
agent = client.post(
"/admin/prompts/preview",
data={"preview_family": ["agent"], "preview_mode": "plan"},
).text
# tool.agent (agent_target), and the mode picked rather than a fixed one.
assert "buildbox" in agent
assert "Plan** mode" in agent
# tool.project_files, context.agent_instructions, context.plan, tool.background
assert "pyproject.toml" in agent
assert "AGENTS.md" in agent
assert "Fix the parser" in agent
task = client.post(
"/admin/prompts/preview", data={"preview_situation": "task", "preview_family": []}
).text
assert "every weekday at 08:00" in task
helper = client.post(
"/admin/prompts/preview", data={"preview_situation": "helper", "preview_family": []}
).text
assert "buildbox" not in helper
assert helper != task
def test_the_preview_gates_its_samples_exactly_as_a_real_request_would(
client: TestClient, registered
):
"""A preview that admitted a fragment the real request would not is worse
than one that omitted it, so the samples follow the same gates: the agent
block on the family, the other two on the situation and on no family at all.
"""
plain = client.post("/admin/prompts/preview", data={"preview_family": ["notes"]}).text
assert "buildbox" not in plain
assert "AGENTS.md" not in plain
assert "every weekday at 08:00" not in plain
def test_the_preview_warns_when_the_cap_would_cut_it_off(client: TestClient, db, registered):
settings_store.update(db, {"max_harness_chars": 60}, key=settings_store.PROMPTS)
body = client.post("/admin/prompts/preview", data={"preview_family": ["web_search"]}).text