A time in no particular zone, and a preview missing what it previews
The first audit pass: everything from 0.8.1 to 0.9.8 read as a whole rather than one feature at a time, starting with what a model is actually told. Four of these had shipped as correct. The date line carried a timezone variable that resolves to nothing until somebody chooses one -- so every default account was told times were "in unless they say otherwise", while two comments asserted the line disappeared instead. The prompt preview built its variables without a chat, which is what eleven fragments are gated on, so the whole agent surface was absent from it whatever was ticked. Plan mode was instructed to keep its plan current with a tool that mode withdraws. And knowledge_get returned a document whole where every sibling reader caps and says so, its description promising exactly that. The subagent guidance was wrong in both directions at once: it denied a documented parameter and named seven of twenty-three allowed commands. Both halves are pinned by tests against the real list and the real schema now, because prose and a constant drift the moment one is edited alone. docs/notes/audit-0.9.md carries the findings that are not fixed here, with why -- the ones whose fix would change what a feature does are the user's call, not this pass's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -34,6 +34,21 @@ def test_the_page_lists_every_fragment(client: TestClient, registered):
|
||||
assert f'name="prompt.{fragment.key}"' in page, fragment.key
|
||||
|
||||
|
||||
def test_the_preview_controls_are_inside_what_the_preview_includes(
|
||||
client: TestClient, registered
|
||||
):
|
||||
"""`hx-include="#prompt-form, #preview-controls"` is the whole wiring, so a
|
||||
control placed outside that container is submitted by nothing and changes
|
||||
nothing -- with no error, which is this codebase's recurring failure. Assert
|
||||
the containment rather than the markup of any one field.
|
||||
"""
|
||||
page = client.get("/admin/prompts").text
|
||||
controls = page.split('id="preview-controls"', 1)[1].split("</div>\n </div>", 1)[0]
|
||||
for name in ("preview_model", "preview_bases", "preview_documents",
|
||||
"preview_situation", "preview_mode", "preview_family"):
|
||||
assert f'name="{name}"' in controls, name
|
||||
|
||||
|
||||
def test_the_page_is_refused_to_a_plain_user(client: TestClient, plain_user):
|
||||
assert client.get("/admin/prompts").status_code == 403
|
||||
assert client.post("/admin/prompts", data={}).status_code == 403
|
||||
@@ -135,6 +150,51 @@ def test_the_preview_escapes_what_a_model_wrote(client: TestClient, db, register
|
||||
assert "<img src=x" in body
|
||||
|
||||
|
||||
def test_the_preview_can_reach_the_fragments_that_need_a_chat(client: TestClient, registered):
|
||||
"""Eleven fragments are gated on variables `context_variables` fills only
|
||||
when it is handed a real `Chat`, and the preview hands it `None`. So the
|
||||
entire agent surface, both scheduling fragments and the helper warning were
|
||||
absent from every preview whatever was ticked -- an administrator editing
|
||||
`tool.agent` previewed a system message with `tool.agent` missing from it,
|
||||
and nothing said so. The samples are what close that.
|
||||
"""
|
||||
agent = client.post(
|
||||
"/admin/prompts/preview",
|
||||
data={"preview_family": ["agent"], "preview_mode": "plan"},
|
||||
).text
|
||||
# tool.agent (agent_target), and the mode picked rather than a fixed one.
|
||||
assert "buildbox" in agent
|
||||
assert "Plan** mode" in agent
|
||||
# tool.project_files, context.agent_instructions, context.plan, tool.background
|
||||
assert "pyproject.toml" in agent
|
||||
assert "AGENTS.md" in agent
|
||||
assert "Fix the parser" in agent
|
||||
|
||||
task = client.post(
|
||||
"/admin/prompts/preview", data={"preview_situation": "task", "preview_family": []}
|
||||
).text
|
||||
assert "every weekday at 08:00" in task
|
||||
|
||||
helper = client.post(
|
||||
"/admin/prompts/preview", data={"preview_situation": "helper", "preview_family": []}
|
||||
).text
|
||||
assert "buildbox" not in helper
|
||||
assert helper != task
|
||||
|
||||
|
||||
def test_the_preview_gates_its_samples_exactly_as_a_real_request_would(
|
||||
client: TestClient, registered
|
||||
):
|
||||
"""A preview that admitted a fragment the real request would not is worse
|
||||
than one that omitted it, so the samples follow the same gates: the agent
|
||||
block on the family, the other two on the situation and on no family at all.
|
||||
"""
|
||||
plain = client.post("/admin/prompts/preview", data={"preview_family": ["notes"]}).text
|
||||
assert "buildbox" not in plain
|
||||
assert "AGENTS.md" not in plain
|
||||
assert "every weekday at 08:00" not in plain
|
||||
|
||||
|
||||
def test_the_preview_warns_when_the_cap_would_cut_it_off(client: TestClient, db, registered):
|
||||
settings_store.update(db, {"max_harness_chars": 60}, key=settings_store.PROMPTS)
|
||||
body = client.post("/admin/prompts/preview", data={"preview_family": ["web_search"]}).text
|
||||
|
||||
Reference in New Issue
Block a user