Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

A time in no particular zone, and a preview missing what it previews

Browse Source 
The first audit pass: everything from 0.8.1 to 0.9.8 read as a whole rather
than one feature at a time, starting with what a model is actually told.

Four of these had shipped as correct. The date line carried a timezone
variable that resolves to nothing until somebody chooses one -- so every
default account was told times were "in  unless they say otherwise", while
two comments asserted the line disappeared instead. The prompt preview
built its variables without a chat, which is what eleven fragments are
gated on, so the whole agent surface was absent from it whatever was
ticked. Plan mode was instructed to keep its plan current with a tool that
mode withdraws. And knowledge_get returned a document whole where every
sibling reader caps and says so, its description promising exactly that.

The subagent guidance was wrong in both directions at once: it denied a
documented parameter and named seven of twenty-three allowed commands.
Both halves are pinned by tests against the real list and the real schema
now, because prose and a constant drift the moment one is edited alone.

docs/notes/audit-0.9.md carries the findings that are not fixed here, with
why -- the ones whose fix would change what a feature does are the user's
call, not this pass's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-07 09:13:54 +02:00
parent 0ce8026bd2
commit e970f10cca
13 changed files with 521 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/harness.py
+29 -4
View File
@@ -188,10 +188,23 @@ def context_variables(
"today": stamp.strftime("%A %-d %B %Y"),
"now": stamp.strftime("%A %-d %B %Y, %H:%M (UTC%z)"),
# Named so a model working out a schedule can say which zone it meant,
# and so `core.today` can carry it without a second fragment. Empty when
# nobody has chosen one, which drops the line rather than printing the
# server's zone as though it were a decision.
"timezone": clock.name_for(user),
# and so `core.today` can carry it without a second fragment.
#
# The fallback is load-bearing and used to be absent. `name_for` returns
# "" for anybody who has never chosen a zone -- the default state of
# every account -- and the comment here claimed that dropped the line
# rather than announcing the server's zone as a decision. It did not:
# `substitute` drops a line only when it is *blank* after expansion, and
# this variable sits inside a sentence, so every such request shipped
# "- Times the person gives you are in unless they say otherwise."
#
# Naming the server's zone was never the thing being avoided anyway.
# `stamp` is `clock.now_for(user)`, which already falls back to it, so
# `{{today}}` and `{{now}}` are *already* in that zone and `{{now}}`
# already prints its offset. Withholding the label from a value the
# model has been given is not restraint, it is a hole. This is the
# fallback `schedule/compile.py` has always had, for the same reason.
"timezone": clock.name_for(user) or str(clock.server_zone()),
"instance_name": branding.for_db(db).name,
"user_name": (user.name or "") if user is not None else "",
"model_name": "",
@@ -241,6 +254,7 @@ def context_variables(
"agent_instructions": "",
"agent_instructions_file": "",
"plan": "",
"plan_editable": "",
# Empty everywhere but a scheduled task's own chat, which is what makes
# it the gate on `core.unattended` as well as the content of
# `context.schedule`. Two fragments, one variable, and no way for the
@@ -346,6 +360,17 @@ def _agent_values(db: DBSession, chat, user) -> dict[str, str]:
# `agent_session.resolve`. A plan the model cannot see is a plan it
# cannot keep current, which is the whole of why this is here.
"plan": plans_service.render_block(context.plan),
# Whether `plan_update` is actually in this request, which is not the
# same question as whether there is a plan. `agent/tools.py` drops it in
# Plan mode -- that mode ends with `plan_submit` instead -- so gating its
# guidance on `plan` alone told a model in Plan mode to "keep it current
# with plan_update as you go" about a tool that was not there, directly
# under `core.tool_list` saying anything unnamed does not exist. The
# fragment's own hint claimed the two coincided. They do not, and this
# is the variable that makes them.
"plan_editable": (
plans_service.render_block(context.plan) if context.mode != policy.MODE_PLAN else ""
),
**_project_instructions(db, chat, context, settings_store),
}
src/lembas/services/prompts.py
+47 -28
View File
@@ -219,8 +219,14 @@ VARIABLES: tuple[Variable, ...] = (
"The current plan",
"The plan this agent chat is working to, with its ids, finished phases "
"collapsed and the active one shown in full. Empty when there is none, "
"which is what keeps both the plan section and plan_update's guidance "
"out of every chat that is not carrying one.",
"which is what keeps the plan section out of every chat not carrying one.",
),
Variable(
"plan_editable",
"The plan, when it can be changed",
"The same text as {{plan}}, but blank in Plan mode -- where plan_update "
"is withdrawn and the turn ends with plan_submit instead. Gates that "
"tool's guidance, so a mode without the tool is not told to use it.",
),
Variable(
"agent_instructions",
@@ -614,9 +620,10 @@ BUILTIN: tuple[Fragment, ...] = (
order=20,
variables=("today", "timezone"),
hint="A model has no clock. Without this it cannot tell whether what it "
"recalls is current, and will not think to check. The timezone line "
"carries its own variable, so it disappears on an instance where nobody "
"has chosen one rather than announcing the server's as a decision.",
"recalls is current, and will not think to check. {{timezone}} is the "
"reader's own zone, or the server's where they have not chosen one -- "
"the same zone the date above is already stated in, so it names a value "
"rather than making a decision.",
default=(
"Today is {{today}}. Your training data stops well before this, so treat "
"anything time-sensitive as something to check rather than something you "
@@ -1314,20 +1321,28 @@ BUILTIN: tuple[Fragment, ...] = (
families=("subagent",),
requires=("agent_target",),
hint="The agent-chat half, gated on `agent_target` so it appears only "
"where there is a machine. What it has to say is what a helper cannot "
"do there, because the failure otherwise is a model planning a whole "
"phase around a helper that will refuse every step of it — a helper "
"reads and may run a short list of read-only commands, and nothing "
"else, whatever mode this chat is in.",
"where there is a machine. What it has to say is where the edge is, "
"because the failure otherwise is a model planning a whole phase around "
"a helper that will refuse every step of it. Two edges, and they are "
"different: **commands** are pinned to a read-only list in every mode, "
"since an unattended chat cannot approve anything; **files** may be "
"written, but only by a helper asked for with write and only from Edit "
"or Auto. An earlier version denied the second outright — which is a "
"documented parameter of the tool beside it — and named seven of the "
"twenty-three allowed commands, so a model avoided commands it had.",
default=(
"- A helper on this machine reads and reports. It can list and read files "
"and run the ordinary read-only commands — ls, cat, grep, find, git status, "
"git log, git diff — and nothing else, in every mode, because there is "
"nobody there to approve anything. Send one to find out where something "
"lives, to read a subsystem and describe it, or to check whether a pattern "
"holds across a tree; make the changes yourself once it reports.\n"
"- Do not send one to build, test, install or run anything: it will be "
"refused a step in and come back having done nothing.\n"
"and run the read-only commands — ls, pwd, cat, head, tail, wc, file, stat, "
"du, df, tree, find, grep, rg, and git status, log, show, diff, branch and "
"remote. Send one to find out where something lives, to read a subsystem "
"and describe it, or to check whether a pattern holds across a tree.\n"
"- Do not send one to build, test, install or run anything. That list is "
"the whole of what it may run, in every mode, because there is nobody "
"there to approve anything else — a helper asked to run the tests is "
"refused a step in and comes back having done nothing.\n"
"- A helper asked for with write can also write and edit files, and only "
"from Edit or Auto mode. Use it for a mechanical change across many files "
"that you have already decided on; keep the deciding for yourself.\n"
"- Ask for what you want back, not for a summary. “The three files that "
"define X and what each does” is usable; “look into X” comes back as prose "
"you have to read the codebase to check."
@@ -1539,12 +1554,14 @@ BUILTIN: tuple[Fragment, ...] = (
group=GROUP_TOOLS,
order=255,
families=("agent",),
requires=("plan",),
hint="Appears once a plan exists, which is also when plan_update is "
"offered. It is about doing the bookkeeping as the work goes rather "
"than at the end -- a plan updated only at the end is a report, and "
"the point of it is being able to see where things are while they are "
"still moving.",
requires=("plan_editable",),
hint="Appears when plan_update is actually offered: a plan exists and "
"this is not Plan mode, which withdraws it in favour of plan_submit. "
"Gated on {{plan}} it appeared in Plan mode too, telling a model to "
"use a tool it did not have. It is about doing the bookkeeping as the "
"work goes rather than at the end -- a plan updated only at the end is "
"a report, and the point of it is being able to see where things are "
"while they are still moving.",
default=(
"- There is a plan for this work, set out below. Keep it current with "
"plan_update as you go rather than at the end: mark a task “doing” when "
@@ -1567,16 +1584,18 @@ BUILTIN: tuple[Fragment, ...] = (
requires=("plan",),
variables=("plan",),
hint="The plan as it stands, including what has already been ticked "
"off. A plan the model cannot see is a plan it cannot update, which "
"is what the whole of plan_update depends on. The ids are shown "
"because they are what plan_update takes.",
"off. A plan the model cannot see is a plan it cannot work to. Shown in "
"every mode including Plan, where the tool for changing it is withdrawn "
"-- so this says what the plan *is* and leaves how to change it to "
"`tool.plan_update`, which is gated on that tool actually being there. "
"The ids are shown because they are what plan_update takes.",
default=(
"### The current plan\n"
"\n"
"{{plan}}\n"
"\n"
"This is the plan as it stands now. Change it with plan_update rather "
"than restating it in your answer, and quote the ids above."
"This is the plan as it stands now. Work to it, and quote the ids above "
"rather than restating the plan in your answer."
),
),
Fragment(
src/lembas/services/subagent.py
+11 -8
View File
@@ -516,14 +516,17 @@ def tool_defs() -> list[ToolDef]:
family=FAMILY_SUBAGENT,
description=(
"Hand one self-contained piece of work to a helper — a second "
"model with the same tools that works on its own and gives you "
"its answer. Use it to cover several independent areas at once: "
"call it several times in one turn and each runs in parallel. "
"It cannot ask you or the reader anything, cannot delegate "
"further, and starts knowing nothing about this conversation, "
"so the task must say everything it needs. Do not use it for "
"something you could do in one call yourself, or for anything "
"needing a decision only the reader can make."
"model that works on its own and gives you its answer. Use it "
"to cover several independent areas at once: call it several "
"times in one turn and each runs in parallel. It gets a "
"narrower set of tools than you: it reads, it cannot ask you or "
"the reader anything, it cannot delegate further, it changes "
"nothing unless you set write, and on a machine it may run only "
"a fixed list of read-only commands. It starts knowing nothing "
"about this conversation, so the task must say everything it "
"needs. Do not use it for something you could do in one call "
"yourself, or for anything needing a decision only the reader "
"can make."
),
parameters={
"type": "object",
src/lembas/services/tools.py
+42 -2
View File
@@ -188,6 +188,21 @@ RISKS = (RISK_READ, RISK_WRITE, RISK_EXECUTE, RISK_ASK)
# single page. Cut with the model told so, rather than refused.
MAX_FETCH_CHARS = 20_000
# The same bound for a knowledge document, and it was missing. `knowledge_get`
# returned `extracted_text` whole while every sibling reader capped and said so
# -- `fetch` above, `file_read`, the memories block, the skill index, the
# project listing. `MAX_EXTRACTED_CHARS` is 120_000 by default and an
# administrator can raise it, so one call on a long PDF filled an ordinary
# window with nothing anywhere reporting that it had.
#
# Larger than a fetched page on purpose. Somebody put this document in the
# library deliberately and named it in a search; a page the model followed a
# link to is a guess. Cut with the model told so rather than refused, which is
# what `fetch` and `file_read` both do -- a reader that fails on exactly the
# documents worth reading is worse than one that hands back the first part and
# says there was more.
MAX_DOCUMENT_CHARS = 40_000
@dataclass
class ToolContext:
@@ -471,6 +486,14 @@ async def _run_knowledge_get(context: ToolContext, args: dict[str, Any]) -> Tool
"results": [{"title": document.title, "id": document.id}],
}
body = document.extracted_text or document.extraction_error or "(no text)"
if len(body) > MAX_DOCUMENT_CHARS:
body = (
f"{body[:MAX_DOCUMENT_CHARS]}\n\n"
f"[Cut off here. This document is {len(document.extracted_text or ''):,} "
f"characters and the first {MAX_DOCUMENT_CHARS:,} are above. Search it "
"with knowledge_search to find the part you need.]"
)
event["truncated"] = True
return ToolOutcome(f"{document.title}\n\n{body}", event)
@@ -912,8 +935,21 @@ async def _run_ask_user(context: ToolContext, args: dict[str, Any]) -> ToolOutco
session-free snapshot that deliberately holds no way to reach one. Getting
here means some other path called `run_tool` directly, and saying so is
better than returning an empty answer the model would treat as a reply.
It read `args["question"]`, singular, against a schema that declares
`questions` and a list -- so the event it built always carried an empty
`query`, and the card showed a refusal with no sign of what had been asked.
Harmless only because this path is unreachable, which is exactly why nothing
caught it: schema drift on a branch no test exercises. Tolerant of the same
spellings `generation._questions_in` accepts, rather than importing it,
which would be a circular import for one field on a dead path.
"""
question = str(args.get("question") or "").strip()
asked: Any = args.get("questions") or args.get("question") or ""
if isinstance(asked, list):
asked = asked[0] if asked else ""
if isinstance(asked, dict):
asked = asked.get("question") or ""
question = str(asked).strip()
return ToolOutcome(
"That question could not be put to anyone, so it has gone unanswered. "
"Carry on without it, or say what you need.",
@@ -997,7 +1033,11 @@ REGISTRY: dict[str, ToolDef] = {
ToolDef(
name="knowledge_get",
family=FAMILY_KNOWLEDGE,
description="Read one knowledge document in full, by the id a search returned.",
description=(
"Read one knowledge document, by the id a search returned. A "
"long one is cut off at the end rather than refused, and you "
"are told when that happened."
),
parameters=_object({"id": _STRING}, ["id"]),
run=_run_knowledge_get,
),