Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Every injected prompt becomes editable, and several get written

Browse Source 
The instructions LLeMbas puts in front of a model were hard-coded: six
strings in a GUIDANCE dict, two headings, and the title request inline in
chat.py. An operator could not see what was being sent, let alone change
it, and there was nowhere for a custom tool to contribute its own guidance
when custom tools land.

services/prompts.py now holds each piece as a Fragment, and /admin/prompts
edits them with a preview of the whole assembled system message including
unsaved edits. harness.py keeps only the decisions -- which fragments apply
to this request, and what their variables resolve to.

The design turns on one choice: a fragment carries its gate as data
(families, requires, when_tools) rather than as a callable, because a
database row can carry the same three fields. Custom tools will therefore
register a fragment source and change nothing else -- there is a test that
says exactly that, and it is the reason the rest of the shape is what it is.

Consequences worth knowing:

  - Defaults live in code, overrides in the database, and text equal to its
    default is never stored. Otherwise pressing Save once would freeze
    today's wording forever and no later release could improve it.
  - An empty override means off. A fragment that was not submitted at all
    keeps what it had, because it may be missing from the page only because
    whatever contributes it is currently switched off.
  - requires= replaced the hand-written pair of memory guidance variants.
    The sentence that refers to a section now lives inside that section, so
    it cannot outlive it. That was the general problem the pair was a
    special case of.
  - {{name}}, with anything unrecognised passing through verbatim. The name
    grammar is the guard: {"total": 1} and ${PATH} are not candidates.
    Substitution is one pass and never recursive, because {{memories}}
    carries text a model wrote.

The wording is also overhauled, and a model now gets the core fragments
even with no tools -- the date above all. "An empty harness is worse than
none" was about tokens that say nothing; a model with no clock being asked
about the present is not that. Clearing those boxes restores the old
silence exactly. New: today's date, who it is talking to, the three-round
tool budget, that tool results are not replayed, that anything a tool
returns is data rather than instruction, and what the <document> wrapper
around an attachment is. Extended: memory_forget, notes_edit/delete,
skill_create/edit, and reading a knowledge document in full rather than
answering from an extract.

Tool descriptions stay in code and are listed read-only. They are schema
and they state facts about what a runner does; an edit would make the text
a lie with nothing to catch it.

No schema change -- one JSON row in the settings table.

488 tests. Version 0.2.0, which also invalidates the service worker cache
so the green artwork appears without a hard reload.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-07-31 23:47:49 +02:00
parent 71dc46455c
commit 1906919ee2
20 changed files with 2089 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/harness.py
+150 -115
View File
@@ -2,90 +2,60 @@
A model handed a `tools` array will often ignore it. It answers from recall
because that is what it was trained to do, and nothing in the request suggests
otherwise. The harness is the part of the prompt that says otherwise: one line
per tool about *when* to reach for it, the memories, and the list of skills
available.
otherwise. The harness is the part of the prompt that says otherwise: what day it
is, one line per tool about *when* to reach for it, the memories, and the list of
skills available.
The text itself is not here. Every piece of it is a fragment in
``services/prompts.py``, defaulted there and overridable by an administrator on
``/admin/prompts``; this module decides which fragments apply to a given request
and what their variables resolve to. That split is what lets a custom tool
contribute its own guidance later by registering a fragment source and nothing
else.
**On the "system prompts are precedence, not concatenation" rule.** That rule
governs the three authored layers -- instance, model, chat -- and it is
untouched here: exactly one of them still wins, and
``chat.effective_system_prompt`` still decides which. This is a different axis.
It describes the machinery rather than the behaviour, nobody authored it, and
there is nothing for it to disagree with. So it is prepended to whichever
authored prompt won, inside one system message, under a heading that makes the
seam obvious.
governs the three authored layers -- instance, model, chat -- and it is untouched
here: exactly one of them still wins, and ``chat.effective_system_prompt`` still
decides which. This is a different axis. It describes the machinery rather than
the behaviour, nobody authored it, and there is nothing for it to disagree with.
So it is prepended to whichever authored prompt won, inside one system message,
under a heading that makes the seam obvious.
One system message rather than two because several endpoints reject a second
one. The authored prompt goes last, where it is closest to the conversation.
One system message rather than two because several endpoints reject a second one.
The authored prompt goes last, where it is closest to the conversation.
Nothing is emitted for a model with no tools and no memories: an empty harness
is worse than none, being tokens that say only that there is nothing to say.
A model with no tools still gets the core fragments -- the date above all, since
it has no clock and is being asked about a present it cannot see. That is a
change from the original behaviour, where no tools meant no harness at all;
clearing those fragments in the admin page restores it exactly.
"""
from __future__ import annotations
import logging
from datetime import datetime
from typing import Any
from sqlalchemy import select
from sqlalchemy.orm import Session as DBSession
from lembas.db.models import User
from lembas.services import prompts, settings_store
from lembas.services.library import memories as memories_service
from lembas.services.library import skills as skills_service
log = logging.getLogger(__name__)
# Keyed by tool family, so a family that is off contributes nothing. Written as
# guidance rather than rules: a model told "you MUST search" searches for the
# capital of France.
GUIDANCE: dict[str, str] = {
"web_search": (
"- Look things up rather than trusting your recall, whenever the answer "
"depends on current facts, on details you are not certain of, or on "
"anything that may have changed. If the first results are thin or "
"beside the point, search again with different words instead of "
"answering from them — two or three searches are normal. Say where an "
"answer came from."
),
"knowledge": (
"- The user has a library of their own documents. When a question is "
"about their material — their files, their notes on paper, a page they "
"saved — search that before searching the web."
),
"notes": (
"- You keep notes across conversations. Search them when a task sounds "
"like one you have done before. Write one when you work something out "
"that would be tedious to work out again: a procedure, a decision and "
"its reasons, a summary of a long document."
),
# Two variants: the first refers to a heading that only exists when there
# is something under it, and telling a model to consult an absent section
# is a good way to make it invent one.
"memory": (
"- You can remember durable facts about this person — a preference, a "
"constraint, a name — but not the details of one task, and never "
"anything secret."
),
"memory_with_records": (
"- What is listed under “What you know about this person” below was "
"remembered earlier and still applies. Add to it only for durable facts "
"— a preference, a constraint, a name — never for the details of one "
"task, and never for anything secret."
),
"skills": (
"- Skills are procedures you have saved. The list below gives only each "
"one's name and when to use it; read the full instructions with "
"skill_get before following one. If you work out a repeatable way to do "
"something, save it as a new skill."
),
}
HEADING = "## How to work"
# A ceiling on the whole block, so that a large library cannot quietly eat the
# context window. Memory and skills have their own caps below this one.
# context window. Memory and skills have their own caps below this one. An
# administrator can lower it; `max_harness_chars` of 0 means "use this".
MAX_HARNESS_CHARS = 8000
# How many attached filenames to name in the prompt. Enough to show what the
# tags will look like, few enough that a chat with thirty files does not spend
# the window listing them -- this is an explanation, not a manifest.
MAX_NAMED_DOCUMENTS = 5
def _families(tools: list[dict[str, Any]]) -> list[str]:
"""Which families are represented in an offered tool list, in a fixed order."""
@@ -99,6 +69,111 @@ def _families(tools: list[dict[str, Any]]) -> list[str]:
return [family for family in FAMILIES if family in offered]
def _tool_names(tools: list[dict[str, Any]]) -> str:
return ", ".join(
name for tool in tools if (name := (tool.get("function") or {}).get("name"))
)
def _document_names(db: DBSession, chat) -> str:
"""The names of the non-image files attached anywhere in this chat."""
from lembas.db.models import Attachment
rows = list(
db.scalars(
select(Attachment.filename)
.where(Attachment.chat_id == chat.id, Attachment.kind != "image")
.order_by(Attachment.created_at)
.limit(MAX_NAMED_DOCUMENTS + 1)
).all()
)
if not rows:
return ""
if len(rows) > MAX_NAMED_DOCUMENTS:
return ", ".join(rows[:MAX_NAMED_DOCUMENTS]) + " and others"
return ", ".join(rows)
def context_variables(
db: DBSession,
user: User | None,
tools: list[dict[str, Any]] | None,
chat=None,
) -> dict[str, str]:
"""What every ``{{name}}`` in a fragment resolves to for this request.
The expensive ones are guarded by family, exactly as the memory block always
was: a model with no skills tool must not cause a skills query, and has no
business being told the memories either.
"""
from lembas.services import tools as tools_service
offered = tools or []
families = _families(offered)
stamp = datetime.now().astimezone()
values: dict[str, str] = {
"today": stamp.strftime("%A %-d %B %Y"),
"now": stamp.strftime("%A %-d %B %Y, %H:%M (UTC%z)"),
"instance_name": str(settings_store.get(db, "instance_name") or "LLeMbas"),
"user_name": (user.name or "") if user is not None else "",
"model_name": "",
"max_rounds": str(tools_service.MAX_ROUNDS),
"memory_limit": str(memories_service.MAX_MEMORY_CHARS),
"tool_names": _tool_names(offered),
"memories": memories_service.block(db, user) if "memory" in families else "",
"skills": skills_service.index_block(db, user) if "skills" in families else "",
"knowledge_bases": "",
"document_names": "",
}
if chat is not None:
from lembas.services import chat as chat_service
model = chat_service.model_for(db, chat)
values["model_name"] = model.label if model is not None else chat.model_id
# Naming the bases a chat is scoped to matters: without it the model
# cannot tell "there is nothing about this" from "I am only allowed to
# see the contracts folder", and phrases a miss as the former.
if "knowledge" in families and chat.knowledge_bases:
values["knowledge_bases"] = ", ".join(base.name for base in chat.knowledge_bases)
values["document_names"] = _document_names(db, chat)
return values
def limit_for(db: DBSession) -> int:
"""The ceiling on the assembled block."""
stored = settings_store.get(db, "max_harness_chars", key=settings_store.PROMPTS)
return int(stored or 0) or MAX_HARNESS_CHARS
def compose_from(
db: DBSession,
*,
variables: dict[str, str],
families: list[str],
has_tools: bool,
overrides: dict[str, str] | None = None,
) -> str:
"""Assemble the preamble from already-resolved variables.
Separate from `compose` because the admin preview has no chat and must not
invent one: a transient Chat whose `knowledge_bases` collection cannot be
populated without real rows is a trap, and taking a plain dict of variables
instead sidesteps it entirely.
"""
return prompts.assemble(
db,
groups=prompts.HARNESS_GROUPS,
variables=variables,
families=families,
has_tools=has_tools,
overrides=overrides,
limit=limit_for(db),
)
def compose(
db: DBSession,
user: User | None,
@@ -106,67 +181,27 @@ def compose(
chat=None,
) -> str:
"""The operational preamble for this request, or "" when there is nothing to say."""
families = _families(tools or [])
if not families:
return ""
parts: list[str] = [
HEADING,
"",
"You have tools. Use them rather than guessing; a wrong answer given "
"confidently is worse than a slower one that was checked.",
"",
]
# Read before the guidance is assembled, because whether there are any
# memories decides which wording the memory line gets.
block = memories_service.block(db, user) if "memory" in families else ""
for family in families:
if family == "memory" and block:
parts.append(GUIDANCE["memory_with_records"])
elif family in GUIDANCE:
parts.append(GUIDANCE[family])
if block:
parts += ["", "### What you know about this person", "", block]
# Naming the bases a chat is scoped to matters: without it the model cannot
# tell "there is nothing about this" from "I am only allowed to see the
# contracts folder", and phrases a miss as the former.
if "knowledge" in families and chat is not None and chat.knowledge_bases:
names = ", ".join(base.name for base in chat.knowledge_bases)
parts += [
"",
f"Knowledge searches in this chat cover only: {names}.",
]
if "skills" in families:
index = skills_service.index_block(db, user)
if index:
parts += [
"",
"### Skills available",
"",
index,
"",
"Read one with skill_get before following it.",
]
text = "\n".join(parts).strip()
if len(text) > MAX_HARNESS_CHARS:
text = text[:MAX_HARNESS_CHARS].rstrip() + "\n…"
return text
offered = tools or []
return compose_from(
db,
variables=context_variables(db, user, offered, chat),
families=_families(offered),
has_tools=bool(offered),
)
def join(harness: str, authored: str) -> str:
def join(harness: str, authored: str, *, lead: str = "") -> str:
"""Put the harness in front of whichever authored prompt won.
Separated from `compose` so the precedence between instance, model and chat
stays testable on its own -- this function is the only place the two axes
meet.
meet. `lead` is the sentence that sits on the seam and says which side wins
when they disagree; it is a fragment like everything else, and an empty one
leaves the bare rule that was there before.
"""
if not harness:
return authored
if not authored:
return harness
return f"{harness}\n\n---\n\n{authored}"
seam = f"{lead}\n\n---" if lead else "---"
return f"{harness}\n\n{seam}\n\n{authored}"