What the audit is worth keeping, and where
Five passes produced a working document that said on its first line it was temporary. This is it being spent rather than abandoned. CLAUDE.md gains eleven paragraphs, each a thing that had shipped looking correct: a handler bound to a shared variable rather than its own socket, a script the base template already loads being loaded again, a control that stays clickable while it awaits permission, "is this name taken?" asked about visibility instead of ownership, root running a file the service account can write, sourcing anything under $PREFIX, a read-only command name that is not a read-only command, 0.0.0.0 being this machine, a folder that is not a label, a file that is not deleted by the row that named it, and a measuring harness that measured an unstyled page and reported a dramatic finding that was entirely an artefact. PLAN.md carries the seven things the audit found and deliberately did not fix, each with why: they change what something does rather than fix what it claims to do, which is not an audit's job. deploy/README.md says why root runs a copy, and that a host installed before this keeps the old wiring until the installer is re-run -- the button cannot fix it, because the button runs the old unit. docs/notes/audit-0.9.md is deleted, having been all three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,7 +20,9 @@ lembas info # paths + counts, useful when confused
|
|||||||
lembas secret-key # generate LEMBAS_SECRET_KEY
|
lembas secret-key # generate LEMBAS_SECRET_KEY
|
||||||
lembas create-admin # create or promote an admin
|
lembas create-admin # create or promote an admin
|
||||||
|
|
||||||
pytest # 2088 tests, ~2min
|
pytest # 2283 tests, ~2.5min
|
||||||
|
pytest -m "not slow" # skips the 365 that stand up a real
|
||||||
|
# server, shell, PTY or git repository
|
||||||
# PLAN.md tracks what is and is not built
|
# PLAN.md tracks what is and is not built
|
||||||
ruff check . # lint (line length 100)
|
ruff check . # lint (line length 100)
|
||||||
python scripts/build_artwork.py # regenerate artwork (SVG + PWA icons;
|
python scripts/build_artwork.py # regenerate artwork (SVG + PWA icons;
|
||||||
@@ -221,6 +223,10 @@ touching the code it names -- these are the same notes, not a summary.
|
|||||||
- `docs/notes/branding.md` -- the branding snapshot and why it is a Jinja global,
|
- `docs/notes/branding.md` -- the branding snapshot and why it is a Jinja global,
|
||||||
where the instance name went and how an upgrade keeps it, how a custom theme
|
where the instance name went and how an upgrade keeps it, how a custom theme
|
||||||
inherits through `data-base`, and why `/branding.css` is a route.
|
inherits through `data-base`, and why `/branding.css` is a route.
|
||||||
|
- `docs/notes/release-checklist.md` -- the manual pass before a release. Not a
|
||||||
|
corner of the code but the opposite: the things a machine cannot test, which
|
||||||
|
is everything needing a real endpoint, a real machine, real hardware or a real
|
||||||
|
pair of eyes. Read it when cutting a release, not while writing code.
|
||||||
- `docs/notes/subagents.md` -- the hidden chat a helper runs in, why `unattended`
|
- `docs/notes/subagents.md` -- the hidden chat a helper runs in, why `unattended`
|
||||||
is a column and not a kind, the two halves that stop a helper stalling on a
|
is a column and not a kind, the two halves that stop a helper stalling on a
|
||||||
card nobody can see, what it may run and why Auto is never inherited, and where
|
card nobody can see, what it may run and why Auto is never inherited, and where
|
||||||
@@ -1645,6 +1651,107 @@ that the right one works, because only the second half would have passed
|
|||||||
throughout. When adding a control that writes, check the verb against the route,
|
throughout. When adding a control that writes, check the verb against the route,
|
||||||
and assert on the row rather than on the response.
|
and assert on the row rather than on the response.
|
||||||
|
|
||||||
|
**A handler bound to a shared variable is bound to the wrong thing.**
|
||||||
|
`terminal.js`'s `socket.onclose` set the module-level `socket` to null, and
|
||||||
|
`close()` *queues* its event rather than firing it -- so on either reconnect
|
||||||
|
path the old socket's close arrived **after** a new one had been assigned and
|
||||||
|
nulled the live one. Output kept arriving, because `onmessage` is bound to the
|
||||||
|
object; every send gates on the variable, so each keystroke was dropped and no
|
||||||
|
resize was ever sent again, on a panel that looked perfectly healthy. It then
|
||||||
|
said "Disconnected" about a shell that had just reconnected, because
|
||||||
|
`closedOnPurpose` had already been cleared for the new connection. Each handler
|
||||||
|
captures its own socket in a local and returns early when `socket !== opened`.
|
||||||
|
|
||||||
|
**A script the base template loads must not be loaded again by a page.**
|
||||||
|
`messages/index.html` listed `composer.js` and `commands.js` in its `scripts`
|
||||||
|
block and `base.html` already loads both, so that screen ran each **twice**.
|
||||||
|
Each is an IIFE with its own state, and `stopPropagation()` does not stop a
|
||||||
|
second listener already bound to the same node: two composer menus stacked, one
|
||||||
|
Enter running a command twice, and `Alt+B`/`Alt+E`/`Alt+T`/`Alt+I` toggling
|
||||||
|
their panel twice, which is to say doing nothing at all. Nothing about that
|
||||||
|
looks like a double load. `tests/test_ui_js.py` sweeps every template against
|
||||||
|
`base.html`'s list.
|
||||||
|
|
||||||
|
**Leave the state a control was in before you await.** `audio.js` moved the
|
||||||
|
microphone out of `idle` only once `getUserMedia` had resolved, and the
|
||||||
|
browser's permission prompt sits in between -- so every click while it was up
|
||||||
|
opened another stream and another recorder, of which only the last was ever
|
||||||
|
stopped. The browser's recording indicator then stayed on until the tab closed.
|
||||||
|
`working` already existed and was already styled; it was the state the button
|
||||||
|
was missing.
|
||||||
|
|
||||||
|
**"Is this name taken?" is a question about ownership, not visibility.**
|
||||||
|
`skills.create` checked with `by_name`, which is scoped through
|
||||||
|
`sharing.visible_to` -- owned *or* shared -- against a `(owner_id, name)`
|
||||||
|
constraint. So a skill somebody shared with you took that name out of your
|
||||||
|
library, and refused yours with "edit it instead", naming a row you cannot edit
|
||||||
|
because sharing grants reading only. `by_name` is right for `skill_get` and
|
||||||
|
`skill_edit`, where a shared skill is what the model is reaching for;
|
||||||
|
`owned_by_name` is the uniqueness question. `documents.create_base` next door
|
||||||
|
had always asked it correctly.
|
||||||
|
|
||||||
|
**Root must not run a file the service account can write.** The update helper's
|
||||||
|
unit pointed `ExecStart` inside the checkout, which `install.sh` clones *as* the
|
||||||
|
service user -- so anything able to write as that account could rewrite the
|
||||||
|
script, create the request file it also owns, and be root. Worse, no compromise
|
||||||
|
was needed: an update pulls new code as that user and root then runs whatever it
|
||||||
|
fetched, so control of the branch was control of root. `install.sh` puts a
|
||||||
|
root-owned copy at `/usr/local/lib/lembas/update.sh`. The cost -- changing
|
||||||
|
`update.sh` needs the installer re-run -- is the right one, and `update.sh`
|
||||||
|
detects both the stale copy and the old wiring. **The test that existed asserted
|
||||||
|
the vulnerable `ExecStart` line verbatim**, and had pinned it in place.
|
||||||
|
|
||||||
|
**Nothing under `$PREFIX` may be sourced.** `update.sh` did `. "$PREFIX/.deploy-env"`.
|
||||||
|
The file is root-owned, having been written with `sudo tee` -- but `$PREFIX` is
|
||||||
|
the service account's own directory at mode 755, and write permission on a
|
||||||
|
*directory* is all it takes to unlink a file and put another there. The two
|
||||||
|
values it wants are parsed with patterns now. The test asserts nothing under
|
||||||
|
`$PREFIX` is sourced rather than naming `.deploy-env`, because the next file
|
||||||
|
read from there would have the same problem.
|
||||||
|
|
||||||
|
**A read-only *name* is not a read-only *command*.** `policy._UNSAFE` stops a
|
||||||
|
line being composed out of two commands and says nothing about one command that
|
||||||
|
composes another itself -- and the obvious read-only tools do: `find -exec` runs
|
||||||
|
a program, `-fprintf` writes a file, `-delete` removes one, `rg --pre` runs a
|
||||||
|
preprocessor per file. None needs a character `_UNSAFE` refuses. `find *` was on
|
||||||
|
`SAFE_COMMANDS`, which is what a **subagent** is pinned to in every mode,
|
||||||
|
unattended, with no card anybody could approve. `policy._ACTION` refuses those
|
||||||
|
flags in `subject()` rather than the list being trimmed: a glob cannot say "and
|
||||||
|
no dangerous flags", and "this one looks read-only" is exactly the reasoning
|
||||||
|
that put `find` there.
|
||||||
|
|
||||||
|
**`0.0.0.0` is this machine.** `hosts._literal` answered from `is_loopback`, and
|
||||||
|
the unspecified addresses are not loopback -- so it returned a *decided* False,
|
||||||
|
which also short-circuited `resolves_here` and skipped the DNS half. Connecting
|
||||||
|
to `0.0.0.0` or `::` goes to loopback on Linux, so an SSH profile pointed there
|
||||||
|
reached this host's own sshd with the guard still reading "off".
|
||||||
|
|
||||||
|
**A folder is not a label.** `effective_system_prompt` walks up from the chat
|
||||||
|
through its folder and that folder's parents, so a chat filed in somebody else's
|
||||||
|
folder takes their system prompt. `_new_chat` resolved the folder, discarded it
|
||||||
|
when it was not the caller's -- and then stored the raw `folder_id` anyway, so
|
||||||
|
the ownership check governed which *seeds* were applied and not where the chat
|
||||||
|
went.
|
||||||
|
|
||||||
|
**A file is not deleted by the row that named it.** `db.delete(chat)` cascades
|
||||||
|
to messages and attachment *rows* and leaves every file on disk, with nothing
|
||||||
|
that will ever look at them again -- `sweep_orphans` only considers uploads that
|
||||||
|
were never attached. `chat_service.delete_chats` is the one way to delete a chat
|
||||||
|
and unlinks first, while the rows still say which files they are. Underneath it,
|
||||||
|
`files.claim` bound `message_id` and never `chat_id`, so anything picked on the
|
||||||
|
*new-chat* screen kept an empty `chat_id` for life -- and six readers filter on
|
||||||
|
that column, so those files were unnamed in the prompt, unopenable in the canvas
|
||||||
|
and invisible to the one caller the cleanup had.
|
||||||
|
|
||||||
|
**Drive the JavaScript, and check the harness before believing it.** Three of
|
||||||
|
the four bugs the testing pass found came from a Node DOM stub. The fourth
|
||||||
|
lesson is about the instrument: a headless-Chromium harness that renders a page
|
||||||
|
through `TestClient` and rewrites its asset URLs reported **all five tab panels
|
||||||
|
visible at once** -- because the templates emit absolute `http://testserver/static/…`
|
||||||
|
URLs via `url_for`, not `/static/…`, so the rewrite matched nothing and it was
|
||||||
|
measuring an unstyled document. A dramatic finding that was entirely an
|
||||||
|
artefact. Any such harness must fail loudly on an unrewritten asset URL.
|
||||||
|
|
||||||
## Releasing, and the changelog
|
## Releasing, and the changelog
|
||||||
|
|
||||||
**Every version bump gets a `CHANGELOG.md` entry, in the same commit.** Not
|
**Every version bump gets a `CHANGELOG.md` entry, in the same commit.** Not
|
||||||
|
|||||||
+27
-6
@@ -95,19 +95,40 @@ The button is opt-in, and the reason is a boundary rather than caution:
|
|||||||
INSTALL_UPDATE_HELPER=1 SITE_HOST=chat.example ./deploy/install.sh
|
INSTALL_UPDATE_HELPER=1 SITE_HOST=chat.example ./deploy/install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
That installs `lembas-update.path` and `lembas-update.service`. The web
|
That installs `lembas-update.path` and `lembas-update.service`, and puts a
|
||||||
|
**root-owned copy** of `update.sh` at `/usr/local/lib/lembas/update.sh`. The web
|
||||||
interface writes `$PREFIX/data/update-requested`; the path unit notices and the
|
interface writes `$PREFIX/data/update-requested`; the path unit notices and the
|
||||||
service runs `update.sh` **as root**, on the configured channel.
|
service runs that copy **as root**, on the configured channel.
|
||||||
|
|
||||||
|
**Why a copy.** The unit used to point inside the checkout, and `install.sh`
|
||||||
|
clones the checkout *as the service user* — so root was executing a file the
|
||||||
|
unprivileged account could rewrite, and one that every update replaces with
|
||||||
|
whatever the branch contained. Either turns a compromise of the web application
|
||||||
|
into root, and the second needs no compromise at all. The cost is that changing
|
||||||
|
`update.sh` needs the installer re-run; the script tells you when its copy has
|
||||||
|
fallen behind, and says so loudly if it finds itself running from inside the
|
||||||
|
checkout.
|
||||||
|
|
||||||
|
**If you installed the helper before 1.0.0, re-run the installer.** The old
|
||||||
|
wiring stays until you do, and the update button cannot fix it — the button runs
|
||||||
|
the old unit.
|
||||||
|
|
||||||
**What that grants.** Anybody who can administer this web interface can then
|
**What that grants.** Anybody who can administer this web interface can then
|
||||||
deploy whatever is on the configured branch and restart the service. That is the
|
deploy whatever is on the configured branch and restart the service. That is the
|
||||||
point of it, and it is why it is not the default.
|
point of it, and it is why it is not the default.
|
||||||
|
|
||||||
**What it deliberately does not grant.** The request file carries nothing that
|
**What it deliberately does not grant.** The request file carries nothing that
|
||||||
reaches a command line — no ref, no branch, no channel, no arguments. Both are
|
reaches a command line — no ref, no branch, no channel, no arguments, and its
|
||||||
baked into the unit at install time, so the button is always "deploy the channel
|
*contents* are never read at all. Both are baked into the unit at install time,
|
||||||
this host was configured with" and never "deploy something else". Re-running the installer without the flag removes both units and the
|
so the button is always "deploy the channel this host was configured with" and
|
||||||
marker, and the page goes back to printing the manual command.
|
never "deploy something else". Re-running the installer without the flag removes
|
||||||
|
both units, the marker and the root-owned copy, and the page goes back to
|
||||||
|
printing the manual command.
|
||||||
|
|
||||||
|
A re-run **keeps the channel this host already follows** rather than resetting it
|
||||||
|
to `stable`: the channel is declared in `lembas.env` and in the unit, a re-run
|
||||||
|
keeps the first while rewriting the second, and an installer that silently moved
|
||||||
|
one half was causing exactly the mismatch the Updates page detects.
|
||||||
|
|
||||||
Without the helper the page says so and shows `sudo …/deploy/update.sh`, which is
|
Without the helper the page says so and shows `sudo …/deploy/update.sh`, which is
|
||||||
the same honest degradation the SSH and search extras have.
|
the same honest degradation the SSH and search extras have.
|
||||||
|
|||||||
@@ -1,416 +0,0 @@
|
|||||||
# The 0.9 audit
|
|
||||||
|
|
||||||
A working document, written during the pass over everything that landed between
|
|
||||||
`0.8.1` and `0.9.8` -- 16 commits, 163 files, +16,795 / −1,608. It exists so the
|
|
||||||
findings survive the session that produced them, and so the security and testing
|
|
||||||
stages have something to work from rather than re-deriving it.
|
|
||||||
|
|
||||||
**This file is temporary.** Everything in it either becomes a `CHANGELOG.md`
|
|
||||||
line, a paragraph in `CLAUDE.md`, or an entry under "After 1.0.0" in `PLAN.md`.
|
|
||||||
It is deleted before the release.
|
|
||||||
|
|
||||||
Each finding carries a disposition:
|
|
||||||
|
|
||||||
- **fixed** -- landed, with a test
|
|
||||||
- **later** -- real, but the fix changes what a feature does, so it is the
|
|
||||||
user's call and it is written up rather than done
|
|
||||||
- **stated** -- correct as it stands, and recorded so the next audit does not
|
|
||||||
spend an hour rediscovering that it is deliberate
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## The harness and the instruction prompts
|
|
||||||
|
|
||||||
### {{timezone}} shipped as a hole — *fixed*
|
|
||||||
|
|
||||||
`clock.name_for` returns `""` for anybody who has never chosen a zone, which is
|
|
||||||
the default state of every account. `substitute` drops a line only when the
|
|
||||||
whole line is **blank** after expansion, and this variable sits inside a
|
|
||||||
sentence, so every such request carried:
|
|
||||||
|
|
||||||
```
|
|
||||||
- Times the person gives you are in unless they say otherwise.
|
|
||||||
```
|
|
||||||
|
|
||||||
Both the comment at `harness.py:191` and the fragment's own hint asserted the
|
|
||||||
line disappeared. Neither had ever been true. The sibling call site
|
|
||||||
(`schedule/compile.py:102`) has always had the fallback.
|
|
||||||
|
|
||||||
Worth noting *why* the fallback is right rather than merely convenient: `stamp`
|
|
||||||
is `clock.now_for(user)`, which already falls back to the server's zone, and
|
|
||||||
`{{now}}` already prints its offset. The times were already in that zone.
|
|
||||||
Withholding the label from a value the model had been given was not restraint.
|
|
||||||
|
|
||||||
### The admin preview could not show most of what it previews — *fixed*
|
|
||||||
|
|
||||||
`admin_prompts.py` built its variables with `context_variables(db, user, [], None)`
|
|
||||||
— no tools, no chat — and backfilled six names. Every other `requires` gate
|
|
||||||
stayed empty, so **eleven fragments could never appear in a preview whatever an
|
|
||||||
administrator ticked**: the whole agent surface, both scheduling fragments and
|
|
||||||
the helper warning. Editing `tool.agent` and pressing preview showed a system
|
|
||||||
message with `tool.agent` missing from it, silently.
|
|
||||||
|
|
||||||
Fixed with sample values, which is what `SAMPLE_DOCUMENTS` already did for
|
|
||||||
attachments, plus two controls the "Tools offered" row could not reach — "This
|
|
||||||
chat is" (ordinary / scheduled task / helper) and an agent-mode select. The
|
|
||||||
samples are gated **exactly as `context_variables` gates the real values**,
|
|
||||||
because a preview that admits a fragment the real request would not is worse
|
|
||||||
than one that omits it.
|
|
||||||
|
|
||||||
### Fragments named tools that had been withdrawn — *fixed for Plan mode, later for the rest*
|
|
||||||
|
|
||||||
`resolve_tools` filters per **tool**; `harness._families` re-derives gates per
|
|
||||||
**family**. So a family survives on its readers while its writers are gone, and
|
|
||||||
the family-gated fragment still names them — under `core.tool_list` saying
|
|
||||||
"Anything not named there does not exist here — calling it costs a round and
|
|
||||||
returns nothing."
|
|
||||||
|
|
||||||
**Fixed:** Plan mode. `agent/tools.py:1185` withdraws `plan_update` there, and
|
|
||||||
`tool.plan_update` was gated on `{{plan}}`, which is set whenever a plan exists
|
|
||||||
in any mode. A new `plan_editable` variable — the plan text, blank in Plan mode
|
|
||||||
— now gates it. Writing the test found a second instance the audit had missed:
|
|
||||||
`context.plan` also said "Change it with plan_update", so that sentence moved to
|
|
||||||
the fragment whose job it is.
|
|
||||||
|
|
||||||
**Later:** a read-only helper keeps seven fragments naming fifteen withdrawn
|
|
||||||
write tools (`notes_create`, `report_write`, `file_write`, `skill_create`, the
|
|
||||||
three schedule writers…). The principled fix is the split `tool.skills` /
|
|
||||||
`tool.skills_write` already demonstrates, applied to `notes`, `report`,
|
|
||||||
`schedule` and `agent_edits` — four more fragment pairs. That is a prompt
|
|
||||||
restructure rather than a bug fix and it is proposed for after 1.0.0. The cost
|
|
||||||
today is bounded: `{{tool_names}}` is authoritative and the model has it, so a
|
|
||||||
helper wastes at most one round discovering a writer is absent.
|
|
||||||
|
|
||||||
### Three tool descriptions disagreed with their runners — *all fixed*
|
|
||||||
|
|
||||||
- **`subagent_run` opened "a second model with the same tools"** and contradicted
|
|
||||||
itself two sentences later. The child is forced to `ask:False`,
|
|
||||||
`subagent:False`, `write:False` with commands pinned to `SAFE_COMMANDS`. The
|
|
||||||
leading clause is what a model reads when sizing a task.
|
|
||||||
- **`knowledge_get` returned `extracted_text` whole** while every sibling reader
|
|
||||||
capped and said so — `fetch` at 20k, `file_read`, the memories block, the
|
|
||||||
skill index, the project listing. `MAX_EXTRACTED_CHARS` defaults to 120,000,
|
|
||||||
so one call on a long PDF filled an ordinary window with nothing reporting it.
|
|
||||||
Its description said "in full", which is why this read as correct: the tool did
|
|
||||||
exactly what it claimed. Now `MAX_DOCUMENT_CHARS = 40_000`, cut with the model
|
|
||||||
told, which is what `fetch` and `file_read` both do.
|
|
||||||
- **`_run_ask_user` read `args["question"]`**, singular, against a schema
|
|
||||||
declaring `questions` and a list — so its event always carried an empty
|
|
||||||
`query`. Harmless only because the path is documented unreachable, which is
|
|
||||||
exactly why nothing caught it.
|
|
||||||
|
|
||||||
### Fragment text that was false — *fixed*
|
|
||||||
|
|
||||||
`tool.subagent_agent` said a helper "reads and reports … and nothing else, in
|
|
||||||
every mode" beside a `write` parameter that makes one write files, and named
|
|
||||||
**seven** of the **twenty-three** commands in `SAFE_COMMANDS` — so a model
|
|
||||||
avoided commands it was allowed, which costs nothing visible and is therefore
|
|
||||||
never reported. Both halves are now pinned by tests against the real list and
|
|
||||||
the real schema, because prose and a constant drift the moment one is edited
|
|
||||||
alone.
|
|
||||||
|
|
||||||
The distinction the wording now has to carry, and did not before: **commands**
|
|
||||||
are fixed to the read-only list in every mode, since an unattended chat cannot
|
|
||||||
approve anything; **files** may be written, but only by a helper asked for with
|
|
||||||
`write` and only from Edit or Auto.
|
|
||||||
|
|
||||||
### `tool.background` promises a notification that can be off — *later*
|
|
||||||
|
|
||||||
"When a background job finishes you are told in a new turn" has no `requires`
|
|
||||||
for `agents.background_notify`, and the runner itself branches on that flag and
|
|
||||||
says "Check on it with `job_output`" instead. One fragment, two behaviours. The
|
|
||||||
fix is a variable and a gate, the same shape as `plan_editable`; deferred only
|
|
||||||
because it wants the split above rather than a third one-off.
|
|
||||||
|
|
||||||
### `FAMILY_ASK` has no fragment at all — *later*
|
|
||||||
|
|
||||||
Every other family in `FAMILIES` contributes one. `ask_user` is offered in
|
|
||||||
nearly every chat and **all** of its guidance lives in its schema description,
|
|
||||||
which is the one thing an administrator cannot edit. Proposed rather than done:
|
|
||||||
it is new prompt text, and new prompt text in an audit stage is how an audit
|
|
||||||
turns into a feature.
|
|
||||||
|
|
||||||
### Smaller, all *stated*
|
|
||||||
|
|
||||||
- **`model_name` is resolved on every request and rendered by no shipped
|
|
||||||
fragment.** It is documented in `VARIABLES` and available to an
|
|
||||||
administrator's override, so it is an escape hatch rather than dead work. The
|
|
||||||
cost is one small `select` on a table with tens of rows.
|
|
||||||
- **Four pairs of fragments share an `order`**, which undercuts "sparse so a
|
|
||||||
custom tool can be slotted between two of these". Ties break alphabetically
|
|
||||||
and deterministically. One has a readable consequence: `context.schedule`
|
|
||||||
sorts below `context.memories`, so a scheduled task's own instruction appears
|
|
||||||
under the reader's remembered facts.
|
|
||||||
- **`core.rounds` is dark on a stock install**, since `DEFAULT_CHAT_ROUNDS = 0`.
|
|
||||||
Deliberate — `core.keep_working` is its complement and exactly one ever fires.
|
|
||||||
- **A row-backed fragment with empty `guidance` can never be selected**, so for
|
|
||||||
those "never configured" and "deliberately switched off" are the same state —
|
|
||||||
the one distinction the absent-vs-empty convention exists to preserve.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Controls
|
|
||||||
|
|
||||||
The four sweeps `CLAUDE.md` is shaped around were run by hand over every
|
|
||||||
template and **all four came back clean**: 68 htmx verbs against 179 registered
|
|
||||||
routes with zero method mismatches, 88 plain form actions and 30
|
|
||||||
`fetch`/`htmx.ajax` call sites with zero, every verb-carrying element declaring
|
|
||||||
its own `hx-target` or `hx-swap="none"` checked against every ancestor, no empty
|
|
||||||
verb attributes, both `form="…"` sites correct, and every `hx-target`,
|
|
||||||
`hx-include` and `from:` selector resolving to an id that exists.
|
|
||||||
|
|
||||||
Making each one a **test that sweeps** is Stage 2's work. A one-off audit that
|
|
||||||
finds nothing is worth much less than the same audit made permanent.
|
|
||||||
|
|
||||||
## Capabilities that are built, documented and unreachable — *Stage 2*
|
|
||||||
|
|
||||||
The `Model.params_json` and `library.share` family: the code is right, the route
|
|
||||||
validates, and nothing can reach it.
|
|
||||||
|
|
||||||
- **Folder nesting.** `PATCH /api/folders/{id}` handles `parent_id` with a cycle
|
|
||||||
guard and `MAX_DEPTH = 8`, `partials/_folder.html` recurses, and `README.md`
|
|
||||||
advertises "arbitrarily nested". No template or script submits `parent_id`.
|
|
||||||
- **Moving a chat into a folder.** `api/chats.py` handles `folder_id` on PATCH;
|
|
||||||
only the composer submits it, at creation. `chat/_composer.html` even carries
|
|
||||||
a comment reasoning about "the only way into a folder was to make the chat
|
|
||||||
elsewhere and move it".
|
|
||||||
- **`Connection.extra_headers_json`** is read into every LLM request and written
|
|
||||||
by no form, so its documented use — OpenRouter's `HTTP-Referer` — is
|
|
||||||
unreachable. *later*: nothing advertises it, so nothing is currently untrue.
|
|
||||||
|
|
||||||
Written and never read: `Chat.compacted_at`, `User.last_login_at`,
|
|
||||||
`Schedule.last_fire_at`, `Schedule.compiled_at`. Neither read nor written:
|
|
||||||
`general.require_approval` (its comment says "reserved"), and the `back` form
|
|
||||||
parameter on two `admin_models` routes.
|
|
||||||
|
|
||||||
**A third, found by asking where generated images go.** `db.delete(chat)`
|
|
||||||
cascades to the message and attachment *rows* and leaves every file on disk;
|
|
||||||
`files.remove_files_for_chats` was written for exactly that and was called from
|
|
||||||
one place, the temporary-chat sweep. The delete button, a schedule's task chat,
|
|
||||||
a helper's chat and account deletion all went straight to `db.delete`.
|
|
||||||
|
|
||||||
Underneath it, a second fault that made the first invisible: `files.claim`
|
|
||||||
bound `message_id` and never `chat_id`, so anything uploaded on the *new-chat*
|
|
||||||
screen kept an empty `chat_id` for the rest of its life. Six readers filter on
|
|
||||||
that column, so for those files the harness did not name them among the attached
|
|
||||||
documents, the canvas refused to open them, and the one caller the cleanup
|
|
||||||
helper had found nothing to remove. Both fixed, and `chat_service.delete_chats`
|
|
||||||
is now the only way a chat is deleted.
|
|
||||||
|
|
||||||
*Stated:* `Message.parent_id` and `content_parts_json` are deliberate forward
|
|
||||||
-looking columns; `Chat.archived` is already recorded in `PLAN.md` as a column
|
|
||||||
nothing surfaces.
|
|
||||||
|
|
||||||
## Verified by rendering, not by reading — *Stage 2*
|
|
||||||
|
|
||||||
There is no browser extension on this machine, so `/admin/prompts` and
|
|
||||||
`/admin/customization` were checked by rendering them through `TestClient`,
|
|
||||||
rewriting the asset URLs to `file://`, and driving headless Chromium against the
|
|
||||||
result. Worth recording because the first run of that harness **measured an
|
|
||||||
unstyled page**: the templates emit absolute `http://testserver/static/…` URLs
|
|
||||||
via `url_for`, not `/static/…`, so the replacement matched nothing and no CSS
|
|
||||||
loaded. It reported all five tab panels visible at once — a dramatic finding
|
|
||||||
that was entirely an artefact. The harness now fails loudly on any unrewritten
|
|
||||||
asset URL.
|
|
||||||
|
|
||||||
What it measured once fixed:
|
|
||||||
|
|
||||||
- `/admin/prompts`, tab bar's on-screen position across the five tabs:
|
|
||||||
**385 → 642px before** (the page moving under the reader), **195px constant
|
|
||||||
after**, with `scrollTop` staying at 0.
|
|
||||||
- `/admin/customization`, document height: **5495px before, 2820px after**.
|
|
||||||
|
|
||||||
## Documentation — *Stage 2*
|
|
||||||
|
|
||||||
- `README.md` is stale by six phases. "Planned: Image generation · OCR ·
|
|
||||||
semantic search" — two of the three shipped. Nothing under Features mentions
|
|
||||||
schedules, reports, subagents, branding, quotas, web push, updates, Docker or
|
|
||||||
LXC.
|
|
||||||
- `CLAUDE.md` says "Three topics live in `docs/notes/`" and lists seven.
|
|
||||||
- `CLAUDE.md` says 2088 tests; there are 2092 before this stage.
|
|
||||||
- `tests/__pycache__/test_zz_{dump,live}*.pyc` are stale bytecode for two files
|
|
||||||
that no longer exist.
|
|
||||||
|
|
||||||
## Security — *Stage 3*
|
|
||||||
|
|
||||||
**Two privilege escalations in the update helper, both root, both fixed.** The
|
|
||||||
helper is the one place this application deliberately crosses a privilege
|
|
||||||
boundary, and it crossed it twice more than intended. Neither is reachable
|
|
||||||
from the web interface: both need code execution as the `lembas` service
|
|
||||||
account first. That is precisely the boundary the unprivileged split exists to
|
|
||||||
hold, so "you need a foothold" is the threat model, not a mitigation.
|
|
||||||
|
|
||||||
**1. Root ran a script the service account owns.**
|
|
||||||
`ExecStart=/bin/bash __PREFIX__/app/deploy/update.sh` — inside the checkout,
|
|
||||||
owned `lembas:lembas`, because `install.sh` clones as that user. So: write your
|
|
||||||
payload into `deploy/update.sh`, `touch data/update-requested` (the service
|
|
||||||
account owns that directory too), and systemd runs it as root. The web
|
|
||||||
interface's `AdminUser` check is not the gate systemd honours.
|
|
||||||
|
|
||||||
There is a second path needing no compromise at all: an update pulls new code
|
|
||||||
*as the service user*, and root then executes whatever `deploy/update.sh` that
|
|
||||||
pull contained. **Control of the branch was control of root.**
|
|
||||||
|
|
||||||
Fixed by installing a root-owned copy at `/usr/local/lib/lembas/update.sh` and
|
|
||||||
pointing the unit there. The cost — improving `update.sh` needs the installer
|
|
||||||
re-run — is the right one: root should not execute a script that arrived over
|
|
||||||
the network a moment ago. The script warns when its own copy has fallen behind.
|
|
||||||
|
|
||||||
**The old test asserted the vulnerable line**
|
|
||||||
(`assert "ExecStart=/bin/bash __PREFIX__/app/deploy/update.sh" in unit`). It
|
|
||||||
passed for the life of the feature and pinned the bug in place — the recurring
|
|
||||||
failure of this codebase, applied to a privilege boundary: an assertion about
|
|
||||||
the text rather than about the property the text was meant to have.
|
|
||||||
|
|
||||||
**2. Root sourced a file the service account can replace.**
|
|
||||||
`. "$PREFIX/.deploy-env"`. The file is root-owned, having been written with
|
|
||||||
`sudo tee` — but `$PREFIX` is the service account's own directory at mode 755,
|
|
||||||
and write permission on a *directory* is all it takes to unlink a file and put
|
|
||||||
another there. On the live host `.deploy-env` did not even exist, so it could
|
|
||||||
simply be created. `.` runs its contents as root.
|
|
||||||
|
|
||||||
This one survived the first fix entirely, and the helper is what made it
|
|
||||||
reachable: before the `.path` unit existed, `update.sh` only ran when an
|
|
||||||
administrator invoked it from a shell. Fixed by parsing the two values it wants
|
|
||||||
with strict patterns instead of sourcing. The test asserts that **nothing**
|
|
||||||
under `$PREFIX` is sourced, rather than naming `.deploy-env`, because the next
|
|
||||||
file read from there would have the same problem.
|
|
||||||
|
|
||||||
**Upgrade note:** a host that installed the helper before this keeps the old
|
|
||||||
unit, and only re-running the installer moves it. `update.sh` now detects that
|
|
||||||
it is running from inside the checkout and says so loudly — otherwise the
|
|
||||||
vulnerable hosts are exactly the ones that never hear about it.
|
|
||||||
|
|
||||||
**Also fixed, sub-threshold as a vulnerability but a real bug:** the share
|
|
||||||
panel built its `hx-vals` by pasting the search term into a JSON string. Jinja
|
|
||||||
escapes the quote for HTML and the parser decodes it again before htmx parses
|
|
||||||
the JSON, so a `"` in a search term ended the string and silently stopped every
|
|
||||||
checkbox in the panel from submitting anything. `q` was the last key, so an
|
|
||||||
injected one would also have won a duplicate-key parse. Built with `| tojson`
|
|
||||||
over the whole object now.
|
|
||||||
|
|
||||||
**3. A read-only command that was not read-only.** `SAFE_COMMANDS` — the list a
|
|
||||||
**subagent** is pinned to, in every mode, unattended, with no approval card
|
|
||||||
possible — contained `find *`. GNU `find` writes files (`-fprintf`), runs
|
|
||||||
programs (`-exec … +`) and deletes them (`-delete`), and none of those needs a
|
|
||||||
character `policy._UNSAFE` refuses. `rg --pre` is the same shape.
|
|
||||||
|
|
||||||
So the chain was: a parent in **Plan** mode — which promises "reads freely,
|
|
||||||
changes nothing" — spawns a helper on a `RISK_READ` tool with no card; the
|
|
||||||
helper's `shell_run` survives because `writes_off` drops `RISK_WRITE` and
|
|
||||||
deliberately keeps `RISK_EXECUTE`; `find . -maxdepth 0 -fprintf
|
|
||||||
~/.ssh/authorized_keys 'ssh-ed25519 …'` matches `find *` and runs. Prompt
|
|
||||||
injection from a page the model just read is enough to start it.
|
|
||||||
|
|
||||||
Fixed with `policy._ACTION`, refusing those flags in `subject()` rather than
|
|
||||||
trimming the allow list — a pattern cannot express "and no dangerous flags",
|
|
||||||
and "this one looks read-only" is exactly the reasoning that put `find *` there.
|
|
||||||
It costs a false refusal on `grep -- -delete`, which is the right direction to
|
|
||||||
be wrong in: a refusal asks, an allow does not.
|
|
||||||
|
|
||||||
**4. `0.0.0.0` walked past the loopback guard.** `_literal` answered from
|
|
||||||
`is_loopback`, and `0.0.0.0`/`::` are `is_unspecified` — so it returned a
|
|
||||||
*decided* `False`, which short-circuited `resolves_here` and skipped the DNS
|
|
||||||
half too. `connect()` to either goes to loopback on Linux, so an SSH profile
|
|
||||||
pointed at `0.0.0.0` reached this host's own sshd: the one spelling of "this
|
|
||||||
machine" that walked past the guard whose whole job is that sentence.
|
|
||||||
|
|
||||||
**5. Push endpoints skipped the SSRF guard.** `POST /api/push/subscribe`
|
|
||||||
checked `startswith("https://")` and nothing else, and `send_one` POSTed to it
|
|
||||||
with no `check_url` — the only outbound client in the codebase not going
|
|
||||||
through the guard. Delivery is triggered by the caller: send a message, close
|
|
||||||
the tab, and `_persist` announces it because nobody is following. Checked now
|
|
||||||
at subscribe **and** again before the POST, since the row outlives the first
|
|
||||||
check.
|
|
||||||
|
|
||||||
**6. A chat could be put in somebody else's folder.** `effective_system_prompt`
|
|
||||||
walks up from the chat through its folder and that folder's parents, so this
|
|
||||||
reads another account's system prompt through a field that looks like a tag.
|
|
||||||
Both paths had it, and `_new_chat`'s is the instructive one: it resolved the
|
|
||||||
folder, discarded it when it was not the caller's, and then stored the **raw
|
|
||||||
id** anyway — so the ownership check governed which *seeds* were applied and
|
|
||||||
not where the chat actually went.
|
|
||||||
|
|
||||||
### Clean
|
|
||||||
|
|
||||||
Checked and found sound: the branding CSS and custom-theme generation (ids and
|
|
||||||
colour values both validated on **read**, so a row written by hand still cannot
|
|
||||||
emit a malformed rule; served as `text/css` rather than inline, so there is no
|
|
||||||
HTML context to escape); the unauthenticated branding asset route (random
|
|
||||||
names, traversal guarded twice, magic-number sniffing, SVG excluded); sharing
|
|
||||||
authorisation on every route; the request file's contents reaching nothing;
|
|
||||||
`updates._git`'s fixed argv; and the container (non-root, no secret baked, no
|
|
||||||
docker socket, loopback only).
|
|
||||||
|
|
||||||
## Carried forward from earlier stages
|
|
||||||
|
|
||||||
Questions raised before the security stage, answered by it:
|
|
||||||
|
|
||||||
- the three hand-rolled redirect loops each re-run `check_url` per hop
|
|
||||||
(confirmed); does each also drop the secret when a hop leaves its origin?
|
|
||||||
- `admin_tools.py` calls `check_url(server.url, allow_private=True)`
|
|
||||||
unconditionally at save — a syntax check, with the per-row flag governing at
|
|
||||||
call time. Confirm that reading is right.
|
|
||||||
- the subagent restriction is the one gate that fails open unattended.
|
|
||||||
`_child_scope` forcing `ask:False`, `subagent:False`, `write:False` and
|
|
||||||
pinning `allow` to `SAFE_COMMANDS` is confirmed present; the question is
|
|
||||||
whether every path reaches it.
|
|
||||||
- `pyproject.toml` pins no upper bounds and `deploy/update.sh` runs
|
|
||||||
`pip install -e` on every update, so a breaking upstream release arrives on a
|
|
||||||
button press. pip's `only-if-needed` default limits the blast radius, which is
|
|
||||||
why this is a note rather than an emergency.
|
|
||||||
|
|
||||||
## Testing — *Stage 4*
|
|
||||||
|
|
||||||
2140 tests became **2283**, and writing them found four bugs that reading had
|
|
||||||
not — three of them by driving the JavaScript under a Node DOM stub, which is
|
|
||||||
the practice `CLAUDE.md` sets out and the reason it does.
|
|
||||||
|
|
||||||
- **The terminal dropped every keystroke after a reconnect.** `socket.onclose`
|
|
||||||
closed over the module-level variable rather than its own connection, and
|
|
||||||
`close()` queues its event: both reconnect paths close the old socket and
|
|
||||||
immediately open a new one, so the old one's `close` arrived afterwards and
|
|
||||||
nulled the *live* socket. Output kept coming, because `onmessage` is bound to
|
|
||||||
the object; every send gates on `socket && readyState === OPEN`. It also said
|
|
||||||
"Disconnected" about a shell that had just reconnected, because
|
|
||||||
`closedOnPurpose` had been cleared for the new connection.
|
|
||||||
- **Two scripts were loaded twice on `/messages`** — once by `base.html` and
|
|
||||||
again by the page. Each is an IIFE with its own state, so four keyboard
|
|
||||||
shortcuts toggled their panel twice and therefore did nothing, `/help` opened
|
|
||||||
two dialogs, and an `@` mention attached its file twice. A sweep now refuses
|
|
||||||
any template re-loading what `base.html` already has.
|
|
||||||
- **The microphone had no guard while the permission prompt was up.** State only
|
|
||||||
left `idle` after `getUserMedia` resolved, so every click in that window
|
|
||||||
opened another stream; only the last was stopped.
|
|
||||||
- **A shared skill took its name out of your own library.** `create` checked
|
|
||||||
uniqueness with `by_name`, which is scoped to what is *visible* — owned or
|
|
||||||
shared — against a `(owner_id, name)` constraint. `documents.create_base` next
|
|
||||||
door asks it correctly. Found by writing HTTP tests for `api/library.py`.
|
|
||||||
|
|
||||||
Also: `--ink-faint` failed the 4.5:1 contrast minimum in **both** themes (3.85
|
|
||||||
in Moria, 3.19 in Shire), measured by computing the ratios in a headless browser
|
|
||||||
rather than by eye. Raised in both.
|
|
||||||
|
|
||||||
**Verified rather than assumed:** the suite now runs on Python **3.11, 3.12 and
|
|
||||||
3.14**. It had only ever run on 3.14, while `Dockerfile` ships 3.12 and
|
|
||||||
`requires-python` claimed 3.11 — the interpreter most people would actually run
|
|
||||||
was the one nothing had tested. Both pass.
|
|
||||||
|
|
||||||
One measurement worth keeping honest: the `slow` marker splits 365 tests that
|
|
||||||
stand up a real server, shell, PTY or git repository, but the fast subset is
|
|
||||||
111s against 140s for everything. The split is for CI, not for a dramatic
|
|
||||||
saving.
|
|
||||||
|
|
||||||
## Still open after the testing stage
|
|
||||||
|
|
||||||
- `api/library.py`: 28 routes, effectively zero HTTP coverage. Highest exposure
|
|
||||||
— auth, ownership and permission gates all live at that boundary.
|
|
||||||
- No real `sync_schema` upgrade test exists. `conftest.py` runs `create_all`
|
|
||||||
then `sync_schema`, so the differ is only ever exercised as a no-op. Hard rule
|
|
||||||
4 has no other safety net.
|
|
||||||
- `cli.py`: zero tests, and `create-admin` is the documented bootstrap.
|
|
||||||
- SSE has three end-to-end tests; `services/sse.py` has none.
|
|
||||||
- `composer.js`, `terminal.js` and `audio.js` — 1,188 lines — have no assertion
|
|
||||||
of any kind.
|
|
||||||
- The suite has only ever run on Python 3.14. The Dockerfile ships 3.12 and
|
|
||||||
`requires-python` claims 3.11.
|
|
||||||
Reference in New Issue
Block a user