Homer
0514568df0
The testing pass: 2140 tests to 2283, and four bugs that no amount of reading had turned up. Three came from driving the JavaScript under a Node DOM stub, which is the practice CLAUDE.md sets out and this is the reason it does. The terminal dropped every keystroke after a reconnect. `onclose` closed over the module-level socket rather than its own, and close() queues its event -- so the old socket's close arrived after a new one was assigned and nulled the live one. Output kept coming, because onmessage is bound to the object, while every send gates on the variable. It also announced "Disconnected" about a shell that had just reconnected. Two scripts were loaded twice on /messages, once by base.html and again by the page. Each is an IIFE with its own state, so four keyboard shortcuts toggled their panel twice and therefore did nothing, /help opened two dialogs, and an @ mention attached its file twice. A sweep refuses any template re-loading what base.html has. The microphone had no guard while the permission prompt was up, so each click opened another stream and only the last was ever stopped. And a skill shared with you took its name out of your own library: create checked uniqueness against what is *visible* rather than what is owned, against a (owner_id, name) constraint, and told you to edit a row you cannot edit. --ink-faint failed the contrast minimum in both themes -- 3.85 and 3.19 against 4.5 -- so the smallest text on every screen was the hardest to read. Measured in a headless browser rather than judged by eye. And the suite runs on 3.11 and 3.12 now as well as 3.14. It had only ever run on 3.14 while the image ships 3.12 and the packaging claimed 3.11: the interpreter most people would run was the one nothing had tested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
155 lines
7.5 KiB
Markdown
155 lines
7.5 KiB
Markdown
# The manual pass, before a release
|
|
|
|
What the suite cannot reach. Everything here needs a real endpoint, a real
|
|
machine, real hardware or a real browser with a person in front of it — which is
|
|
to say, everything where the failure is "it works but nobody could use it".
|
|
|
|
Run it against the live instance. Tick nothing you have not actually seen.
|
|
|
|
Times are rough and assume things are already configured.
|
|
|
|
---
|
|
|
|
## 1. A model answers at all (5 min)
|
|
|
|
- [ ] Send a message. The reply streams in **as it is written**, not all at once
|
|
at the end. (A reply that arrives complete means something is buffering —
|
|
a proxy, or a worker that collected the response.)
|
|
- [ ] The thinking block, on a reasoning model: opens, shows a duration, and the
|
|
duration is not the same number on every round.
|
|
- [ ] Stop mid-reply. What arrived is kept, the bubble is marked stopped rather
|
|
than errored, and the composer returns to Send.
|
|
- [ ] Navigate away mid-reply and come back. The reply is still running and the
|
|
transcript catches up.
|
|
- [ ] Close the tab mid-reply, reopen the chat. The reply finished without you.
|
|
- [ ] Regenerate a reply. The old one is replaced, not appended.
|
|
- [ ] Edit an earlier message. Everything after it goes, and the conversation
|
|
runs on from there.
|
|
|
|
## 2. The composer (5 min)
|
|
|
|
- [ ] Type `/` — the menu appears on the **first** press, not the second.
|
|
- [ ] Choose a command with Enter. The box is left empty, not holding `/help`.
|
|
- [ ] Tab completes the highlighted command.
|
|
- [ ] `//` escapes: the message sends as written.
|
|
- [ ] A message that merely starts with a slash and is not a command **sends**.
|
|
- [ ] Type `@` and pick a file. The token stays in the sentence *and* a chip
|
|
appears.
|
|
- [ ] The highlighting behind `/` and `@` sits exactly over the text, at every
|
|
width, and does not drift as the box grows.
|
|
- [ ] Send. The highlighting clears with the box rather than a keystroke later.
|
|
- [ ] `Ctrl/⌘+Enter` sends from anywhere in the form.
|
|
- [ ] In an agent chat, the toolbar stays **one row** at every window width.
|
|
Send and the microphone never wrap to a second line.
|
|
|
|
## 3. Attachments and images (10 min)
|
|
|
|
- [ ] Drag an image in. It is downscaled and the model can describe it.
|
|
- [ ] Paste a screenshot. Same.
|
|
- [ ] A PDF: the text reaches the model; a scanned one says so rather than
|
|
contributing nothing silently.
|
|
- [ ] Rename a `.txt` to `.png` and upload it. It is stored as text.
|
|
- [ ] Attach from the **new-chat screen**, send, then delete the chat. The file
|
|
is gone from `data/uploads/attachments`. *(This is the 0.9.10 fix; before
|
|
it, the row went and the file stayed.)*
|
|
- [ ] Generate an image, if a ComfyUI is configured. It appears in the chat, and
|
|
deleting the chat removes the file.
|
|
|
|
## 4. Agent chats — needs a real SSH host (15 min)
|
|
|
|
- [ ] Add a connection. The fingerprint is shown **before** anything is sent.
|
|
- [ ] Each mode does what it says: **Manual** shows everything first, **Edit**
|
|
writes freely but asks before commands, **Auto** asks nothing, **Plan**
|
|
changes nothing and ends with a plan.
|
|
- [ ] Approve, refuse, and *edit* a proposed command. The edited one is what
|
|
runs, and the transcript says so.
|
|
- [ ] "Always allow this" — the next matching command runs without asking.
|
|
- [ ] Open the terminal panel. Type. Close the panel and reopen: the session
|
|
survived and the scrollback is there.
|
|
- [ ] **Change the connection while the terminal is open**, then type. Every
|
|
keystroke still reaches the shell. *(This is the 0.9.12 fix — before it,
|
|
output kept arriving and input was silently dropped.)*
|
|
- [ ] Start a long command in the background, navigate away, come back. You are
|
|
told it finished.
|
|
- [ ] Open the canvas, pick a file by browsing rather than typing a path, edit
|
|
it, save. The file changed on the far side.
|
|
- [ ] Try to point a connection at `127.0.0.1` and at `0.0.0.0`. **Both refused**
|
|
unless an administrator has opened the switch.
|
|
|
|
## 5. Things that happen later (10 min, plus waiting)
|
|
|
|
- [ ] Ask the model to schedule something ten minutes out. It uses the tool
|
|
rather than writing a note, and says the timing back **in words**.
|
|
- [ ] Check the Scheduled list: the timing shown matches what you asked for, in
|
|
your timezone.
|
|
- [ ] Wait for it to fire. A report is filed, or a message arrives.
|
|
- [ ] With the tab **closed**, a scheduled run reaches you by push (if enabled).
|
|
- [ ] The dot, the tab-title count and the system notification do not all fire
|
|
at once for the same arrival.
|
|
|
|
## 6. Sharing and permissions — needs two accounts (10 min)
|
|
|
|
- [ ] Share a note with the second account. They can read it and cannot edit it.
|
|
- [ ] "Shared with me" lists it.
|
|
- [ ] The second account cannot see anything not shared with them, **including
|
|
as an administrator**.
|
|
- [ ] Delete the second account. No share anywhere still names it.
|
|
- [ ] Set a group quota, spend past it, and confirm the reply ends with an
|
|
explanation rather than an empty bubble.
|
|
|
|
## 7. Audio — needs real hardware (5 min)
|
|
|
|
- [ ] Dictate a message. `Alt+M` starts it; the transcript lands in the box and
|
|
the highlighting repaints.
|
|
- [ ] Press the microphone **three times quickly** while the permission prompt
|
|
is up. Only one recording starts, and the browser's recording indicator
|
|
goes out when you stop. *(0.9.12.)*
|
|
- [ ] `Alt+R` reads the last reply aloud.
|
|
- [ ] Read-aloud-automatically does not re-read an old reply when you reopen a
|
|
chat.
|
|
|
|
## 8. The look of it (10 min)
|
|
|
|
Both themes, and a custom one.
|
|
|
|
- [ ] Tab through a page with the keyboard. Every control shows where you are.
|
|
- [ ] Narrow the window to a phone width on `/admin/models`, `/admin/prompts`
|
|
and a chat. Nothing is cut off and nothing needs sideways scrolling.
|
|
- [ ] Hints and timestamps are readable, not grey-on-grey. *(0.9.12 raised
|
|
`--ink-faint` in both themes; this is the one to eyeball.)*
|
|
- [ ] Switch tabs on `/admin/prompts`. The page does not jump and no screenful
|
|
of nothing appears. *(0.9.10.)*
|
|
- [ ] Make a custom theme with four colours. It composes, and the focus rings
|
|
pick up the new accent.
|
|
- [ ] Install to the home screen. The icon and the name are the branded ones.
|
|
|
|
## 9. Upgrading (15 min)
|
|
|
|
The one nobody does until it matters.
|
|
|
|
- [ ] From a **copy** of a real 0.8.x database, start the new version. It boots,
|
|
the chats are there, and nothing in the log says a column is missing.
|
|
- [ ] `/admin/updates` shows a version rather than a sha, and the release notes
|
|
come from the tag.
|
|
- [ ] Press Update. The service restarts and comes back.
|
|
- [ ] Re-run `install.sh`. The channel does **not** move on its own. *(0.9.12.)*
|
|
- [ ] `sudo ls -l /usr/local/lib/lembas/update.sh` — owned by root. If systemd's
|
|
`ExecStart` still points inside the checkout, the helper is on the old
|
|
wiring and the script says so loudly when it runs.
|
|
- [ ] A fresh install into a container, from nothing, following the README only.
|
|
|
|
---
|
|
|
|
## What the suite already covers, so you do not have to
|
|
|
|
Not a suggestion to skip it — a note on where the machine has already looked, so
|
|
your time goes where it cannot.
|
|
|
|
- Every tool's gating, and that a chat can only narrow what it was granted
|
|
- The four agent modes against a real SSH server, and the approval loop
|
|
- Reply steps, metrics, compaction, queueing and rewind
|
|
- The schema upgrade, with rows, from an 0.8.1-shaped database
|
|
- Every library route at the HTTP boundary: ownership, sharing, deletes
|
|
- The SSRF guard on every outbound path
|
|
- The whole suite on Python 3.11, 3.12 and 3.14
|