Jaroslav Beneš
17f3fa1946
A long conversation eventually just stops working. Compaction summarises the earlier turns and sends the summary in their place. The messages are kept. They stay in the transcript behind a collapsed divider and simply stop being part of the request, which is what makes the button safe to press and automatic compaction safe to have at all: a summary that came out badly is a bad turn, not a lost conversation. Stored on the Chat, not as a synthetic Message. A synthetic row needs a role -- `system` breaks the one-system-message rule the moment build_messages emits it beside the harness, and user/assistant makes it a turn people can edit, regenerate from and copy, indistinguishable from a real one in all four places a bubble is rendered. Worse, "editing rewinds, it does not branch" would silently delete it and leave no marker that compaction had happened at all. The summary goes out as a user turn and an assistant turn, not one. A leading assistant breaks templates requiring the first non-system message to be user; a lone leading user produces user, user whenever the kept history starts on a user turn -- which it always does, because the cutoff lands on a finished reply. compacted_through_id is a plain id rather than a foreign key: migrations.py compiles only the column type, so a REFERENCES clause would exist on a fresh database and not on an upgraded one, and a constraint half the fleet has is worse than none. cutoff_message validates it on every read instead, and a rewind past the boundary clears it. Compacting again summarises only the delta, with the previous summary supplied to be subsumed. Re-summarising the whole chat each time grows quadratically and eventually exceeds the window it is protecting. Automatically at the top of _run, not in post_message: that route's contract is to return immediately and leave the slow part to a resumable connection, and it also means build_request is called once, after compaction, with no second assembly path. The trigger is the last reply's recorded usage plus an estimate of the new turn -- retrospective because true prompt_tokens are only knowable after a response, plus the delta because otherwise fifty thousand characters pasted into the composer overflow a window that read 90% last turn. It never fires when the context length is unknown. It does fire on estimated counts, which is safe here precisely because nothing is lost. _maybe_compact never raises: a failure logs and sends the uncompacted request. A `status` event says "Summarising earlier messages…" in the meantime, because a silent multi-second pause before the first token is what a hang looks like. The wording is three fragments under Admin - Prompts. Clearing task.compact turns compaction off entirely. Also adds compaction.moment(): SQLite does not store the offset, so a row loaded from disk is naive while one in the session's identity map keeps its tzinfo, and comparing the two raises. Every comparison here is between exactly those. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13 KiB
LLeMbas — plan and status
Where the project is, what is deliberately not built yet, and the decisions
that would be expensive to revisit. Kept current as work lands; the detail of
how things work lives in CLAUDE.md.
Status: usable daily. Streaming chat, attachments, reasoning, tool calling
with web search, a knowledge library, notes, memory and skills, speech in and
out, users and groups, model administration, installable as an app. 437 tests,
ruff clean.
The shape of it
A self-hosted web UI for OpenAI-compatible endpoints, written in Python, themed after Middle-earth.
| Stack | FastAPI + Jinja + htmx + a little Alpine |
| Build step | none — no Node, no npm, no CDN at runtime |
| Database | SQLite, schema synchronised additively at startup |
| Deployment | systemd unit + nginx vhost, one worker |
These are load-bearing. Dropping the no-build rule or moving off SQLite would be a different project, not a refactor.
Done
Chat
- Streaming replies over server-sent events
- Markdown renders progressively — re-rendered whole every 100ms rather than appending tokens, because a list or code fence is only correct once its context exists
- Syntax highlighting (Pygments), sanitised with nh3
- Generation runs in the background — a task, not the request. Navigate away, open another chat, close the tab: the reply keeps being written and reattaching replays the whole state
- Stop — the send button becomes Stop while writing; what arrived is kept
- Rewind — edit one of your own turns and the conversation runs on from there. Truncates rather than branching
- Copy, regenerate, automatic chat titles
- Chats created on first message, so an abandoned composer leaves nothing
- Unread indicator — a green dot and a toast when a reply lands while you were elsewhere
- Folders, arbitrarily nested; deleting one keeps the chats inside it
- Per-reply metrics — tokens, context used as a percentage, tokens/second,
live while streaming and kept afterwards. Estimated with a
~when the endpoint reports no usage - Compaction — a button, and automatically at a configurable percentage of the model's context. Summarised turns are kept and collapsed, not deleted
- Temporary chats — never listed, swept after a day, with a Keep button
- An admin-only request inspector beside the thread
Tools
- Tool calling — one reply is a bounded loop of requests, not one request. Text produced before a call is kept
- Web search as the first tool: DuckDuckGo (no setup), SearXNG or Firecrawl, chosen in the admin area
- Only offered to models flagged
tools, because an endpoint without support rejects the whole request rather than ignoring the array - Sources stay in the transcript; results are not replayed as context on the next turn, for the same reasons reasoning is not
The library
- Knowledge bases — documents, images and saved web pages, grouped into named collections and ingested through the same pipeline as chat attachments, searched with SQLite FTS5
- A chat can be pointed at particular bases, so "answer from the contracts folder" is a different question from "answer from everything I have"
- Notes — longer things the model writes down and searches later; editable by hand, because they are yours
- Memory — short facts, injected on every turn to a budget rather than searched, and managed in your settings
- Skills — saved procedures. Only the name and description are injected; the body is fetched when the model decides it applies
- A model may write and revise its own notes, memories and skills. Every skill revision is kept, attributed and revertible — the safety story is a record and a way back, not a gate
- Sharing — a knowledge base, a note or a skill can be shared with a group or with named people, read-only. One visibility rule, and administrators do not bypass it. Documents are shared through their base
- The harness — an operational prompt assembled from what a model actually has, so the tools get used rather than ignored
- Attach menu: file, image, a web page fetched on the spot, or a document from the library
Audio
- Dictation — record in the composer, transcribed by any OpenAI-shaped
/v1/audio/transcriptionsendpoint. The recording never touches disk - Read aloud — any
/v1/audio/speechendpoint, with the voice list discovered from the server where it offers one - Instance defaults in Admin, per-reader overrides in Settings — voice, speed, dictation language, and whether replies play automatically
Models and reasoning
- OpenAI-compatible connections with encrypted keys and model discovery
- Reasoning display —
reasoning_contentand inline<think>tags, collapsed by default, labelled with how long it took, never replayed as context - Model admin as a list plus a page per model; scales to hundreds
- Ordering, pinning (a sidebar shortcut, not a reordering), instance default, per-user default, images, capability flags
- Custom model picker showing avatars, descriptions and capabilities
Attachments
- Drag, paste or pick images, PDFs and text files
- Images downscaled and sent to vision models as content parts
- PDF and text extracted at upload and placed in the prompt
- Type decided by inspecting bytes, random names on disk, non-images served
as downloads with
nosniff - No OCR: a scanned PDF says so rather than silently contributing nothing
People
- Accounts, argon2, revocable server-side sessions, self-service password change
- Users and groups with permissions that union rather than override
- Model access restricted to chosen groups
- Registration toggle, instance settings stored in the database
Prompts
- Three layers — instance, model, chat — with the most specific winning outright rather than being concatenated
- Every injected fragment editable at
/admin/prompts: the tool guidance, the memory and skill sections, the seam above the authored prompt, and the request that names a chat {{variables}}with a legend, values shown as they currently resolve, and pass-through for anything that is not one- A preview of the whole assembled system message, including unsaved edits
- Defaults in code and overrides in the database, so improving a default still reaches an instance that never edited it
Suggestions
- Admin-managed cards on the new-chat screen; three seeded once at startup
Interface
- Installable — manifest, generated PWA icons, a service worker for the
shell and a themed offline page. The worker deliberately never touches
/api/: a reply is an event stream and caching one breaks it - Two themes (
moria,shire) from one set of design tokens - Every control sized from
--control-h, so rows line up by construction - Toasts and dialogs of our own; no
window.confirmanywhere - Original SVG artwork generated from a single source
Operations
- Additive schema sync — new tables and columns applied at startup
deploy/— systemd unit and nginx templates, install and update scripts
Not built yet
In the order they are likely to be worth doing.
Custom tools and MCP servers
An MCP client managing configured servers, their tools surfaced alongside the
built-in ones. The loop they plug into exists now — services/tools.py is a
registry of thirteen tools and services/generation.py already runs bounded
rounds — so this is a client and an admin screen rather than a change to how
chat works.
Agentic execution
Two modes, as originally specified:
- local — subprocess on the machine LLeMbas runs on
- remote — SSH connection profiles, with
shell.run/fs.read/fs.write
Needs a confirmation model before it does anything. Note that the systemd unit
is deliberately only ProtectSystem=full rather than strict because of
this — revisit the hardening when the real filesystem needs are known.
Image generation
Left until last from the start, as it needs heavy customisation. ComfyUI is already running on this machine and is the obvious first target.
Smaller things
- OCR for scanned PDFs
- Conversation branching —
Message.parent_idexists unused; needs a UI for choosing between versions, which is why rewind truncates for now - Chat export (Markdown, JSON)
- Semantic search in the library — the retrieval service is one call, so an embedding backend can go behind it without touching the tools or the UI
- Archived chats — the column exists, nothing surfaces it
- Per-user quotas
Known limits
Worth knowing before they surprise someone.
One worker. The generation registry and the stop mechanism are in-process. Running several workers needs that state in the database or a broker, because the request following a reply would not necessarily land in the process writing it.
A restart abandons replies in flight. Shutdown cancels them and keeps what each had. There is no resume.
Schema changes are additive only. New tables and columns apply themselves;
renames, drops and retypes are manual against the SQLite file. MANUAL_STEPS
in db/migrations.py is where such a step gets recorded.
Attachments live on disk, unreferenced files are swept at startup. No deduplication, no size quota.
Unread is polled every 10 seconds. A push channel would be more responsive but means an always-on connection per tab for the sake of a green dot.
Installing needs HTTPS or localhost. Service workers are unavailable over plain HTTP, so a LAN install without TLS is a normal browser tab. The microphone is unavailable for the same reason.
Tool calling needs a model that supports it. The tools flag is an
administrator's assertion, not something endpoints reliably advertise. Set it on
a model that cannot, and its replies fail rather than degrade.
Library search is keyword, not semantic. FTS5 ranks well and needs no dependency or embedding endpoint, but "how do I get paid" will not find a document that says "invoicing".
A model can write its own skills, and they take effect at once. Marked as model-authored and fully revertible, but a model that has just read a hostile page could save a skill that outlives the conversation. The mitigation is that it is visible and undoable, not that it was prevented.
Deliberate decisions
Recorded because each looks like an oversight until you know the reason.
- No JavaScript build step. Browser libraries are hash-pinned and committed. A self-hosted tool should work offline and not report page views to a CDN.
- Permissions union, never deny. With denies, "why can this user not do X" cannot be answered without simulating every group.
- System prompts replace, never stack. Two layers that disagree give the model contradictory instructions and nobody can tell which is losing.
- Rewind truncates, does not branch. Branching needs a UI for choosing between versions; "go back and try again from here" is what was asked for.
- Pinning is a shortcut, not an ordering. A picker whose order silently differs from the admin screen is confusing.
- Images only reach models marked
vision. Not graceful degradation: most endpoints reject the entire request rather than ignoring an image part. Tools are gated the same way, for the same reason. - Sharing grants reading, never writing. Two people editing one note with no history and no merge is worse than the inconvenience of copying it.
- Memory is never shareable. A record about a person is not content to hand round.
- Knowledge attached to a message is copied, not referenced. History must not change under a conversation because a document was edited later.
- The harness is prepended to the authored prompt, not a fourth layer. It describes the machinery; the authored layers describe the behaviour. Only one authored layer still wins.
- Tool results are not replayed. Like reasoning: the answer already contains what the model made of them, and replaying stale results into every later request wastes the window and sends small models into search loops.
- The service worker caches the shell, never a page with a user in it. A cached conversation would be a snapshot that silently went stale, belonging to whoever was signed in last.
- Markdown rendered server-side. One code path produces the streamed and the stored view, so they cannot disagree.
- This repository is public. Deployment hostnames, ports and paths stay out
of it;
deploy/is templates, and the real values live in private notes.