A streamed completion carries no token counts unless you ask for them, and
`stream_options: {include_usage: true}` is how. Not every server implements
it, and an unknown key is a 400 from some -- the same hazard as sending a
tools array to an endpoint without support. So it is asked for once per base
URL per process, and an endpoint that refuses is remembered and retried
without it. The retry is safe because the status is checked before a single
line is read: nothing has been yielded, so there is nothing to duplicate.
chunk_usage() reads the resulting chunk. It needed no change to the loop
above it: a usage chunk carries `choices: []`, which is exactly the shape
delta_text, delta_reasoning, delta_tool_calls and finish_reason have always
returned early on. All-zero counts are treated as absent, because some
servers attach zeros to every chunk and the real numbers only at the end.
services/tokens.py is the fallback for endpoints that never report: four
characters to a token, counting the tools array because thirteen schemas is
a meaningful slice of a short window, and counting nothing for an image
because its cost depends on tiling and an invented number would be worse
than the omission. Crude on purpose -- a real tokeniser means one per model
family, for a figure that is displayed beside a tilde.
Nothing uses any of this yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A column rather than a key in capabilities_json, which is rebuilt wholesale
from the submitted checkboxes on every save and would destroy a number
living in it.
0 means unknown, and unknown has to stay tellable from small: the context
percentage and automatic compaction both refuse to act on a figure nobody
supplied. Filled in from /v1/models where the runner advertises it --
OpenRouter, vLLM and llama.cpp each spell it differently, so context_from()
reads the four spellings actually in use, accepts a quoted number but not
"8192 tokens", and rejects anything outside 256..10,000,000. Applied on
discovery only when nothing is set: a refresh must never undo a correction,
since an administrator sets this precisely because the endpoint was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four pieces of work.
**Installable.** A manifest carrying the instance name, PWA icons rasterised
from the existing mark at design time, a service worker and a themed offline
page. The worker caches the shell only and bails out on /api/, /auth/, /admin/
and anything accepting text/event-stream -- passing a reply stream through a
worker turns it into one delivery at the end, or nothing. It is served from
GET /sw.js rather than the static mount because a worker's scope is the path it
came from.
**Send and Stop are one button.** They were two, and the hidden one was never
hidden: `.btn` is display: inline-flex, which outranks the browser's own
`[hidden] { display: none }`, so Stop sat permanently beside Send. app.css now
forces the attribute to win -- every control toggled with `hidden` depended on
that -- and the composer renders one button carrying both icons, with ui.js
flipping data-composer-action and the type with it.
**Audio.** Speech to text and text to speech against any OpenAI-shaped
/v1/audio/* endpoint: dictate into the composer, have a reply read out.
Instance settings in Admin, per-reader overrides in Settings, with the voice
list discovered from the server where it offers one. Recorded audio is capped
and never written to disk -- it is not an attachment, it has no owner, and
nothing would ever sweep it.
**Web search, as a tool.** This is the tool loop PLAN.md described as the real
work: one reply is now a bounded sequence of requests rather than one. The model
asks, the tool runs, the result goes back and it is asked again, up to three
rounds. Providers are DuckDuckGo (no setup), SearXNG and Firecrawl.
Two decisions worth stating. Tools are only offered to models flagged `tools`,
because an endpoint without support rejects the whole request rather than
ignoring the array -- the same reason images only reach models flagged
`vision`. And tool results are not replayed as context on the next turn, for the
same reasons reasoning is not: the answer already contains what the model made
of them, and replaying stale results into every later request wastes the window
and reliably sends a small model into a search loop. The sources stay visible in
the transcript instead.
Search results are untrusted third-party text and are treated as such: escaped,
and only http/https URLs rendered as links.
338 tests, ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four features, plus the schema machinery they needed.
**Schema sync.** The first live instance had data in it, and create_all
only creates missing *tables* -- a new column silently never appeared.
db/migrations.py now diffs the declared models against the database and
ALTER TABLE ... ADD COLUMN for what is missing, deriving a backfill
default from the column type (SQLite refuses a NOT NULL column without
one, and a Python-side `default=dict` cannot be expressed in DDL).
Verified against a copy of the live database: eight changes applied, all
rows preserved, second run a no-op. Renames, drops and retypes are still
manual and say so.
**Permissions.** A flat set of named booleans: an instance baseline
widened by each group the user belongs to. A group grants and never
denies -- with denies, "why can this user not do X" cannot be answered
without simulating every group. Admins bypass entirely, because an admin
can grant it back to themselves in two clicks and pretending otherwise
is theatre. Model *access* is separate: public, or granted to groups.
The picker is not the boundary -- switching a chat to a model you cannot
reach is a 403.
**Model settings.** Ordering, pinned-first, an instance default and a
per-user default, display names, descriptions, capability flags, and
uploaded images. Images are stored and served locally rather than by
URL: a remote URL makes every page render a request to a third party.
Uploads are validated by magic number, not the declared content type,
and stored under a random name. Models with no image get a generated
initial whose hue is derived from the model id, so it is stable.
**Reasoning display.** Streams into its own collapsible block above the
answer, labelled "Thought for 14 seconds", collapsed once finished, and
never replayed as context on the next turn. Two sources: the
reasoning_content delta field, and <think> tags inline in content -- the
latter needs a streaming splitter because the tags arrive split across
chunks. Models emitting no reasoning show nothing, via a :has() rule
rather than JavaScript. Verified against qwen35-9b on llama-swap: 694
reasoning events, 52 answer tokens, cleanly separated.
Two bugs found and fixed while testing:
- A bare `Mapped[list]` relationship is treated by SQLAlchemy as a scalar
and returns None instead of []. It needs the element type.
- FastAPI substitutes the default for an empty form value, so with
`x: str | None = Form(None)` a submitted `x=` is indistinguishable from
an absent field. That silently broke clearing a system prompt or a
temperature. update_chat now reads the raw form and checks key presence.
143 tests, ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LLeMbas now runs end to end. Register, add an OpenAI-compatible
connection, and hold a real streaming conversation organised into
folders. Verified against the local llama-swap instance.
Streaming is the one genuinely tricky part. Sending a message returns
two HTML fragments -- the user bubble and an empty assistant bubble
carrying an sse-connect -- and that attribute is the ONLY thing that
starts a generation. Rendering an incomplete assistant message as a
streaming shell falls out of the same template, which means loading a
page whose last reply never finished simply picks it up again.
Details worth knowing about, each commented where it matters:
- SSE payloads are split across several data: lines. A raw newline in
one data: line truncates the event, which shows up the first time a
model emits a code block.
- Markdown is rendered server-side by the same helper for both the page
and the final streamed frame, so the two cannot disagree. The fence
renderer is replaced outright rather than using markdown-it's
highlight option, which re-wraps output in a second <pre>.
- escape_text is html.escape, not nh3.clean_text: it escapes character
by character, so escaping stream chunks separately equals escaping
the whole string.
- The stream opens its own session via session_scope(); it outlives the
request handler and the dependency-scoped session may be closed.
- Deleting a folder keeps the chats inside it (FK is SET NULL). Losing
a conversation to a mis-clicked folder delete is unforgivable.
- Login failures use one message for "no such account" and "wrong
password" so the form cannot enumerate registered addresses.
Also adds deploy/ for the gamebox install at https://chat.lan: system
unit, nginx vhost with buffering off (buffering on turns streaming into
one lump at the end), and install/update scripts following the same
service-user and /srv bind-mount conventions as llama-swap and comfyui.
70 tests, ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Jaroslav Beneš