Show what a reply cost, live and afterwards

Tokens, how full the context is, and tokens per second -- as chips under
each assistant bubble, updating while the reply streams and still there when
it finishes.

The numbers come from one Metrics object built either from the generation
still being written or from the row it left behind. That is the point rather
than tidiness: the finished bubble is re-rendered from the database the
instant the stream ends, so two code paths would make the figures visibly
jump at exactly the moment someone is watching them. Here the only thing
that changes is that an estimate may become exact.

Message.usage_json has existed and been dead since the schema was written.
It is the store.

Two counts that look like one. prompt and completion are summed across tool
rounds -- what the reply cost. context_tokens is overwritten each round with
that round's prompt plus completion -- what the window actually holds. A
three-round reply pays for its prompt three times and only ever occupies the
window once, so a single number would be wrong for one of the two questions.

Generation gains started_at as a field rather than a local in _run, because
_follow is a different function that sees only the Generation and otherwise
has nothing to compute a live speed against. It also carries a prompt
estimate taken before the first chunk, since real usage arrives in one chunk
at the very end and a percentage that appears only after the reply is
useless.

Everything is marked with a tilde when the endpoint reported nothing, and
the percentage is simply absent when no context length is set: unknown has
to stay tellable from small, and a percentage of an unknown total is a
made-up number in a place people trust numbers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 00:40:44 +02:00
parent ff58ada6bf
commit b8618b0c91
8 changed files with 485 additions and 1 deletions
+14
View File
@@ -21,6 +21,7 @@ from lembas.services import audio as audio_service
from lembas.services import chat as chat_service
from lembas.services import files as files_service
from lembas.services import generation as generation_service
from lembas.services import metrics as metrics_service
from lembas.services import sse
from lembas.services.markdown import escape_text, render_markdown
from lembas.web.templating import render, templates
@@ -257,6 +258,7 @@ async def _follow(chat_id: str, message_id: str) -> AsyncIterator[str]:
yield sse.event("tools", _tool_activity(generation.tool_events))
if generation.content:
yield sse.event("render", render_markdown(generation.text))
yield sse.event("metrics", _metrics_html(generation))
last_frame = time.monotonic()
if generation.done:
@@ -314,6 +316,18 @@ async def _follow(chat_id: str, message_id: str) -> AsyncIterator[str]:
yield sse.event("close", "")
def _metrics_html(generation) -> str:
"""The metric chips for a reply still being written.
Built from the same Metrics object the finished bubble uses, so the numbers
do not jump when the stream ends -- the only thing that changes is that an
estimate may have become exact.
"""
return templates.get_template("chat/_metrics.html").render(
{"metrics": metrics_service.from_generation(generation)}
)
def _thread_context(db: DBSession, chat: Chat, user: User) -> dict:
"""Everything chat/_thread.html needs to render the conversation."""
messages = list(