Tokens, how full the context is, and tokens per second -- as chips under
each assistant bubble, updating while the reply streams and still there when
it finishes.
The numbers come from one Metrics object built either from the generation
still being written or from the row it left behind. That is the point rather
than tidiness: the finished bubble is re-rendered from the database the
instant the stream ends, so two code paths would make the figures visibly
jump at exactly the moment someone is watching them. Here the only thing
that changes is that an estimate may become exact.
Message.usage_json has existed and been dead since the schema was written.
It is the store.
Two counts that look like one. prompt and completion are summed across tool
rounds -- what the reply cost. context_tokens is overwritten each round with
that round's prompt plus completion -- what the window actually holds. A
three-round reply pays for its prompt three times and only ever occupies the
window once, so a single number would be wrong for one of the two questions.
Generation gains started_at as a field rather than a local in _run, because
_follow is a different function that sees only the Generation and otherwise
has nothing to compute a live speed against. It also carries a prompt
estimate taken before the first chunk, since real usage arrives in one chunk
at the very end and a percentage that appears only after the reply is
useless.
Everything is marked with a tilde when the endpoint reported nothing, and
the percentage is simply absent when no context length is set: unknown has
to stay tellable from small, and a percentage of an unknown total is a
made-up number in a place people trust numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A streamed completion carries no token counts unless you ask for them, and
`stream_options: {include_usage: true}` is how. Not every server implements
it, and an unknown key is a 400 from some -- the same hazard as sending a
tools array to an endpoint without support. So it is asked for once per base
URL per process, and an endpoint that refuses is remembered and retried
without it. The retry is safe because the status is checked before a single
line is read: nothing has been yielded, so there is nothing to duplicate.
chunk_usage() reads the resulting chunk. It needed no change to the loop
above it: a usage chunk carries `choices: []`, which is exactly the shape
delta_text, delta_reasoning, delta_tool_calls and finish_reason have always
returned early on. All-zero counts are treated as absent, because some
servers attach zeros to every chunk and the real numbers only at the end.
services/tokens.py is the fallback for endpoints that never report: four
characters to a token, counting the tools array because thirteen schemas is
a meaningful slice of a short window, and counting nothing for an image
because its cost depends on tiling and an invented number would be worse
than the omission. Crude on purpose -- a real tokeniser means one per model
family, for a figure that is displayed beside a tilde.
Nothing uses any of this yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A column rather than a key in capabilities_json, which is rebuilt wholesale
from the submitted checkboxes on every save and would destroy a number
living in it.
0 means unknown, and unknown has to stay tellable from small: the context
percentage and automatic compaction both refuse to act on a figure nobody
supplied. Filled in from /v1/models where the runner advertises it --
OpenRouter, vLLM and llama.cpp each spell it differently, so context_from()
reads the four spellings actually in use, accepts a quoted number but not
"8192 tokens", and rejects anything outside 256..10,000,000. Applied on
discovery only when nothing is set: a refresh must never undo a correction,
since an administrator sets this precisely because the endpoint was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Jaroslav Beneš