A title call that could not survive a model that thinks
Reported: chat names never regenerate after the first reply. They were regenerating; the request was being made and the answer thrown away. `complete()` returns `message.content` verbatim, and a model that emits `<think>` inline puts its thinking in exactly the field the title is read from. So the title came back as "<think>Okay, the user wants a short title for" -- or, once the too-long guard caught that, as the first prompt trimmed, which is indistinguishable from titling never having run. That is what was being seen. Underneath it, `max_tokens: 24`. Ample for six words, and nowhere near enough for a model that reasons first: the budget goes on thinking and the content field comes back empty or holding an unclosed tag. Too small is not a shorter title, it is no title at all. Both fixed: the reply goes through `reasoning.strip_reasoning`, and the budget is `TITLE_MAX_TOKENS` with room to think. Reproduced first against the four shapes an endpoint actually answers with -- three of them were broken -- and the tests are written from those. What I did *not* do is ask for a low reasoning effort on the call, which would make it much cheaper and was the obvious move. `reasoning_effort` and `chat_template_kwargs` appear only where somebody has opted in, so that a provider strict about unknown parameters sees exactly the request it always did. An LLMError here is caught and turned into a fallback title -- so a 400 would be titling silently switching itself off, which is the failure this commit exists to fix. The token budget makes the room instead. The shipped prompt now asks for a leading emoji, as requested. Asked for rather than assumed: a model that ignores it gives a title without one, and an administrator who does not want them clears the word. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,7 +20,7 @@ lembas info # paths + counts, useful when confused
|
||||
lembas secret-key # generate LEMBAS_SECRET_KEY
|
||||
lembas create-admin # create or promote an admin
|
||||
|
||||
pytest # 1403 tests, ~87s
|
||||
pytest # 1408 tests, ~87s
|
||||
# PLAN.md tracks what is and is not built
|
||||
ruff check . # lint (line length 100)
|
||||
python scripts/build_artwork.py # regenerate artwork (SVG + PWA icons;
|
||||
@@ -210,6 +210,23 @@ treats a bare `Mapped[list]` as a scalar and hands back `None` instead of `[]`.
|
||||
Always write `Mapped[list[Group]]`, with a `TYPE_CHECKING` import if the class
|
||||
lives in another module.
|
||||
|
||||
**Auto-titling is a request like any other, and it was reading the wrong field
|
||||
of one.** `complete()` hands back `message.content` verbatim, and a model that
|
||||
emits `<think>` inline puts its thinking in exactly that field -- so a title
|
||||
came back as "<think>Okay, the user wants a short title for", or, once the
|
||||
too-long guard caught that, as the first prompt trimmed, which looks precisely
|
||||
like titling never having run. `generate_title` puts the reply through
|
||||
`reasoning.strip_reasoning` now. The budget was 24 tokens, which is ample for
|
||||
six words and nowhere near enough for a model that thinks first: too small is
|
||||
not a shorter title, it is no title, because the thinking consumes the budget
|
||||
and content comes back empty. It is `TITLE_MAX_TOKENS` and generous.
|
||||
|
||||
Deliberately **not** `apply_effort(body, "low")`, tempting as that is. Those two
|
||||
fields appear only where somebody has opted in, precisely so a provider strict
|
||||
about unknown parameters sees the request it always did -- and an LLMError here
|
||||
is caught and turned into a fallback title, so a 400 would be titling silently
|
||||
switching itself off. The token budget is what makes room for the thinking.
|
||||
|
||||
**Reasoning arrives two ways.** A `reasoning_content` delta field (llama.cpp,
|
||||
llama-swap, vLLM) or `<think>` tags inline in `content` (Ollama and friends).
|
||||
`services/reasoning.py` handles the second with a streaming splitter, because
|
||||
|
||||
Reference in New Issue
Block a user