An effort the model had never heard of
Reported from a live instance, on Bonsai: Jinja Exception: Unexpected reasoning effort high. Supported types are xhigh (default), medium, and low. Effort goes out two ways because no single field works, and the second -- chat_template_kwargs -- is not a parameter the server interprets. It is rendered into the model's own chat template, which does not ignore a value it does not know: it calls raise_exception, and the request dies before a token. So a perfectly ordinary option, drawn by this application in its own menu, took the whole reply with it. The vocabulary is per model and nobody agrees. gpt-oss takes low/medium/high. Bonsai takes low/medium/xhigh and refuses high. OpenAI has added minimal, xhigh and max at different points, and which of them a given model accepts varies again. One global tuple was going to be wrong for somebody whatever it held. A model carries its own list now, and the picker, the slash command and the request builder all read it. A column rather than a key in capabilities_json, for the reason context_length is one: that dict is rebuilt wholesale from the submitted checkboxes on every save. And it corrects itself. A refusal retries the reply once without the effort rather than losing it -- safe only because the template renders before any token, so nothing has been emitted, and there is a guard that keeps it that way -- then narrows the model's list. Bonsai's error states what it does take, so that is what gets stored. Note the parser bug, because it is a good one: "high" is a substring of "xhigh", so reading the advertised list by substring learned `high` from a sentence explaining that `high` is the problem. Whole words now, with a test named after it. /effort reads its levels off the picker instead of a second copy of the list kept in the browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -387,7 +387,15 @@ def build_request(
|
||||
):
|
||||
body["tool_choice"] = {"type": "function", "function": {"name": force_tool}}
|
||||
|
||||
apply_effort(body, (chat.params_json or {}).get("reasoning_effort"))
|
||||
# The model's own vocabulary, looked up here rather than passed in: every
|
||||
# caller of `build_request` would otherwise have to remember, which is the
|
||||
# trap `audio_service.template_flags` fell into.
|
||||
chat_model = model_for(db, chat)
|
||||
apply_effort(
|
||||
body,
|
||||
(chat.params_json or {}).get("reasoning_effort"),
|
||||
efforts_for(chat_model) if chat_model is not None else None,
|
||||
)
|
||||
return body
|
||||
|
||||
|
||||
@@ -405,7 +413,42 @@ def build_request(
|
||||
# an effort on sends neither field and is byte-for-byte what it was. An endpoint
|
||||
# strict about unknown parameters will refuse the extra one -- but on a chat
|
||||
# somebody deliberately set an effort on, not on every chat in the instance.
|
||||
EFFORTS = ("low", "medium", "high")
|
||||
# Every reasoning effort this application understands, and the subset a model
|
||||
# gets when nobody has said otherwise.
|
||||
#
|
||||
# 🚨 These are two different questions and conflating them is what broke a
|
||||
# chat on Bonsai: `EFFORTS` was `("low", "medium", "high")` and was used both to
|
||||
# validate what somebody chose *and* to decide what to offer, so a model whose
|
||||
# vocabulary is low/medium/**xhigh** could not be given its own top setting,
|
||||
# and the one it was given -- `high` -- made its chat template call
|
||||
# `raise_exception` and took the whole reply with it.
|
||||
#
|
||||
# The known list is the union across providers, which have not agreed: OpenAI
|
||||
# has added `minimal`, `xhigh` and `max` at different points; gpt-oss takes
|
||||
# low/medium/high; Bonsai takes low/medium/xhigh and refuses high. `none` is
|
||||
# deliberately absent -- this application already spells that `off`, and two
|
||||
# spellings of off is the failure this codebase keeps cataloguing.
|
||||
EFFORTS = ("minimal", "low", "medium", "high", "xhigh", "max")
|
||||
|
||||
# What a model is offered when its own list is empty. The three every reasoning
|
||||
# model since the first one has understood.
|
||||
DEFAULT_EFFORTS = ("low", "medium", "high")
|
||||
|
||||
|
||||
def efforts_for(model) -> tuple[str, ...]:
|
||||
"""The efforts this model accepts, in the order they should be offered.
|
||||
|
||||
A model's own list when an administrator has set one or the endpoint has
|
||||
taught us one (see `generation._narrow_efforts`), and the common three
|
||||
otherwise. Filtered against `EFFORTS` on the way out, so a value stored by
|
||||
an older release -- or learned from an endpoint that advertised something
|
||||
this application has never heard of -- cannot reach a request body.
|
||||
"""
|
||||
stored = list(getattr(model, "reasoning_efforts", None) or [])
|
||||
chosen = [value for value in stored if value in EFFORTS]
|
||||
if not chosen:
|
||||
return DEFAULT_EFFORTS
|
||||
return tuple(value for value in EFFORTS if value in chosen)
|
||||
|
||||
|
||||
def resolved_effort(chat) -> str:
|
||||
@@ -427,9 +470,19 @@ def resolved_effort(chat) -> str:
|
||||
return value if value in EFFORTS else ""
|
||||
|
||||
|
||||
def apply_effort(body: dict[str, Any], effort: str | None) -> None:
|
||||
"""Put a chosen reasoning effort into a request body, in both forms."""
|
||||
if not effort or effort not in EFFORTS:
|
||||
def apply_effort(
|
||||
body: dict[str, Any], effort: str | None, supported: tuple[str, ...] | None = None
|
||||
) -> None:
|
||||
"""Put a chosen reasoning effort into a request body, in both forms.
|
||||
|
||||
`supported` is the model's own vocabulary. An effort outside it is dropped
|
||||
rather than sent, because the second form below is not advisory: it reaches
|
||||
the model's Jinja chat template, and a template that does not know the value
|
||||
raises rather than ignoring it -- which fails the whole request, not the
|
||||
parameter.
|
||||
"""
|
||||
allowed = supported or DEFAULT_EFFORTS
|
||||
if not effort or effort not in allowed:
|
||||
return
|
||||
body["reasoning_effort"] = effort
|
||||
kwargs = dict(body.get("chat_template_kwargs") or {})
|
||||
|
||||
Reference in New Issue
Block a user