Reading the answer instead of asking somebody to know it
llama-server publishes the loaded model's Jinja chat template on /props, and
that template is the very thing that rejects a reasoning effort it does not
recognise -- so the accepted set is written down, authoritatively, in a place
this application can simply read. There is a button on the model's page that
does.
The parser handles both shapes a template uses: the values inline in the test
that rejects them (Bonsai), and a named list set elsewhere with nothing near
the mention spelling them out (gpt-oss). It is deliberately conservative,
because a wrong answer here silently removes a level somebody is entitled to:
only known efforts count, an unrelated list of quoted strings is ignored, and a
single match is read as a default -- `{%- set reasoning_effort = 'medium' %}`
-- rather than as a vocabulary of one.
An endpoint with no such route says so. OpenAI and vLLM do not publish a
template, and "this cannot tell us" must not be recorded as "this model accepts
nothing".
/props sits at the server root, beside the OpenAI-compatible surface rather
than inside it, so a base URL written as .../v1 needs the suffix stripped.
Getting that wrong is a silent 404 that looks like detection simply not
working, so there is a test on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -68,6 +68,19 @@ class Endpoint:
|
||||
base = f"{base}/v1"
|
||||
return f"{base}/{path.lstrip('/')}"
|
||||
|
||||
def root_url(self, path: str) -> str:
|
||||
"""A URL at the *server's* root rather than under `/v1`.
|
||||
|
||||
llama-server's own endpoints -- `/props` is the one that matters here --
|
||||
sit beside the OpenAI-compatible surface, not inside it. A base URL may
|
||||
be written either way (`http://host:8080` or `.../v1`), so the suffix is
|
||||
stripped rather than assumed absent.
|
||||
"""
|
||||
base = self.base_url.rstrip("/")
|
||||
if base.endswith("/v1"):
|
||||
base = base[: -len("/v1")]
|
||||
return f"{base}/{path.lstrip('/')}"
|
||||
|
||||
def headers(self) -> dict[str, str]:
|
||||
headers = {"Content-Type": "application/json", **self.extra_headers}
|
||||
# Local endpoints frequently need no key at all; sending an empty
|
||||
@@ -77,6 +90,30 @@ class Endpoint:
|
||||
return headers
|
||||
|
||||
|
||||
async def fetch_chat_template(endpoint: Endpoint) -> str:
|
||||
"""The model's own Jinja chat template, from llama-server's `/props`.
|
||||
|
||||
The one place the truth about a model's accepted values is actually
|
||||
written down: `/props` returns `chat_template` verbatim, and that template
|
||||
is what raises when it meets a `reasoning_effort` it does not know.
|
||||
|
||||
Returns "" rather than raising for anything that is not a llama-server --
|
||||
OpenAI, vLLM and the rest have no such route, and "this endpoint cannot
|
||||
tell us" is a normal answer here, not a failure.
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
response = await client.get(
|
||||
endpoint.root_url("props"), headers=endpoint.headers()
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except (httpx.HTTPError, ValueError, json.JSONDecodeError):
|
||||
return ""
|
||||
template = payload.get("chat_template") if isinstance(payload, dict) else ""
|
||||
return template if isinstance(template, str) else ""
|
||||
|
||||
|
||||
def describe_http_error(exc: httpx.HTTPStatusError) -> str:
|
||||
"""Turn an upstream error response into something worth reading.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user