Models know how much context they hold

A column rather than a key in capabilities_json, which is rebuilt wholesale
from the submitted checkboxes on every save and would destroy a number
living in it.

0 means unknown, and unknown has to stay tellable from small: the context
percentage and automatic compaction both refuse to act on a figure nobody
supplied. Filled in from /v1/models where the runner advertises it --
OpenRouter, vLLM and llama.cpp each spell it differently, so context_from()
reads the four spellings actually in use, accepts a quoted number but not
"8192 tokens", and rejects anything outside 256..10,000,000. Applied on
discovery only when nothing is set: a refresh must never undo a correction,
since an administrator sets this precisely because the endpoint was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 00:33:50 +02:00
parent 6dd13b2e9d
commit 2fe736aa6a
6 changed files with 205 additions and 2 deletions
+35
View File
@@ -155,6 +155,41 @@ async def list_models(endpoint: Endpoint) -> list[dict[str, Any]]:
return models
# Where the runners that bother to advertise a context length put it. There is
# no standard field, so this is a list of what the common ones actually emit.
_CONTEXT_KEYS = ("context_length", "max_model_len", "context_window", "max_context_length")
# Below the first, the number is not a context length; above the second it is a
# typo or a different unit. Either way, better to record nothing than a wrong
# figure a percentage would then be computed from.
MIN_CONTEXT = 256
MAX_CONTEXT = 10_000_000
def context_from(entry: dict[str, Any]) -> int:
"""A model's context length as advertised by /v1/models, or 0 if it is not.
Strings are accepted because some servers quote the number, but only when
they are digits alone -- "8192 tokens" is a label, not a measurement.
"""
candidates = [entry.get(key) for key in _CONTEXT_KEYS]
meta = entry.get("meta")
if isinstance(meta, dict):
candidates += [meta.get("n_ctx"), *(meta.get(key) for key in _CONTEXT_KEYS)]
for value in candidates:
if isinstance(value, bool):
continue
if isinstance(value, str):
value = value.strip()
if not value.isdigit():
continue
value = int(value)
if isinstance(value, int) and MIN_CONTEXT <= value <= MAX_CONTEXT:
return value
return 0
async def stream_chat(
endpoint: Endpoint,
payload: dict[str, Any],