Models know how much context they hold
A column rather than a key in capabilities_json, which is rebuilt wholesale from the submitted checkboxes on every save and would destroy a number living in it. 0 means unknown, and unknown has to stay tellable from small: the context percentage and automatic compaction both refuse to act on a figure nobody supplied. Filled in from /v1/models where the runner advertises it -- OpenRouter, vLLM and llama.cpp each spell it differently, so context_from() reads the four spellings actually in use, accepts a quoted number but not "8192 tokens", and rejects anything outside 256..10,000,000. Applied on discovery only when nothing is set: a refresh must never undo a correction, since an administrator sets this precisely because the endpoint was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -155,6 +155,41 @@ async def list_models(endpoint: Endpoint) -> list[dict[str, Any]]:
|
||||
return models
|
||||
|
||||
|
||||
# Where the runners that bother to advertise a context length put it. There is
|
||||
# no standard field, so this is a list of what the common ones actually emit.
|
||||
_CONTEXT_KEYS = ("context_length", "max_model_len", "context_window", "max_context_length")
|
||||
|
||||
# Below the first, the number is not a context length; above the second it is a
|
||||
# typo or a different unit. Either way, better to record nothing than a wrong
|
||||
# figure a percentage would then be computed from.
|
||||
MIN_CONTEXT = 256
|
||||
MAX_CONTEXT = 10_000_000
|
||||
|
||||
|
||||
def context_from(entry: dict[str, Any]) -> int:
|
||||
"""A model's context length as advertised by /v1/models, or 0 if it is not.
|
||||
|
||||
Strings are accepted because some servers quote the number, but only when
|
||||
they are digits alone -- "8192 tokens" is a label, not a measurement.
|
||||
"""
|
||||
candidates = [entry.get(key) for key in _CONTEXT_KEYS]
|
||||
meta = entry.get("meta")
|
||||
if isinstance(meta, dict):
|
||||
candidates += [meta.get("n_ctx"), *(meta.get(key) for key in _CONTEXT_KEYS)]
|
||||
|
||||
for value in candidates:
|
||||
if isinstance(value, bool):
|
||||
continue
|
||||
if isinstance(value, str):
|
||||
value = value.strip()
|
||||
if not value.isdigit():
|
||||
continue
|
||||
value = int(value)
|
||||
if isinstance(value, int) and MIN_CONTEXT <= value <= MAX_CONTEXT:
|
||||
return value
|
||||
return 0
|
||||
|
||||
|
||||
async def stream_chat(
|
||||
endpoint: Endpoint,
|
||||
payload: dict[str, Any],
|
||||
|
||||
Reference in New Issue
Block a user