Models know how much context they hold

A column rather than a key in capabilities_json, which is rebuilt wholesale
from the submitted checkboxes on every save and would destroy a number
living in it.

0 means unknown, and unknown has to stay tellable from small: the context
percentage and automatic compaction both refuse to act on a figure nobody
supplied. Filled in from /v1/models where the runner advertises it --
OpenRouter, vLLM and llama.cpp each spell it differently, so context_from()
reads the four spellings actually in use, accepts a quoted number but not
"8192 tokens", and rejects anything outside 256..10,000,000. Applied on
discovery only when nothing is set: a refresh must never undo a correction,
since an administrator sets this precisely because the endpoint was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 00:33:50 +02:00
parent 6dd13b2e9d
commit 2fe736aa6a
6 changed files with 205 additions and 2 deletions
+15 -2
View File
@@ -14,7 +14,7 @@ from lembas.api.deps import AdminUser, Db
from lembas.db.models import Connection, Model, User
from lembas.services import settings_store
from lembas.services.crypto import UNCHANGED_SENTINEL, decrypt, encrypt, mask
from lembas.services.llm.openai_client import Endpoint, LLMError, list_models
from lembas.services.llm.openai_client import Endpoint, LLMError, context_from, list_models
from lembas.web.templating import render
log = logging.getLogger(__name__)
@@ -198,8 +198,21 @@ async def _refresh_models(db: DBSession, connection: Connection) -> tuple[int, s
model_id = str(entry["id"])[:300]
seen.add(model_id)
if model_id in existing:
# A context length is filled in only when nobody has one yet. A
# refresh must never overwrite a number an administrator typed --
# they are usually correcting the endpoint.
model = existing[model_id]
if not model.context_length:
model.context_length = context_from(entry)
continue
db.add(Model(connection_id=connection.id, model_id=model_id, position=next_position))
db.add(
Model(
connection_id=connection.id,
model_id=model_id,
position=next_position,
context_length=context_from(entry),
)
)
next_position += 1
# Models that vanished upstream are dropped, so the picker never offers