Models know how much context they hold
A column rather than a key in capabilities_json, which is rebuilt wholesale from the submitted checkboxes on every save and would destroy a number living in it. 0 means unknown, and unknown has to stay tellable from small: the context percentage and automatic compaction both refuse to act on a figure nobody supplied. Filled in from /v1/models where the runner advertises it -- OpenRouter, vLLM and llama.cpp each spell it differently, so context_from() reads the four spellings actually in use, accepts a quoted number but not "8192 tokens", and rejects anything outside 256..10,000,000. Applied on discovery only when nothing is set: a refresh must never undo a correction, since an administrator sets this precisely because the endpoint was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+15
-2
@@ -14,7 +14,7 @@ from lembas.api.deps import AdminUser, Db
|
||||
from lembas.db.models import Connection, Model, User
|
||||
from lembas.services import settings_store
|
||||
from lembas.services.crypto import UNCHANGED_SENTINEL, decrypt, encrypt, mask
|
||||
from lembas.services.llm.openai_client import Endpoint, LLMError, list_models
|
||||
from lembas.services.llm.openai_client import Endpoint, LLMError, context_from, list_models
|
||||
from lembas.web.templating import render
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -198,8 +198,21 @@ async def _refresh_models(db: DBSession, connection: Connection) -> tuple[int, s
|
||||
model_id = str(entry["id"])[:300]
|
||||
seen.add(model_id)
|
||||
if model_id in existing:
|
||||
# A context length is filled in only when nobody has one yet. A
|
||||
# refresh must never overwrite a number an administrator typed --
|
||||
# they are usually correcting the endpoint.
|
||||
model = existing[model_id]
|
||||
if not model.context_length:
|
||||
model.context_length = context_from(entry)
|
||||
continue
|
||||
db.add(Model(connection_id=connection.id, model_id=model_id, position=next_position))
|
||||
db.add(
|
||||
Model(
|
||||
connection_id=connection.id,
|
||||
model_id=model_id,
|
||||
position=next_position,
|
||||
context_length=context_from(entry),
|
||||
)
|
||||
)
|
||||
next_position += 1
|
||||
|
||||
# Models that vanished upstream are dropped, so the picker never offers
|
||||
|
||||
Reference in New Issue
Block a user