Models know how much context they hold

A column rather than a key in capabilities_json, which is rebuilt wholesale
from the submitted checkboxes on every save and would destroy a number
living in it.

0 means unknown, and unknown has to stay tellable from small: the context
percentage and automatic compaction both refuse to act on a figure nobody
supplied. Filled in from /v1/models where the runner advertises it --
OpenRouter, vLLM and llama.cpp each spell it differently, so context_from()
reads the four spellings actually in use, accepts a quoted number but not
"8192 tokens", and rejects anything outside 256..10,000,000. Applied on
discovery only when nothing is set: a refresh must never undo a correction,
since an administrator sets this precisely because the endpoint was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 00:33:50 +02:00
parent 6dd13b2e9d
commit 2fe736aa6a
6 changed files with 205 additions and 2 deletions
+10
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import contextlib
import logging
from fastapi import APIRouter, File, Form, HTTPException, Request, Response, UploadFile, status
@@ -12,6 +13,7 @@ from sqlalchemy.orm import Session as DBSession
from lembas.api.deps import AdminUser, Db, RequiredUser
from lembas.db.models import Connection, Group, Model
from lembas.services import settings_store, uploads
from lembas.services.llm.openai_client import MAX_CONTEXT
from lembas.web.templating import render
log = logging.getLogger(__name__)
@@ -213,6 +215,7 @@ async def update_model(
pinned: bool = Form(False),
public: bool = Form(False),
position: str = Form(""),
context_length: str = Form(""),
group_ids: list[str] = Form(default=[]),
capability: list[str] = Form(default=[]),
) -> Response:
@@ -221,6 +224,13 @@ async def update_model(
model.display_name = display_name.strip()[:300]
model.description = description.strip()[:2000]
model.system_prompt = system_prompt.strip()[:8000]
# A string, so an emptied field is distinguishable and junk can be ignored
# rather than becoming a 422 -- the same shape `position` uses below.
if context_length.strip():
with contextlib.suppress(ValueError):
model.context_length = min(max(int(context_length), 0), MAX_CONTEXT)
else:
model.context_length = 0
model.enabled = enabled
model.pinned = pinned
model.public = public