Models know how much context they hold

A column rather than a key in capabilities_json, which is rebuilt wholesale
from the submitted checkboxes on every save and would destroy a number
living in it.

0 means unknown, and unknown has to stay tellable from small: the context
percentage and automatic compaction both refuse to act on a figure nobody
supplied. Filled in from /v1/models where the runner advertises it --
OpenRouter, vLLM and llama.cpp each spell it differently, so context_from()
reads the four spellings actually in use, accepts a quoted number but not
"8192 tokens", and rejects anything outside 256..10,000,000. Applied on
discovery only when nothing is set: a refresh must never undo a correction,
since an administrator sets this precisely because the endpoint was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-01 00:33:50 +02:00
parent 50484b2483
commit 90623b461d
6 changed files with 205 additions and 2 deletions
+11
View File
@@ -114,6 +114,17 @@ class Model(UUIDPrimaryKey, Timestamps, Base):
# Default sampling params applied to new chats using this model.
params_json: Mapped[dict[str, Any]] = mapped_column(JSONDict, default=dict)
# How many tokens this model can hold. 0 means unknown, which is what an
# endpoint that does not advertise it leaves behind -- and unknown has to
# stay tellable from "small", because the context percentage and automatic
# compaction both refuse to act on a number nobody supplied.
#
# A column rather than a key in capabilities_json: that dict is rebuilt
# wholesale from the submitted checkboxes on every save (api/admin_models.py),
# so a number living in it would be destroyed the next time an administrator
# ticked anything.
context_length: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
connection: Mapped[Connection] = relationship(back_populates="models")
groups: Mapped[list[Group]] = relationship(
"Group", secondary=model_groups, back_populates="models"