Models know how much context they hold
A column rather than a key in capabilities_json, which is rebuilt wholesale from the submitted checkboxes on every save and would destroy a number living in it. 0 means unknown, and unknown has to stay tellable from small: the context percentage and automatic compaction both refuse to act on a figure nobody supplied. Filled in from /v1/models where the runner advertises it -- OpenRouter, vLLM and llama.cpp each spell it differently, so context_from() reads the four spellings actually in use, accepts a quoted number but not "8192 tokens", and rejects anything outside 256..10,000,000. Applied on discovery only when nothing is set: a refresh must never undo a correction, since an administrator sets this precisely because the endpoint was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -114,6 +114,17 @@ class Model(UUIDPrimaryKey, Timestamps, Base):
|
||||
# Default sampling params applied to new chats using this model.
|
||||
params_json: Mapped[dict[str, Any]] = mapped_column(JSONDict, default=dict)
|
||||
|
||||
# How many tokens this model can hold. 0 means unknown, which is what an
|
||||
# endpoint that does not advertise it leaves behind -- and unknown has to
|
||||
# stay tellable from "small", because the context percentage and automatic
|
||||
# compaction both refuse to act on a number nobody supplied.
|
||||
#
|
||||
# A column rather than a key in capabilities_json: that dict is rebuilt
|
||||
# wholesale from the submitted checkboxes on every save (api/admin_models.py),
|
||||
# so a number living in it would be destroyed the next time an administrator
|
||||
# ticked anything.
|
||||
context_length: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
|
||||
connection: Mapped[Connection] = relationship(back_populates="models")
|
||||
groups: Mapped[list[Group]] = relationship(
|
||||
"Group", secondary=model_groups, back_populates="models"
|
||||
|
||||
Reference in New Issue
Block a user