Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 19 Wiki Activity

A crowd in one chat

Browse Source 
The chat's own model answers, then each other member in order, then the order runs
backwards asking each whether it disagrees, ending at the main model, which either
closes or sends them round again. Design and reasoning: LLeMbas.wiki/Crowd-chats.

THE SPEAKER SEAM, WHICH IS ALSO A BUG FIX

`chat_service.speaker_for` makes the *message* name the answering model and the
chat only the default. That closes a live half-wired feature -- `wake_chat` takes a
model override and `schedule/runner` passes one, and it reached the row and never
the request, so a schedule naming another model got the chat's model wearing the
other one's name.

The seam is wider than `build_request`: `{{model_name}}`, the authored prompt's
model layer, `vision` (where a wrong answer makes the endpoint reject the whole
request), the effort vocabulary (which raises inside the model's own chat template,
and whose refusal narrows every Model row sharing the id), `resolve_tools`,
`context_length` -> `_too_big`, and `ToolContext.model_id`. `resolve_endpoint` may
now only write back `chat.connection_id` when the speaker *is* the chat's model.

WHY N CHAINED REPLIES

`Generation` is one reply's state and `_follow` streams per message, so one
generation cannot stream into nine bubbles and `ensure` would not know which of the
nine it was after a restart. A subagent per speaker cannot work either: its answer
comes back as a tool result and tool results are never replayed, so speaker 3 could
not see speaker 2 -- which is the whole point. Chained, exactly one incomplete row
exists at a time, and `tests/test_crowd_chain.py` asserts that at every
observation.

The round lives on `Message.crowd_json`, not on the chat: the row is the authority,
and chat-level state would describe turns a rewind or a restart had removed.
`crowd.next_turn` is pure, so all eight refusals are tested with no endpoint.

THREE RULES, EACH A BUG WRITTEN THE OTHER WAY ROUND

- `if not _advance_crowd(g): _drain(g)` -- advancing must *suppress* draining, or a
  queued human turn puts a second incomplete row beside the next speaker's.
- `_advance_crowd` refuses unless the finishing row is the newest, or regenerating
  member 2 creates a second member 3 and two chains race down one turn.
- an error skips one speaker and two in a row end the round: the usual failure is a
  small member's window overflowing, and `_drain`'s stop-on-error would kill every
  crowd at whichever member is smallest.

Each other speaker's turn is relabelled as attributed user content, which is both
how a model can disagree with words it did not write and how the history keeps
alternating. The per-speaker instruction is payload-only -- as a row it could be
dropped from the request by a `created_at` tie, and every later speaker would answer
it. Compaction, titling and the notification are gated to once per turn; `_inject`
is off during a round; the way back gets no tools and a member is treated as
unattended.

Membership stores the model as text with no foreign key: "Test & refresh" deletes
and recreates Model rows, and a cascade would empty the crowd out of every chat.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Homer
2026-09-26 13:38:52 +00:00
co-authored by Claude Opus 5
parent ac51dd46cc
commit da0797ccad
27 changed files with 3018 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/chat.py
+287 -31
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import logging
import re
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from typing import Any
@@ -46,19 +47,71 @@ TITLE_MAX_TOKENS = 512
TEMPORARY_LIFETIME = timedelta(hours=24)
def resolve_endpoint(db: DBSession, chat: Chat) -> tuple[Endpoint, str]:
"""Find the connection and model a chat should use.
@dataclass(frozen=True)
class Speaker:
"""Which model is answering one reply, and through which connection.
The pair and not the id, because `Model` is unique on
`(connection_id, model_id)`: the same name can live behind two endpoints and
an id alone does not say which. `images/tool.py:_reviewer` already resolves a
model this way.
Frozen, and passed rather than re-derived, for the reason `Endpoint` is a
snapshot: a generation outlives the request that started it, and "who is
answering" must not be able to change underneath a reply that is already
streaming.
"""
model_id: str
connection_id: str | None = None
def speaker_for(db: DBSession, chat: Chat, message: Message | None = None) -> Speaker:
"""Who is answering: the message being written into, or else the chat.
**The row names the model and the chat is only the default.** Until 1.6.0 the
answering model was `chat.model_id` and nothing else, while `Message.model_id`
was written on every placeholder and read only for display -- so the bubble's
avatar and the request could disagree, and did: `wake_chat` accepts a
`model_id` override and `schedule/runner` passes `schedule.model_id or
chat.model_id`, which reached the row and never reached the request. A
schedule naming another model got the chat's model wearing the other one's
name.
Reading it off the row is also what makes a reply survive a restart, because
`_follow` calls `ensure`, which starts a *new* generation against the same
row -- so anything the request depends on has to be durable, and the registry
is not. This is the rule the reload story was won with: the row is the
authority.
"""
if message is not None and (message.model_id or "").strip():
return Speaker(message.model_id, getattr(message, "connection_id", None) or None)
return Speaker(chat.model_id, chat.connection_id)
def resolve_endpoint(
db: DBSession, chat: Chat, speaker: Speaker | None = None
) -> tuple[Endpoint, str]:
"""Find the connection and model a reply should use.
Chats store the model id as text rather than a foreign key so history
survives an admin deleting a connection, which means the mapping back to a
live connection has to be resolved at send time and can legitimately fail.
`speaker` defaults to the chat's own model, so every existing caller behaves
exactly as it did.
"""
if not chat.model_id:
speaker = speaker or speaker_for(db, chat)
if not speaker.model_id:
raise LLMError("This chat has no model selected.")
# Whether resolving a fallback may be *written back* to the chat. It may only
# when the speaker is the chat's own model: a crowd member or a schedule's
# model finding its way to another connection must not repoint the chat.
speaks_for_chat = speaker.model_id == chat.model_id
connection: Connection | None = None
if chat.connection_id:
connection = db.get(Connection, chat.connection_id)
if speaker.connection_id:
connection = db.get(Connection, speaker.connection_id)
if connection is None or not connection.enabled:
# The original connection is gone or disabled. Any enabled connection
@@ -67,7 +120,7 @@ def resolve_endpoint(db: DBSession, chat: Chat) -> tuple[Endpoint, str]:
select(Model)
.join(Connection)
.where(
Model.model_id == chat.model_id,
Model.model_id == speaker.model_id,
Model.enabled.is_(True),
Connection.enabled.is_(True),
)
@@ -76,13 +129,14 @@ def resolve_endpoint(db: DBSession, chat: Chat) -> tuple[Endpoint, str]:
if model is None:
raise LLMError(
f"No enabled connection currently offers the model "
f"'{chat.model_id}'. Pick another model for this chat."
f"'{speaker.model_id}'. Pick another model for this chat."
)
connection = model.connection
chat.connection_id = connection.id
db.commit()
if speaks_for_chat:
chat.connection_id = connection.id
db.commit()
return Endpoint.from_connection(connection), chat.model_id
return Endpoint.from_connection(connection), speaker.model_id
def document_context(message: Message) -> str:
@@ -190,7 +244,9 @@ def folder_system_prompt(db: DBSession, chat: Chat) -> str:
return ""
def effective_system_prompt(db: DBSession, chat: Chat) -> str:
def effective_system_prompt(
db: DBSession, chat: Chat, speaker: Speaker | None = None
) -> str:
"""The system prompt a chat actually runs with.
Four layers, most specific wins outright:
@@ -214,9 +270,9 @@ def effective_system_prompt(db: DBSession, chat: Chat) -> str:
if inherited := folder_system_prompt(db, chat):
return inherited
model = db.scalar(
select(Model).where(Model.model_id == chat.model_id).order_by(Model.position)
)
# The *answering* model's layer, which is not always the chat's: a crowd
# member speaking in somebody else's chat brings its own prompt with it.
model = model_row(db, speaker or Speaker(chat.model_id, chat.connection_id))
if model is not None and (model.system_prompt or "").strip():
return model.system_prompt.strip()
@@ -230,6 +286,7 @@ def build_messages(
upto: Message | None = None,
vision: bool = False,
system_prompt: str | None = None,
speaker: Speaker | None = None,
) -> list[dict]:
"""Assemble the message list to send upstream.
@@ -302,24 +359,196 @@ def build_messages(
continue
payload.append(message_payload(message, vision=vision))
if speaker is not None:
payload = _as_one_speaker_sees_it(db, payload, history, speaker, upto=upto)
return payload
def model_for(db: DBSession, chat: Chat) -> Model | None:
"""The Model row a chat is using, or None if it has gone.
def _as_one_speaker_sees_it(
db: DBSession,
payload: list[dict[str, Any]],
history: list[Message],
speaker: Speaker,
*,
upto: Message | None = None,
) -> list[dict[str, Any]]:
"""Rewrite a crowd transcript from one speaker's point of view.
Two problems, one pass.
**Another speaker's reply must not arrive as this one's own prior turn.** Sent
verbatim, every assistant message in the payload reads as something *this*
model said -- so it defends sentences it never wrote, and cannot disagree with
them, which is the whole point of the backward pass. Each other speaker's turn
is therefore relabelled as user content behind a fragment-driven "«Label»
said:".
**Consecutive assistant turns break strict-alternation chat templates**, which
this project already knows: `task.compact_ack` exists so a compacted history
still alternates, and several templates reject one that does not. Relabelling
fixes that by construction, and the adjacent user turns it creates are merged.
⚠ The relabelled entry is built here rather than by calling `message_payload`
with a swapped role. That function attaches image parts when the role is
`user` and the model has vision, so a swapped assistant turn carrying a
generated image would silently become a multimodal list -- and an endpoint
that rejects one rejects every later turn with it.
"""
from lembas.services import prompts as prompts_service
# Nothing to do for the ordinary case: one model, and every assistant turn in
# the payload is its own.
others = {
message.model_id
for message in history
if message.role == ROLE_ASSISTANT
and (message.model_id or "")
and message.model_id != speaker.model_id
}
if not others:
return payload
labels = {
model_id: (row.label if (row := model_row(db, Speaker(model_id))) else model_id)
for model_id in others
}
template = prompts_service.resolve(db, "crowd.said") or "{{crowd_speaker}} answered:"
# The payload and the history line up only over the message rows: the system
# turn and a compaction pair come first and belong to nobody. Walking from the
# end is what pairs them without counting.
rows = [
message
for message in history
if not (upto is not None and message.id == upto.id)
]
rewritten: list[dict[str, Any]] = []
for index, entry in enumerate(payload):
row = None
offset = index - (len(payload) - len(rows))
if 0 <= offset < len(rows):
row = rows[offset]
if (
row is not None
and entry.get("role") == ROLE_ASSISTANT
and (row.model_id or "") in others
):
lead = template.replace("{{crowd_speaker}}", labels[row.model_id])
body = entry.get("content")
rewritten.append(
{"role": ROLE_USER, "content": f"{lead}\n\n{body if isinstance(body, str) else ''}"}
)
continue
rewritten.append(entry)
return _merge_user_turns(rewritten)
def _with_crowd_instruction(
db: DBSession, payload: list[dict[str, Any]], turn, *, again: bool
) -> list[dict[str, Any]]:
"""Append what this speaker has been asked to do, as the closing user turn.
🚨 **Payload only. No row is written for it.** Writing the instruction into the
transcript the way `wake_chat` writes a background job's turn was the first
design and is wrong three times over. `build_messages` orders history by
`created_at` alone and `break`s at the placeholder, so on a shared microsecond
the placeholder sorts first and the instruction is dropped from the request
entirely -- the hazard `thread_tail` already carries an explicit tiebreak for.
It would double the rows in a turn, all of them bubbles somebody has to scroll
past. And every later speaker would read the previous speaker's instruction as
an ordinary user turn and answer that too.
The compaction summary is inserted the same way and for the same reason: a
turn in the payload with nothing behind it (`build_messages`).
"""
from lembas.services import crowd as crowd_service
from lembas.services import prompts as prompts_service
if turn.phase == crowd_service.PHASE_OUT:
key = "crowd.turn"
elif turn.phase == crowd_service.PHASE_BACK:
key = "crowd.disagree"
else:
# Two fragments, not one with a clause in it: inviting a choice the model
# cannot express is worse than not offering it, and a model without the
# tools capability has no `crowd_again` to call.
key = "crowd.close" if again else "crowd.close_final"
text = (prompts_service.resolve(db, key) or "").strip()
if not text:
# Cleared on purpose is the administrator switching this wording off, and
# an empty user turn is not a thing to send.
return payload
return _merge_user_turns([*payload, {"role": ROLE_USER, "content": text}])
def _merge_user_turns(payload: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Fold adjacent user turns into one, so the history still alternates.
Only where both are plain strings: a turn carrying content parts is a
multimodal message and joining one to a string would destroy it.
"""
merged: list[dict[str, Any]] = []
for entry in payload:
last = merged[-1] if merged else None
if (
last is not None
and last.get("role") == ROLE_USER
and entry.get("role") == ROLE_USER
and isinstance(last.get("content"), str)
and isinstance(entry.get("content"), str)
):
merged[-1] = {
**last,
"content": f"{last['content']}\n\n{entry['content']}",
}
continue
merged.append(entry)
return merged
def model_row(db: DBSession, speaker: Speaker) -> Model | None:
"""The Model row a speaker names, or None if it has gone.
Looked up by id rather than held as a foreign key, for the same reason
resolve_endpoint does: chats store the model as text so history survives an
administrator deleting a connection.
administrator deleting a connection. The connection narrows it when one is
named, because two connections may offer the same id and their capabilities,
context length and effort lists are separate rows.
"""
if not speaker.model_id:
return None
if speaker.connection_id:
exact = db.scalar(
select(Model).where(
Model.model_id == speaker.model_id,
Model.connection_id == speaker.connection_id,
)
)
if exact is not None:
return exact
return db.scalar(
select(Model).where(Model.model_id == chat.model_id).order_by(Model.position)
select(Model).where(Model.model_id == speaker.model_id).order_by(Model.position)
)
def model_supports(db: DBSession, chat: Chat, capability: str) -> bool:
"""Whether the chat's current model is marked as having a capability."""
model = model_for(db, chat)
def model_for(db: DBSession, chat: Chat) -> Model | None:
"""The Model row a chat is using. The display answer; see `model_row`."""
return model_row(db, Speaker(chat.model_id, chat.connection_id))
def model_supports(
db: DBSession, chat: Chat, capability: str, speaker: Speaker | None = None
) -> bool:
"""Whether the answering model is marked as having a capability.
⚠ Worth getting right per speaker rather than per chat: `vision` decides
whether image parts go into the body, and an endpoint sent an image by a
model that cannot take one rejects **the whole request**, not the image.
"""
model = model_row(db, speaker) if speaker is not None else model_for(db, chat)
return bool(model and (model.capabilities_json or {}).get(capability))
@@ -331,12 +560,21 @@ def build_request(
tools: list[dict[str, Any]] | None = None,
user=None,
force_tool: str = "",
speaker: Speaker | None = None,
crowd_turn=None,
crowd_again: bool = False,
) -> dict[str, Any]:
"""The whole request body, tools and harness included.
Composed here rather than in the generation loop so that "what gets sent"
has one answer, and so the harness cannot be forgotten by a future caller
that offers tools.
`speaker` is who is answering; it defaults to the chat's own model, so a
caller that does not care behaves exactly as it did. Everything that differs
per model is resolved from it and not from the chat: the model name sent, the
vision decision, the authored prompt's model layer, `{{model_name}}`, the
personality, and the reasoning-effort vocabulary.
"""
from lembas.services import harness as harness_service
from lembas.services import prompts as prompts_service
@@ -346,10 +584,15 @@ def build_request(
for key, value in (chat.params_json or {}).items()
if key in FORWARDED_PARAMS and value not in (None, "")
}
speaker = speaker or speaker_for(db, chat, upto)
if crowd_turn is None and upto is not None:
from lembas.services import crowd as crowd_service
crowd_turn = crowd_service.state_of(upto)
# Images are only sent to a model an administrator has marked as having
# vision. Sending them to one that has not is not a graceful degradation:
# most endpoints reject the whole request.
vision = model_supports(db, chat, "vision")
vision = model_supports(db, chat, "vision", speaker=speaker)
if user is None:
from lembas.db.models import User
@@ -360,18 +603,23 @@ def build_request(
# behaviour. See services/harness.py for why these are joined rather than
# being two competing layers.
system = harness_service.join(
harness_service.compose(db, user, tools, chat),
effective_system_prompt(db, chat),
harness_service.compose(db, user, tools, chat, speaker=speaker),
effective_system_prompt(db, chat, speaker),
lead=prompts_service.render(db, "seam.authored_lead", {}),
)
body: dict[str, Any] = {
"model": chat.model_id,
"model": speaker.model_id,
"messages": build_messages(
db, chat, upto=upto, vision=vision, system_prompt=system
db, chat, upto=upto, vision=vision, system_prompt=system, speaker=speaker
),
**params,
}
if crowd_turn is not None:
body["messages"] = _with_crowd_instruction(
db, body["messages"], crowd_turn, again=crowd_again
)
if tools:
body["tools"] = tools
# Making the model call one particular tool, for `/image` -- the whole
@@ -388,14 +636,22 @@ def build_request(
):
body["tool_choice"] = {"type": "function", "function": {"name": force_tool}}
# The model's own vocabulary, looked up here rather than passed in: every
# caller of `build_request` would otherwise have to remember, which is the
# trap `audio_service.template_flags` fell into.
chat_model = model_for(db, chat)
# The *answering* model's own vocabulary, looked up here rather than passed
# in: every caller of `build_request` would otherwise have to remember, which
# is the trap `audio_service.template_flags` fell into.
#
# ⚠ Per speaker and not per chat, and this one is not cosmetic: the
# vocabularies genuinely differ -- gpt-oss takes low/medium/high, a Bonsai
# takes low/medium/xhigh and *raises inside its chat template* on high -- so
# a chat's effort handed to another model fails the whole reply rather than
# being ignored. `_learn_refused_effort` then narrows every Model row sharing
# that id, so getting this wrong would also corrupt other models' lists as a
# side effect.
speaking_model = model_row(db, speaker)
apply_effort(
body,
(chat.params_json or {}).get("reasoning_effort"),
efforts_for(chat_model) if chat_model is not None else None,
efforts_for(speaking_model) if speaking_model is not None else None,
)
return body