The chat's own model answers, then each other member in order, then the order runs
backwards asking each whether it disagrees, ending at the main model, which either
closes or sends them round again. Design and reasoning: LLeMbas.wiki/Crowd-chats.
THE SPEAKER SEAM, WHICH IS ALSO A BUG FIX
`chat_service.speaker_for` makes the *message* name the answering model and the
chat only the default. That closes a live half-wired feature -- `wake_chat` takes a
model override and `schedule/runner` passes one, and it reached the row and never
the request, so a schedule naming another model got the chat's model wearing the
other one's name.
The seam is wider than `build_request`: `{{model_name}}`, the authored prompt's
model layer, `vision` (where a wrong answer makes the endpoint reject the whole
request), the effort vocabulary (which raises inside the model's own chat template,
and whose refusal narrows every Model row sharing the id), `resolve_tools`,
`context_length` -> `_too_big`, and `ToolContext.model_id`. `resolve_endpoint` may
now only write back `chat.connection_id` when the speaker *is* the chat's model.
WHY N CHAINED REPLIES
`Generation` is one reply's state and `_follow` streams per message, so one
generation cannot stream into nine bubbles and `ensure` would not know which of the
nine it was after a restart. A subagent per speaker cannot work either: its answer
comes back as a tool result and tool results are never replayed, so speaker 3 could
not see speaker 2 -- which is the whole point. Chained, exactly one incomplete row
exists at a time, and `tests/test_crowd_chain.py` asserts that at every
observation.
The round lives on `Message.crowd_json`, not on the chat: the row is the authority,
and chat-level state would describe turns a rewind or a restart had removed.
`crowd.next_turn` is pure, so all eight refusals are tested with no endpoint.
THREE RULES, EACH A BUG WRITTEN THE OTHER WAY ROUND
- `if not _advance_crowd(g): _drain(g)` -- advancing must *suppress* draining, or a
queued human turn puts a second incomplete row beside the next speaker's.
- `_advance_crowd` refuses unless the finishing row is the newest, or regenerating
member 2 creates a second member 3 and two chains race down one turn.
- an error skips one speaker and two in a row end the round: the usual failure is a
small member's window overflowing, and `_drain`'s stop-on-error would kill every
crowd at whichever member is smallest.
Each other speaker's turn is relabelled as attributed user content, which is both
how a model can disagree with words it did not write and how the history keeps
alternating. The per-speaker instruction is payload-only -- as a row it could be
dropped from the request by a `created_at` tie, and every later speaker would answer
it. Compaction, titling and the notification are gated to once per turn; `_inject`
is off during a round; the way back gets no tools and a member is treated as
unattended.
Membership stores the model as text with no foreign key: "Test & refresh" deletes
and recreates Model rows, and a cascade would empty the crowd out of every chat.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
243 lines
8.0 KiB
Python
243 lines
8.0 KiB
Python
"""Choosing a crowd, and reading one.
|
|
|
|
Two screens and one rule each. The picker may only ever offer and accept models
|
|
*this person* can reach — a control checked in the template and not in the route is
|
|
advisory, and a crafted request walks past it. The transcript has to say which
|
|
speaker a bubble is and which pass it belongs to, because nine bubbles for one
|
|
question are otherwise indistinguishable from nine people talking at once.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
from sqlalchemy import select
|
|
|
|
from lembas.db.models import (
|
|
ROLE_ASSISTANT,
|
|
ROLE_USER,
|
|
Chat,
|
|
Connection,
|
|
Group,
|
|
Model,
|
|
User,
|
|
)
|
|
from lembas.services import chat as chat_service
|
|
from lembas.services import crowd as crowd_service
|
|
from lembas.services import settings_store
|
|
from lembas.services.crypto import encrypt
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def crowd_on(db, registered):
|
|
settings_store.update(db, {"enabled": True}, key=settings_store.CROWD)
|
|
connection = Connection(
|
|
name="Test", base_url="http://127.0.0.1:1", api_key_encrypted=encrypt("")
|
|
)
|
|
db.add(connection)
|
|
db.commit()
|
|
for index, name in enumerate(("main-model", "second-model", "third-model")):
|
|
db.add(
|
|
Model(
|
|
connection_id=connection.id,
|
|
model_id=name,
|
|
display_name=name,
|
|
position=index,
|
|
capabilities_json={"tools": True},
|
|
)
|
|
)
|
|
db.commit()
|
|
|
|
|
|
def _user(db) -> User:
|
|
return db.scalars(select(User).order_by(User.created_at)).first()
|
|
|
|
|
|
def _chat(db) -> Chat:
|
|
connection = db.scalars(select(Connection)).first()
|
|
chat = Chat(
|
|
user_id=_user(db).id, title="t", model_id="main-model", connection_id=connection.id
|
|
)
|
|
db.add(chat)
|
|
db.commit()
|
|
return chat
|
|
|
|
|
|
def _members(db, chat) -> list[str]:
|
|
db.expire_all()
|
|
return [
|
|
row.model_id
|
|
for row in sorted(db.get(Chat, chat.id).crowd, key=lambda r: r.position)
|
|
]
|
|
|
|
|
|
# --- Choosing -----------------------------------------------------------------
|
|
def test_the_panel_offers_the_other_models(client, db):
|
|
chat = _chat(db)
|
|
page = client.get(f"/chat/{chat.id}").text
|
|
assert 'name="crowd_model_ids"' in page
|
|
assert 'name="crowd_model_ids" value="second-model"' in page
|
|
# Never the chat's own model: it would answer twice in a row. Asserted with
|
|
# the field name attached, because the model *picker* on the same page quite
|
|
# correctly offers it.
|
|
assert 'name="crowd_model_ids" value="main-model"' not in page
|
|
|
|
|
|
def test_the_panel_is_absent_while_the_feature_is_off(client, db):
|
|
settings_store.update(db, {"enabled": False}, key=settings_store.CROWD)
|
|
chat = _chat(db)
|
|
assert 'name="crowd_model_ids"' not in client.get(f"/chat/{chat.id}").text
|
|
|
|
|
|
def test_ticking_a_model_adds_it_in_order(client, db):
|
|
chat = _chat(db)
|
|
client.patch(
|
|
f"/api/chats/{chat.id}",
|
|
data={"crowd_model_ids": ["second-model", "third-model"]},
|
|
)
|
|
assert _members(db, chat) == ["second-model", "third-model"]
|
|
|
|
|
|
def test_clearing_every_box_clears_the_crowd(client, db):
|
|
"""The single field always sent is what makes this possible: an absent
|
|
checkbox carries no signal of its own."""
|
|
chat = _chat(db)
|
|
client.patch(f"/api/chats/{chat.id}", data={"crowd_model_ids": ["second-model"]})
|
|
assert _members(db, chat) == ["second-model"]
|
|
|
|
client.patch(f"/api/chats/{chat.id}", data={"crowd_model_ids": [""]})
|
|
assert _members(db, chat) == []
|
|
|
|
|
|
def test_a_model_this_person_cannot_reach_is_refused(client, db):
|
|
"""Checked in the route, not only in the template. Otherwise the picker is
|
|
advisory."""
|
|
group = Group(name="Wheel")
|
|
db.add(group)
|
|
restricted = db.scalar(select(Model).where(Model.model_id == "third-model"))
|
|
restricted.public = False
|
|
restricted.groups = [group]
|
|
user = _user(db)
|
|
user.role = "user"
|
|
db.commit()
|
|
chat = _chat(db)
|
|
|
|
client.patch(
|
|
f"/api/chats/{chat.id}",
|
|
data={"crowd_model_ids": ["second-model", "third-model"]},
|
|
)
|
|
|
|
assert _members(db, chat) == ["second-model"]
|
|
|
|
|
|
def test_the_chats_own_model_cannot_be_added(client, db):
|
|
chat = _chat(db)
|
|
client.patch(f"/api/chats/{chat.id}", data={"crowd_model_ids": ["main-model"]})
|
|
assert _members(db, chat) == []
|
|
|
|
|
|
def test_the_same_model_twice_is_one_member(client, db):
|
|
chat = _chat(db)
|
|
client.patch(
|
|
f"/api/chats/{chat.id}", data={"crowd_model_ids": ["second-model", "second-model"]}
|
|
)
|
|
assert _members(db, chat) == ["second-model"]
|
|
|
|
|
|
def test_the_cap_trims_what_is_accepted(client, db):
|
|
settings_store.update(db, {"max_models": 1}, key=settings_store.CROWD)
|
|
chat = _chat(db)
|
|
client.patch(
|
|
f"/api/chats/{chat.id}", data={"crowd_model_ids": ["second-model", "third-model"]}
|
|
)
|
|
assert _members(db, chat) == ["second-model"]
|
|
|
|
|
|
def test_the_panel_says_what_a_turn_will_cost(client, db):
|
|
"""The thing somebody will not have thought about: a turn is
|
|
speakers x rounds x 2 - 1 replies, and each is a whole reply."""
|
|
chat = _chat(db)
|
|
client.patch(
|
|
f"/api/chats/{chat.id}", data={"crowd_model_ids": ["second-model", "third-model"]}
|
|
)
|
|
page = client.get(f"/chat/{chat.id}").text
|
|
assert "5 replies a turn" in page
|
|
|
|
|
|
def test_a_member_that_can_no_longer_be_reached_is_shown_struck_through(client, db):
|
|
"""Membership is text with no foreign key, so the row outlives the model. Saying
|
|
so beats both deleting it and pretending it still answers."""
|
|
chat = _chat(db)
|
|
client.patch(f"/api/chats/{chat.id}", data={"crowd_model_ids": ["third-model"]})
|
|
gone = db.scalar(select(Model).where(Model.model_id == "third-model"))
|
|
db.delete(gone)
|
|
db.commit()
|
|
|
|
page = client.get(f"/chat/{chat.id}").text
|
|
|
|
assert "Skipped" in page
|
|
assert "<s>third-model</s>" in page
|
|
|
|
|
|
# --- Reading ------------------------------------------------------------------
|
|
def _bubble(db, chat, *, phase, index=1, of=3, stopped="", round_=1) -> str:
|
|
from lembas.api import chats as chats_api
|
|
|
|
chat_service.create_message(db, chat, ROLE_USER, "What should we do?")
|
|
message = chat_service.create_message(
|
|
db, chat, ROLE_ASSISTANT, "Something.", model_id="second-model"
|
|
)
|
|
message.crowd_json = crowd_service.Turn(
|
|
turn="u1", round=round_, phase=phase, index=index, of=of,
|
|
started_at=crowd_service.now_stamp(), stopped=stopped,
|
|
).as_json()
|
|
db.commit()
|
|
return chats_api._render_bubble(db, chat, _user(db), message)
|
|
|
|
|
|
def test_a_bubble_on_the_way_out_says_which_speaker_it_is(db):
|
|
chat = _chat(db)
|
|
html = _bubble(db, chat, phase=crowd_service.PHASE_OUT, index=1, of=3)
|
|
assert "2 of 3" in html
|
|
|
|
|
|
def test_a_bubble_on_the_way_back_says_so_and_is_quieter(db):
|
|
chat = _chat(db)
|
|
html = _bubble(db, chat, phase=crowd_service.PHASE_BACK)
|
|
assert "on the way back" in html
|
|
assert "msg--crowd-back" in html
|
|
|
|
|
|
def test_the_closing_bubble_says_it_is_closing(db):
|
|
chat = _chat(db)
|
|
html = _bubble(db, chat, phase=crowd_service.PHASE_CLOSE, index=0)
|
|
assert "closing" in html
|
|
|
|
|
|
def test_a_later_round_is_numbered(db):
|
|
chat = _chat(db)
|
|
html = _bubble(db, chat, phase=crowd_service.PHASE_OUT, round_=2)
|
|
assert "round 2" in html
|
|
|
|
|
|
def test_why_a_round_ended_is_shown_where_it_ended(db):
|
|
"""Otherwise a crowd that ran out of rounds or time simply stops, which reads
|
|
as the feature failing rather than as a limit doing its job."""
|
|
chat = _chat(db)
|
|
assert "no rounds left" in _bubble(
|
|
db, chat, phase=crowd_service.PHASE_CLOSE, stopped=crowd_service.STOPPED_ROUNDS
|
|
)
|
|
|
|
|
|
def test_an_ordinary_bubble_carries_no_crowd_chip(db):
|
|
from lembas.api import chats as chats_api
|
|
|
|
chat = _chat(db)
|
|
chat_service.create_message(db, chat, ROLE_USER, "Hello")
|
|
message = chat_service.create_message(
|
|
db, chat, ROLE_ASSISTANT, "Hello back.", model_id="main-model"
|
|
)
|
|
html = chats_api._render_bubble(db, chat, _user(db), message)
|
|
assert "of 3" not in html
|
|
assert "msg--crowd-back" not in html
|
|
assert "closing" not in html
|