1 Commits
Author SHA1 Message Date
HomerandClaude Opus 5 8503c6c775 A crowd that does not agree with whoever spoke last
Three fixes to how a round behaves, found by reading one real round on the live
instance rather than by testing it.

A member asked "what would you have done differently" answered the person's
original question again instead of critiquing what was already there. Fine on a
question with one answer; on a request to *make* something it is an invitation.
`crowd.turn` now says to respond to what is above and not to re-answer.

The model that opened the round, told to write the final answer and take what
the others got right, abandoned its own good answer and adopted the newcomer's
position with no argument anywhere for why. Both closing fragments now say that
an answer is not the worse one for having been written first, and that agreement
with no argument behind it is not a reason to change.

That second one is not cosmetic: all three answers from the observed round were
compiled. The original and the critic's alternative both build; the merged
answer that was actually delivered does not. A crowd's failure mode is not
looping -- the caps handle that -- it is converging on the last thing said.

Third, the reply that opens a round now carries a chip like every other one. It
is the single contribution the crowd does not start, so there was nothing to
stamp it with until the round began, and a two-model round rendered as an
unmarked reply followed by one saying "2 of 2". The stamp is display state and
never scheduling state: `crowd.scheduling_state` hides it from everything that
decides what happens next, because fed to the scheduler it would inherit the
round's clock -- regenerating the opening an hour later would end the round with
"out of time" before anybody spoke -- and would hand that reply a member's tools
and a member's instruction.

And the chip was never translated. It is now, with the count as placeholders
rather than three t() calls around one sentence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-26 21:10:25 +00:00
10 changed files with 262 additions and 22 deletions
+44
View File
@@ -16,6 +16,50 @@ for 1.0.0 have something to be assembled from.
## Unreleased ## Unreleased
## 1.8.1
Three fixes to how a crowd behaves, found by reading one real round on the live
instance rather than by testing: two models, one round, a question that asked for
something to be *made*.
- **A member no longer answers the question again.** Asked to pick a language and
write an example, the main model wrote Python; the second model gave a genuinely
useful critique of it — and then answered the original question itself, in a
different language. Nothing in its instruction said not to. It now says so:
*respond to what is above you; do not answer the person's original request again
yourself.* A member that produces a rival answer is not a second opinion, it is
a second first opinion, and it is what takes a round off the question.
- **The model that opened the round no longer capitulates.** Told to write the
final answer and take what the others got right, it abandoned its own perfectly
good answer, wrote *"I agree that Rust is the superior choice"* with no argument
anywhere for why, and rewrote everything in the newcomer's language. Both
closing instructions now carry: *your own answer is not automatically the worse
one for having been written first; change your position where somebody gave you
a reason, and say what the reason was.*
This mattered more than it reads. All three answers were compiled: the original
Python was fine, the critic's Rust compiled and ran — and **the merged answer
that was actually delivered did not compile at all**. A crowd that ends by
agreeing with whoever spoke last can be worse than the model that started it.
- **The bubble that opens a round now says `1 of 3` like every other one.** It was
the single contribution with no chip, because the crowd does not start it — the
composer does, and a round only begins when it finishes. So a two-model round
read as an ordinary reply followed by one labelled `2 of 2`, with no 1 anywhere.
It is stamped when the round begins, and that stamp is deliberately invisible to
everything that decides what happens next: fed to the scheduler it would inherit
the round's clock, so regenerating the opening an hour later would end the round
with "out of time" before anybody spoke.
- Fixed: **the crowd chip was never translated.** `1 of 3`, `on the way back`,
`closing`, `no rounds left` and the rest were English on a Slovak instance.
**Worth knowing, and not a bug:** with **two** models there is no backward pass at
all. The way back would contain only the model that opened the round, whose turn
*is* the close — so `crowd.disagree` never fires. You need at least three models
before a single "do you disagree" bubble can exist.
## 1.8.0 ## 1.8.0
- **The crowd is where you would look for it.** In 1.6.0 the only way to add a - **The crowd is where you would look for it.** In 1.6.0 the only way to add a
+1 -1
View File
@@ -1,3 +1,3 @@
"""LLeMbas - a Middle-earth themed web UI for OpenAI-compatible LLM endpoints.""" """LLeMbas - a Middle-earth themed web UI for OpenAI-compatible LLM endpoints."""
__version__ = "1.8.0" __version__ = "1.8.1"
+4 -1
View File
@@ -588,7 +588,10 @@ def build_request(
if crowd_turn is None and upto is not None: if crowd_turn is None and upto is not None:
from lembas.services import crowd as crowd_service from lembas.services import crowd as crowd_service
crowd_turn = crowd_service.state_of(upto) # `scheduling_state`: the opening reply carries a stamp for the chip's
# sake, and regenerating it must still build an ordinary first answer --
# not one told that "the answers above are quoted, yours comes next".
crowd_turn = crowd_service.scheduling_state(upto)
# Images are only sent to a model an administrator has marked as having # Images are only sent to a model an administrator has marked as having
# vision. Sending them to one that has not is not a graceful degradation: # vision. Sending them to one that has not is not a graceful degradation:
# most endpoints reject the whole request. # most endpoints reject the whole request.
+37
View File
@@ -134,6 +134,41 @@ def state_of(message: Message | None) -> Turn | None:
return None return None
def is_opening(state: Turn | None) -> bool:
"""Whether this state is the main model's opening reply.
`phase=out, index=0` is **display state and never scheduling state**. The
opening reply is not started by the crowd -- the composer starts it, exactly
as it starts every other reply, and a round only begins when it *finishes*.
Stamping it afterwards is what lets the transcript say `1 of 3` on the bubble
that opened the round; before that it was the one contribution with no chip,
so a two-model round read as an ordinary reply followed by a crowd.
Everything that asks "is a round already in progress?" has to skip it, or the
stamp changes behaviour it was never meant to touch -- see `scheduling_state`.
"""
return state is not None and state.phase == PHASE_OUT and state.index == 0
def scheduling_state(message: Message | None) -> Turn | None:
"""The round state the scheduler should act on: `state_of`, minus the opening.
Two things would break if the opening stamp were fed to `next_turn` as real
state, and both are silent:
* **`started_at` would be inherited on a regenerate.** Regenerating the
opening reply an hour later would hand `next_turn` an hour-old clock and the
round would stop with "out of time" before anybody spoke.
* **The once-per-turn gates key off "no state at all"** -- compaction, the
title, the unread push. A stamped opening reads as a later speaker, and each
of them would be skipped for the turn that is supposed to have them.
So the stamp is written where the transcript reads it and nowhere else.
"""
state = state_of(message)
return None if is_opening(state) else state
def now_stamp() -> str: def now_stamp() -> str:
return datetime.now(UTC).isoformat() return datetime.now(UTC).isoformat()
@@ -374,9 +409,11 @@ __all__ = [
"Turn", "Turn",
"elapsed", "elapsed",
"is_newest", "is_newest",
"is_opening",
"member_speakers", "member_speakers",
"next_turn", "next_turn",
"now_stamp", "now_stamp",
"scheduling_state",
"state_of", "state_of",
"tool_defs", "tool_defs",
"unreachable_members", "unreachable_members",
+32 -5
View File
@@ -661,7 +661,10 @@ async def _run(generation: Generation) -> None:
# once, here, and used for three decisions: which tools it may have, # once, here, and used for three decisions: which tools it may have,
# which instruction closes its request, and whether it may ask for # which instruction closes its request, and whether it may ask for
# another round. # another round.
crowd_state = crowd_service.state_of(message) # `scheduling_state` for the reason `build_request` gives: the
# opening reply's stamp is for the transcript, and regenerating it
# must not hand it a member's tools or a member's instruction.
crowd_state = crowd_service.scheduling_state(message)
crowd_settings = settings_store.crowd(db) crowd_settings = settings_store.crowd(db)
may_ask_again = bool( may_ask_again = bool(
crowd_state is not None crowd_state is not None
@@ -2232,7 +2235,11 @@ def _advance_crowd(generation: Generation) -> bool:
speakers = crowd_service.member_speakers(db, chat, owner_user) speakers = crowd_service.member_speakers(db, chat, owner_user)
speakers = speakers[: int(settings["max_models"]) + 1] speakers = speakers[: int(settings["max_models"]) + 1]
state = crowd_service.state_of(message) # `scheduling_state` and not `state_of`: the opening reply carries a
# stamp for the transcript's sake (so it can say `1 of 3`), and that
# stamp must not read as "a round is already running" -- it would
# inherit the old clock on a regenerate. See `crowd.is_opening`.
state = crowd_service.scheduling_state(message)
# The turn a round belongs to: the user message this all answers. # The turn a round belongs to: the user message this all answers.
turn_id = state.turn if state is not None else _turn_anchor(db, message) turn_id = state.turn if state is not None else _turn_anchor(db, message)
following = crowd_service.next_turn( following = crowd_service.next_turn(
@@ -2254,6 +2261,22 @@ def _advance_crowd(generation: Generation) -> bool:
db.commit() db.commit()
return False return False
if state is None:
# The round begins here, so stamp the reply that opened it. It is
# the only contribution that is not started by the crowd, and
# before this it was the only one with no chip -- which made a
# two-model round read as an ordinary reply followed by a crowd,
# and left the reader counting "2 of 2" with no 1 in sight. Same
# turn and same `started_at`, so the bubbles group.
message.crowd_json = crowd_service.Turn(
turn=following.turn,
round=following.round,
phase=crowd_service.PHASE_OUT,
index=0,
of=following.of,
started_at=following.started_at,
).as_json()
speaker = speakers[following.index] speaker = speakers[following.index]
placeholder = chat_service.create_message( placeholder = chat_service.create_message(
db, db,
@@ -2281,10 +2304,14 @@ def _opens_the_turn(message: Message) -> bool:
"""Whether this reply is the first one answering a question. """Whether this reply is the first one answering a question.
True for every ordinary reply, and for a crowd only for the main model's True for every ordinary reply, and for a crowd only for the main model's
opening turn -- which is the one with no crowd state on it at all, because a opening turn. That reply has no crowd state while it is being written -- a
round begins when that reply *finishes*. round begins when it *finishes* -- and once the round has begun it carries the
opening stamp, which `is_opening` reads as "still the one that opens the
turn". Both are the same answer to this question, and missing the second means
a reply that has already been compacted-for and titled gets it again on the
next look.
""" """
return crowd_service.state_of(message) is None return crowd_service.scheduling_state(message) is None
def _opens_the_turn_id(generation: Generation) -> bool: def _opens_the_turn_id(generation: Generation) -> bool:
+38 -8
View File
@@ -2023,17 +2023,28 @@ BUILTIN: tuple[Fragment, ...] = (
group=GROUP_TASKS, group=GROUP_TASKS,
order=451, order=451,
hint="Added as the last turn when a member speaks on the forward pass. " hint="Added as the last turn when a member speaks on the forward pass. "
"The failure to word against is a member that repeats what has already " "Two failures to word against. One is a member that repeats what has "
"been said in different words, which is what makes a crowd feel like an " "already been said in different words, which makes a crowd an echo "
"echo rather than a second opinion.", "rather than a second opinion. The other only shows up on a request that "
"asks for something to be *made* -- write this, pick one, draft that -- "
"where a member reads the original instruction as addressed to it too "
"and produces a rival answer beside its critique. That is not a second "
"opinion either; it is two first opinions, and it is what sends a round "
"off the question.",
default=( default=(
"You are one of several models answering this. The answers above are " "You are one of several models answering this. The answers above are "
"quoted with the name of whoever wrote them; yours comes next.\n" "quoted with the name of whoever wrote them; yours comes next.\n"
"\n" "\n"
"Respond to what is above you. Do not answer the person's original "
"request again yourself — that has been done, and your turn is about "
"what was done with it.\n"
"\n"
"Add what is missing, correct what is wrong, and say what you would " "Add what is missing, correct what is wrong, and say what you would "
"have done differently. Do not restate what has already been said to " "have done differently and why. Where you would have made a different "
"show that you agree with it — if you have nothing to add, say so in " "choice, say what it would buy — naming an alternative is not the same "
"one line and stop. Be brief: somebody is reading all of these." "as giving a reason to prefer it. Do not restate what has already been "
"said to show that you agree with it — if you have nothing to add, say "
"so in one line and stop. Be brief: somebody is reading all of these."
), ),
), ),
Fragment( Fragment(
@@ -2066,11 +2077,23 @@ BUILTIN: tuple[Fragment, ...] = (
"round. Its own fragment rather than a sentence inside the one below, " "round. Its own fragment rather than a sentence inside the one below, "
"because inviting a choice a model cannot express is worse than not " "because inviting a choice a model cannot express is worse than not "
"offering it: on a model without the tools capability there is no " "offering it: on a model without the tools capability there is no "
"crowd_again to call, and that is the case the next fragment covers.", "crowd_again to call, and that is the case the next fragment covers.\n"
"\n"
"The failure to word against is capitulation: the model that opened the "
"round abandoning its own answer because somebody spoke after it. A "
"closing turn told only to synthesise will follow the last speaker, "
"which is how a crowd ends up less accurate than the model that started "
"it.",
default=( default=(
"You opened this and you are closing it. The others have answered and " "You opened this and you are closing it. The others have answered and "
"have had the chance to disagree.\n" "have had the chance to disagree.\n"
"\n" "\n"
"Your own answer is not automatically the worse one for having been "
"written first. Change your position where somebody gave you a reason, "
"and say what the reason was; agreement with no argument behind it is "
"not a reason, and neither is a member having moved on to something "
"else.\n"
"\n"
"Write the answer the person actually asked for. Take what the others " "Write the answer the person actually asked for. Take what the others "
"got right, say where you disagree with them and why, and name " "got right, say where you disagree with them and why, and name "
"anything still unresolved rather than papering over it. Attribute " "anything still unresolved rather than papering over it. Attribute "
@@ -2091,11 +2114,18 @@ BUILTIN: tuple[Fragment, ...] = (
"is reached, or this model has no tools and so cannot ask. It says the " "is reached, or this model has no tools and so cannot ask. It says the "
"answer has to be final rather than inviting a choice that would be " "answer has to be final rather than inviting a choice that would be "
"ignored, which is the difference between a feature and a feature that " "ignored, which is the difference between a feature and a feature that "
"looks like one.", "looks like one. It carries the same guard against capitulation as the "
"fragment above, and for the same reason.",
default=( default=(
"You opened this and you are closing it, and this is the last turn: " "You opened this and you are closing it, and this is the last turn: "
"there will be no further round.\n" "there will be no further round.\n"
"\n" "\n"
"Your own answer is not automatically the worse one for having been "
"written first. Change your position where somebody gave you a reason, "
"and say what the reason was; agreement with no argument behind it is "
"not a reason, and neither is a member having moved on to something "
"else.\n"
"\n"
"Write the answer the person actually asked for. Take what the others " "Write the answer the person actually asked for. Take what the others "
"got right, say where you disagree with them and why, and attribute " "got right, say where you disagree with them and why, and attribute "
"what you took from whom. Where the disagreement is unresolved, say so " "what you took from whom. Where the disagreement is unresolved, say so "
+7
View File
@@ -1085,6 +1085,13 @@ MESSAGES.update(
"%(models)s models answer each turn, over up to %(rounds)s rounds.": ( "%(models)s models answer each turn, over up to %(rounds)s rounds.": (
"Na každý ťah odpovedá %(models)s modelov, a to najviac v %(rounds)s kolách." "Na každý ťah odpovedá %(models)s modelov, a to najviac v %(rounds)s kolách."
), ),
"%(n)s of %(total)s": "%(n)s z %(total)s",
"on the way back": "na ceste späť",
"closing": "uzatvára",
"round %(n)s": "kolo %(n)s",
"no rounds left": "už žiadne kolá",
"out of time": "vypršal čas",
"two endpoints failed": "dva endpointy zlyhali",
"Check for due work every": "Kontrolovať splatnú prácu každých", "Check for due work every": "Kontrolovať splatnú prácu každých",
"How often it looks": "Ako často sa pozerá", "How often it looks": "Ako často sa pozerá",
"Nothing may repeat faster than": "Nič sa nesmie opakovať častejšie než", "Nothing may repeat faster than": "Nič sa nesmie opakovať častejšie než",
+7 -7
View File
@@ -88,24 +88,24 @@
replies: a round produces more bubbles than it has models in it. #} replies: a round produces more bubbles than it has models in it. #}
<span class="badge"> <span class="badge">
{% if crowd.get("phase") == "out" %} {% if crowd.get("phase") == "out" %}
{{ crowd.get("index", 0) + 1 }} of {{ crowd.get("of", 1) }} {{ t("%(n)s of %(total)s", n=crowd.get("index", 0) + 1, total=crowd.get("of", 1)) }}
{% elif crowd.get("phase") == "back" %} {% elif crowd.get("phase") == "back" %}
on the way back {{ t("on the way back") }}
{% else %} {% else %}
closing {{ t("closing") }}
{% endif %} {% endif %}
{% if crowd.get("round", 1) > 1 %} · round {{ crowd.get("round") }}{% endif %} {% if crowd.get("round", 1) > 1 %} · {{ t("round %(n)s", n=crowd.get("round")) }}{% endif %}
</span> </span>
{% if crowd.get("stopped") %} {% if crowd.get("stopped") %}
{# Why a round ended, where it ended. Without this a crowd that ran out of {# Why a round ended, where it ended. Without this a crowd that ran out of
rounds or time simply stops, which reads as the feature failing. #} rounds or time simply stops, which reads as the feature failing. #}
<span class="badge badge--warning" title="{{ t('The round ended here') }}"> <span class="badge badge--warning" title="{{ t('The round ended here') }}">
{% if crowd.get("stopped") == "rounds" %} {% if crowd.get("stopped") == "rounds" %}
no rounds left {{ t("no rounds left") }}
{% elif crowd.get("stopped") == "time" %} {% elif crowd.get("stopped") == "time" %}
out of time {{ t("out of time") }}
{% else %} {% else %}
two endpoints failed {{ t("two endpoints failed") }}
{% endif %} {% endif %}
</span> </span>
{% endif %} {% endif %}
+67
View File
@@ -177,6 +177,73 @@ def test_the_round_is_recorded_on_every_row(db, started):
assert len(anchors) == 1 assert len(anchors) == 1
def test_the_reply_that_opened_the_round_is_stamped_too(db, started):
"""The opening bubble says `1 of 3` like every other one.
It is the one contribution the crowd does not start -- the composer does --
so until the round begins there is nothing to stamp it with. Before this, a
two-model round rendered as an unmarked reply followed by one saying `2 of 2`,
with no 1 anywhere.
"""
chat = _crowd_chat(db)
opening = _opening_reply(db, chat)
assert crowd_service.state_of(opening) is None, "nothing to say before it finishes"
assert _advance(db, chat, opening)
db.expire_all()
state = crowd_service.state_of(opening)
assert state is not None
assert (state.phase, state.index) == (crowd_service.PHASE_OUT, 0)
assert state.of == 3
def test_the_opening_stamp_belongs_to_the_same_round(db, started):
chat = _crowd_chat(db)
opening = _opening_reply(db, chat)
order = []
assert _advance(db, chat, opening)
db.expire_all()
order = _incomplete(db, chat)
opened = crowd_service.state_of(opening)
first = crowd_service.state_of(order[0])
# Same question, same clock -- or the chips group two bubbles of one round
# under two different rounds.
assert opened.turn == first.turn
assert opened.started_at == first.started_at
assert opened.round == first.round == 1
def test_the_opening_stamp_is_not_scheduling_state(db, started):
"""It must read as "no round yet" everywhere that decides what happens next.
Fed to the scheduler it would be a member at index 0, which inherits the old
`started_at` -- so regenerating the opening an hour later would end the round
with "out of time" before anybody spoke -- and it would hand that reply a
member's tools and a member's instruction instead of an ordinary first answer.
"""
chat = _crowd_chat(db)
opening = _opening_reply(db, chat)
assert _advance(db, chat, opening)
db.expire_all()
assert crowd_service.state_of(opening) is not None
assert crowd_service.scheduling_state(opening) is None
assert crowd_service.is_opening(crowd_service.state_of(opening))
assert generation_service._opens_the_turn(opening)
def test_a_later_speaker_is_not_mistaken_for_the_opening(db, started):
chat = _crowd_chat(db)
order = _run_round(db, chat, started)
for message in order:
state = crowd_service.state_of(message)
assert not crowd_service.is_opening(state)
# `==` and not `is`: `state_of` builds a fresh Turn on every call.
assert crowd_service.scheduling_state(message) == state
def test_each_speaker_carries_its_own_connection(db, started): def test_each_speaker_carries_its_own_connection(db, started):
"""So `speaker_for` resolves the pair rather than guessing at the id.""" """So `speaker_for` resolves the pair rather than guessing at the id."""
chat = _crowd_chat(db) chat = _crowd_chat(db)
+25
View File
@@ -281,6 +281,31 @@ def test_a_bubble_on_the_way_out_says_which_speaker_it_is(db):
assert "2 of 3" in html assert "2 of 3" in html
def test_the_bubble_that_opened_the_round_says_it_is_first(db):
"""The opening reply is stamped once the round begins, so it says `1 of 3`.
Before that it was the one contribution with no chip at all, which made a
two-model round read as an ordinary answer followed by one labelled `2 of 2`.
"""
chat = _chat(db)
html = _bubble(db, chat, phase=crowd_service.PHASE_OUT, index=0, of=3)
assert "1 of 3" in html
def test_the_chip_is_translated(db):
"""It is prose a person reads, and it was English on a Slovak instance."""
from lembas.web import i18n
chat = _chat(db)
i18n.activate("sk")
try:
html = _bubble(db, chat, phase=crowd_service.PHASE_BACK, index=1, of=3)
finally:
i18n.activate("en")
assert "na ceste späť" in html
assert "on the way back" not in html
def test_a_bubble_on_the_way_back_says_so_and_is_quieter(db): def test_a_bubble_on_the_way_back_says_so_and_is_quieter(db):
chat = _chat(db) chat = _chat(db)
html = _bubble(db, chat, phase=crowd_service.PHASE_BACK) html = _bubble(db, chat, phase=crowd_service.PHASE_BACK)