Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8503c6c775
|
@@ -16,6 +16,50 @@ for 1.0.0 have something to be assembled from.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## 1.8.1
|
||||
|
||||
Three fixes to how a crowd behaves, found by reading one real round on the live
|
||||
instance rather than by testing: two models, one round, a question that asked for
|
||||
something to be *made*.
|
||||
|
||||
- **A member no longer answers the question again.** Asked to pick a language and
|
||||
write an example, the main model wrote Python; the second model gave a genuinely
|
||||
useful critique of it — and then answered the original question itself, in a
|
||||
different language. Nothing in its instruction said not to. It now says so:
|
||||
*respond to what is above you; do not answer the person's original request again
|
||||
yourself.* A member that produces a rival answer is not a second opinion, it is
|
||||
a second first opinion, and it is what takes a round off the question.
|
||||
|
||||
- **The model that opened the round no longer capitulates.** Told to write the
|
||||
final answer and take what the others got right, it abandoned its own perfectly
|
||||
good answer, wrote *"I agree that Rust is the superior choice"* with no argument
|
||||
anywhere for why, and rewrote everything in the newcomer's language. Both
|
||||
closing instructions now carry: *your own answer is not automatically the worse
|
||||
one for having been written first; change your position where somebody gave you
|
||||
a reason, and say what the reason was.*
|
||||
|
||||
This mattered more than it reads. All three answers were compiled: the original
|
||||
Python was fine, the critic's Rust compiled and ran — and **the merged answer
|
||||
that was actually delivered did not compile at all**. A crowd that ends by
|
||||
agreeing with whoever spoke last can be worse than the model that started it.
|
||||
|
||||
- **The bubble that opens a round now says `1 of 3` like every other one.** It was
|
||||
the single contribution with no chip, because the crowd does not start it — the
|
||||
composer does, and a round only begins when it finishes. So a two-model round
|
||||
read as an ordinary reply followed by one labelled `2 of 2`, with no 1 anywhere.
|
||||
It is stamped when the round begins, and that stamp is deliberately invisible to
|
||||
everything that decides what happens next: fed to the scheduler it would inherit
|
||||
the round's clock, so regenerating the opening an hour later would end the round
|
||||
with "out of time" before anybody spoke.
|
||||
|
||||
- Fixed: **the crowd chip was never translated.** `1 of 3`, `on the way back`,
|
||||
`closing`, `no rounds left` and the rest were English on a Slovak instance.
|
||||
|
||||
**Worth knowing, and not a bug:** with **two** models there is no backward pass at
|
||||
all. The way back would contain only the model that opened the round, whose turn
|
||||
*is* the close — so `crowd.disagree` never fires. You need at least three models
|
||||
before a single "do you disagree" bubble can exist.
|
||||
|
||||
## 1.8.0
|
||||
|
||||
- **The crowd is where you would look for it.** In 1.6.0 the only way to add a
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""LLeMbas - a Middle-earth themed web UI for OpenAI-compatible LLM endpoints."""
|
||||
|
||||
__version__ = "1.8.0"
|
||||
__version__ = "1.8.1"
|
||||
|
||||
@@ -588,7 +588,10 @@ def build_request(
|
||||
if crowd_turn is None and upto is not None:
|
||||
from lembas.services import crowd as crowd_service
|
||||
|
||||
crowd_turn = crowd_service.state_of(upto)
|
||||
# `scheduling_state`: the opening reply carries a stamp for the chip's
|
||||
# sake, and regenerating it must still build an ordinary first answer --
|
||||
# not one told that "the answers above are quoted, yours comes next".
|
||||
crowd_turn = crowd_service.scheduling_state(upto)
|
||||
# Images are only sent to a model an administrator has marked as having
|
||||
# vision. Sending them to one that has not is not a graceful degradation:
|
||||
# most endpoints reject the whole request.
|
||||
|
||||
@@ -134,6 +134,41 @@ def state_of(message: Message | None) -> Turn | None:
|
||||
return None
|
||||
|
||||
|
||||
def is_opening(state: Turn | None) -> bool:
|
||||
"""Whether this state is the main model's opening reply.
|
||||
|
||||
`phase=out, index=0` is **display state and never scheduling state**. The
|
||||
opening reply is not started by the crowd -- the composer starts it, exactly
|
||||
as it starts every other reply, and a round only begins when it *finishes*.
|
||||
Stamping it afterwards is what lets the transcript say `1 of 3` on the bubble
|
||||
that opened the round; before that it was the one contribution with no chip,
|
||||
so a two-model round read as an ordinary reply followed by a crowd.
|
||||
|
||||
Everything that asks "is a round already in progress?" has to skip it, or the
|
||||
stamp changes behaviour it was never meant to touch -- see `scheduling_state`.
|
||||
"""
|
||||
return state is not None and state.phase == PHASE_OUT and state.index == 0
|
||||
|
||||
|
||||
def scheduling_state(message: Message | None) -> Turn | None:
|
||||
"""The round state the scheduler should act on: `state_of`, minus the opening.
|
||||
|
||||
Two things would break if the opening stamp were fed to `next_turn` as real
|
||||
state, and both are silent:
|
||||
|
||||
* **`started_at` would be inherited on a regenerate.** Regenerating the
|
||||
opening reply an hour later would hand `next_turn` an hour-old clock and the
|
||||
round would stop with "out of time" before anybody spoke.
|
||||
* **The once-per-turn gates key off "no state at all"** -- compaction, the
|
||||
title, the unread push. A stamped opening reads as a later speaker, and each
|
||||
of them would be skipped for the turn that is supposed to have them.
|
||||
|
||||
So the stamp is written where the transcript reads it and nowhere else.
|
||||
"""
|
||||
state = state_of(message)
|
||||
return None if is_opening(state) else state
|
||||
|
||||
|
||||
def now_stamp() -> str:
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
@@ -374,9 +409,11 @@ __all__ = [
|
||||
"Turn",
|
||||
"elapsed",
|
||||
"is_newest",
|
||||
"is_opening",
|
||||
"member_speakers",
|
||||
"next_turn",
|
||||
"now_stamp",
|
||||
"scheduling_state",
|
||||
"state_of",
|
||||
"tool_defs",
|
||||
"unreachable_members",
|
||||
|
||||
@@ -661,7 +661,10 @@ async def _run(generation: Generation) -> None:
|
||||
# once, here, and used for three decisions: which tools it may have,
|
||||
# which instruction closes its request, and whether it may ask for
|
||||
# another round.
|
||||
crowd_state = crowd_service.state_of(message)
|
||||
# `scheduling_state` for the reason `build_request` gives: the
|
||||
# opening reply's stamp is for the transcript, and regenerating it
|
||||
# must not hand it a member's tools or a member's instruction.
|
||||
crowd_state = crowd_service.scheduling_state(message)
|
||||
crowd_settings = settings_store.crowd(db)
|
||||
may_ask_again = bool(
|
||||
crowd_state is not None
|
||||
@@ -2232,7 +2235,11 @@ def _advance_crowd(generation: Generation) -> bool:
|
||||
speakers = crowd_service.member_speakers(db, chat, owner_user)
|
||||
speakers = speakers[: int(settings["max_models"]) + 1]
|
||||
|
||||
state = crowd_service.state_of(message)
|
||||
# `scheduling_state` and not `state_of`: the opening reply carries a
|
||||
# stamp for the transcript's sake (so it can say `1 of 3`), and that
|
||||
# stamp must not read as "a round is already running" -- it would
|
||||
# inherit the old clock on a regenerate. See `crowd.is_opening`.
|
||||
state = crowd_service.scheduling_state(message)
|
||||
# The turn a round belongs to: the user message this all answers.
|
||||
turn_id = state.turn if state is not None else _turn_anchor(db, message)
|
||||
following = crowd_service.next_turn(
|
||||
@@ -2254,6 +2261,22 @@ def _advance_crowd(generation: Generation) -> bool:
|
||||
db.commit()
|
||||
return False
|
||||
|
||||
if state is None:
|
||||
# The round begins here, so stamp the reply that opened it. It is
|
||||
# the only contribution that is not started by the crowd, and
|
||||
# before this it was the only one with no chip -- which made a
|
||||
# two-model round read as an ordinary reply followed by a crowd,
|
||||
# and left the reader counting "2 of 2" with no 1 in sight. Same
|
||||
# turn and same `started_at`, so the bubbles group.
|
||||
message.crowd_json = crowd_service.Turn(
|
||||
turn=following.turn,
|
||||
round=following.round,
|
||||
phase=crowd_service.PHASE_OUT,
|
||||
index=0,
|
||||
of=following.of,
|
||||
started_at=following.started_at,
|
||||
).as_json()
|
||||
|
||||
speaker = speakers[following.index]
|
||||
placeholder = chat_service.create_message(
|
||||
db,
|
||||
@@ -2281,10 +2304,14 @@ def _opens_the_turn(message: Message) -> bool:
|
||||
"""Whether this reply is the first one answering a question.
|
||||
|
||||
True for every ordinary reply, and for a crowd only for the main model's
|
||||
opening turn -- which is the one with no crowd state on it at all, because a
|
||||
round begins when that reply *finishes*.
|
||||
opening turn. That reply has no crowd state while it is being written -- a
|
||||
round begins when it *finishes* -- and once the round has begun it carries the
|
||||
opening stamp, which `is_opening` reads as "still the one that opens the
|
||||
turn". Both are the same answer to this question, and missing the second means
|
||||
a reply that has already been compacted-for and titled gets it again on the
|
||||
next look.
|
||||
"""
|
||||
return crowd_service.state_of(message) is None
|
||||
return crowd_service.scheduling_state(message) is None
|
||||
|
||||
|
||||
def _opens_the_turn_id(generation: Generation) -> bool:
|
||||
|
||||
@@ -2023,17 +2023,28 @@ BUILTIN: tuple[Fragment, ...] = (
|
||||
group=GROUP_TASKS,
|
||||
order=451,
|
||||
hint="Added as the last turn when a member speaks on the forward pass. "
|
||||
"The failure to word against is a member that repeats what has already "
|
||||
"been said in different words, which is what makes a crowd feel like an "
|
||||
"echo rather than a second opinion.",
|
||||
"Two failures to word against. One is a member that repeats what has "
|
||||
"already been said in different words, which makes a crowd an echo "
|
||||
"rather than a second opinion. The other only shows up on a request that "
|
||||
"asks for something to be *made* -- write this, pick one, draft that -- "
|
||||
"where a member reads the original instruction as addressed to it too "
|
||||
"and produces a rival answer beside its critique. That is not a second "
|
||||
"opinion either; it is two first opinions, and it is what sends a round "
|
||||
"off the question.",
|
||||
default=(
|
||||
"You are one of several models answering this. The answers above are "
|
||||
"quoted with the name of whoever wrote them; yours comes next.\n"
|
||||
"\n"
|
||||
"Respond to what is above you. Do not answer the person's original "
|
||||
"request again yourself — that has been done, and your turn is about "
|
||||
"what was done with it.\n"
|
||||
"\n"
|
||||
"Add what is missing, correct what is wrong, and say what you would "
|
||||
"have done differently. Do not restate what has already been said to "
|
||||
"show that you agree with it — if you have nothing to add, say so in "
|
||||
"one line and stop. Be brief: somebody is reading all of these."
|
||||
"have done differently and why. Where you would have made a different "
|
||||
"choice, say what it would buy — naming an alternative is not the same "
|
||||
"as giving a reason to prefer it. Do not restate what has already been "
|
||||
"said to show that you agree with it — if you have nothing to add, say "
|
||||
"so in one line and stop. Be brief: somebody is reading all of these."
|
||||
),
|
||||
),
|
||||
Fragment(
|
||||
@@ -2066,11 +2077,23 @@ BUILTIN: tuple[Fragment, ...] = (
|
||||
"round. Its own fragment rather than a sentence inside the one below, "
|
||||
"because inviting a choice a model cannot express is worse than not "
|
||||
"offering it: on a model without the tools capability there is no "
|
||||
"crowd_again to call, and that is the case the next fragment covers.",
|
||||
"crowd_again to call, and that is the case the next fragment covers.\n"
|
||||
"\n"
|
||||
"The failure to word against is capitulation: the model that opened the "
|
||||
"round abandoning its own answer because somebody spoke after it. A "
|
||||
"closing turn told only to synthesise will follow the last speaker, "
|
||||
"which is how a crowd ends up less accurate than the model that started "
|
||||
"it.",
|
||||
default=(
|
||||
"You opened this and you are closing it. The others have answered and "
|
||||
"have had the chance to disagree.\n"
|
||||
"\n"
|
||||
"Your own answer is not automatically the worse one for having been "
|
||||
"written first. Change your position where somebody gave you a reason, "
|
||||
"and say what the reason was; agreement with no argument behind it is "
|
||||
"not a reason, and neither is a member having moved on to something "
|
||||
"else.\n"
|
||||
"\n"
|
||||
"Write the answer the person actually asked for. Take what the others "
|
||||
"got right, say where you disagree with them and why, and name "
|
||||
"anything still unresolved rather than papering over it. Attribute "
|
||||
@@ -2091,11 +2114,18 @@ BUILTIN: tuple[Fragment, ...] = (
|
||||
"is reached, or this model has no tools and so cannot ask. It says the "
|
||||
"answer has to be final rather than inviting a choice that would be "
|
||||
"ignored, which is the difference between a feature and a feature that "
|
||||
"looks like one.",
|
||||
"looks like one. It carries the same guard against capitulation as the "
|
||||
"fragment above, and for the same reason.",
|
||||
default=(
|
||||
"You opened this and you are closing it, and this is the last turn: "
|
||||
"there will be no further round.\n"
|
||||
"\n"
|
||||
"Your own answer is not automatically the worse one for having been "
|
||||
"written first. Change your position where somebody gave you a reason, "
|
||||
"and say what the reason was; agreement with no argument behind it is "
|
||||
"not a reason, and neither is a member having moved on to something "
|
||||
"else.\n"
|
||||
"\n"
|
||||
"Write the answer the person actually asked for. Take what the others "
|
||||
"got right, say where you disagree with them and why, and attribute "
|
||||
"what you took from whom. Where the disagreement is unresolved, say so "
|
||||
|
||||
@@ -1085,6 +1085,13 @@ MESSAGES.update(
|
||||
"%(models)s models answer each turn, over up to %(rounds)s rounds.": (
|
||||
"Na každý ťah odpovedá %(models)s modelov, a to najviac v %(rounds)s kolách."
|
||||
),
|
||||
"%(n)s of %(total)s": "%(n)s z %(total)s",
|
||||
"on the way back": "na ceste späť",
|
||||
"closing": "uzatvára",
|
||||
"round %(n)s": "kolo %(n)s",
|
||||
"no rounds left": "už žiadne kolá",
|
||||
"out of time": "vypršal čas",
|
||||
"two endpoints failed": "dva endpointy zlyhali",
|
||||
"Check for due work every": "Kontrolovať splatnú prácu každých",
|
||||
"How often it looks": "Ako často sa pozerá",
|
||||
"Nothing may repeat faster than": "Nič sa nesmie opakovať častejšie než",
|
||||
|
||||
@@ -88,24 +88,24 @@
|
||||
replies: a round produces more bubbles than it has models in it. #}
|
||||
<span class="badge">
|
||||
{% if crowd.get("phase") == "out" %}
|
||||
{{ crowd.get("index", 0) + 1 }} of {{ crowd.get("of", 1) }}
|
||||
{{ t("%(n)s of %(total)s", n=crowd.get("index", 0) + 1, total=crowd.get("of", 1)) }}
|
||||
{% elif crowd.get("phase") == "back" %}
|
||||
on the way back
|
||||
{{ t("on the way back") }}
|
||||
{% else %}
|
||||
closing
|
||||
{{ t("closing") }}
|
||||
{% endif %}
|
||||
{% if crowd.get("round", 1) > 1 %} · round {{ crowd.get("round") }}{% endif %}
|
||||
{% if crowd.get("round", 1) > 1 %} · {{ t("round %(n)s", n=crowd.get("round")) }}{% endif %}
|
||||
</span>
|
||||
{% if crowd.get("stopped") %}
|
||||
{# Why a round ended, where it ended. Without this a crowd that ran out of
|
||||
rounds or time simply stops, which reads as the feature failing. #}
|
||||
<span class="badge badge--warning" title="{{ t('The round ended here') }}">
|
||||
{% if crowd.get("stopped") == "rounds" %}
|
||||
no rounds left
|
||||
{{ t("no rounds left") }}
|
||||
{% elif crowd.get("stopped") == "time" %}
|
||||
out of time
|
||||
{{ t("out of time") }}
|
||||
{% else %}
|
||||
two endpoints failed
|
||||
{{ t("two endpoints failed") }}
|
||||
{% endif %}
|
||||
</span>
|
||||
{% endif %}
|
||||
|
||||
@@ -177,6 +177,73 @@ def test_the_round_is_recorded_on_every_row(db, started):
|
||||
assert len(anchors) == 1
|
||||
|
||||
|
||||
def test_the_reply_that_opened_the_round_is_stamped_too(db, started):
|
||||
"""The opening bubble says `1 of 3` like every other one.
|
||||
|
||||
It is the one contribution the crowd does not start -- the composer does --
|
||||
so until the round begins there is nothing to stamp it with. Before this, a
|
||||
two-model round rendered as an unmarked reply followed by one saying `2 of 2`,
|
||||
with no 1 anywhere.
|
||||
"""
|
||||
chat = _crowd_chat(db)
|
||||
opening = _opening_reply(db, chat)
|
||||
assert crowd_service.state_of(opening) is None, "nothing to say before it finishes"
|
||||
|
||||
assert _advance(db, chat, opening)
|
||||
db.expire_all()
|
||||
|
||||
state = crowd_service.state_of(opening)
|
||||
assert state is not None
|
||||
assert (state.phase, state.index) == (crowd_service.PHASE_OUT, 0)
|
||||
assert state.of == 3
|
||||
|
||||
|
||||
def test_the_opening_stamp_belongs_to_the_same_round(db, started):
|
||||
chat = _crowd_chat(db)
|
||||
opening = _opening_reply(db, chat)
|
||||
order = []
|
||||
assert _advance(db, chat, opening)
|
||||
db.expire_all()
|
||||
order = _incomplete(db, chat)
|
||||
|
||||
opened = crowd_service.state_of(opening)
|
||||
first = crowd_service.state_of(order[0])
|
||||
# Same question, same clock -- or the chips group two bubbles of one round
|
||||
# under two different rounds.
|
||||
assert opened.turn == first.turn
|
||||
assert opened.started_at == first.started_at
|
||||
assert opened.round == first.round == 1
|
||||
|
||||
|
||||
def test_the_opening_stamp_is_not_scheduling_state(db, started):
|
||||
"""It must read as "no round yet" everywhere that decides what happens next.
|
||||
|
||||
Fed to the scheduler it would be a member at index 0, which inherits the old
|
||||
`started_at` -- so regenerating the opening an hour later would end the round
|
||||
with "out of time" before anybody spoke -- and it would hand that reply a
|
||||
member's tools and a member's instruction instead of an ordinary first answer.
|
||||
"""
|
||||
chat = _crowd_chat(db)
|
||||
opening = _opening_reply(db, chat)
|
||||
assert _advance(db, chat, opening)
|
||||
db.expire_all()
|
||||
|
||||
assert crowd_service.state_of(opening) is not None
|
||||
assert crowd_service.scheduling_state(opening) is None
|
||||
assert crowd_service.is_opening(crowd_service.state_of(opening))
|
||||
assert generation_service._opens_the_turn(opening)
|
||||
|
||||
|
||||
def test_a_later_speaker_is_not_mistaken_for_the_opening(db, started):
|
||||
chat = _crowd_chat(db)
|
||||
order = _run_round(db, chat, started)
|
||||
for message in order:
|
||||
state = crowd_service.state_of(message)
|
||||
assert not crowd_service.is_opening(state)
|
||||
# `==` and not `is`: `state_of` builds a fresh Turn on every call.
|
||||
assert crowd_service.scheduling_state(message) == state
|
||||
|
||||
|
||||
def test_each_speaker_carries_its_own_connection(db, started):
|
||||
"""So `speaker_for` resolves the pair rather than guessing at the id."""
|
||||
chat = _crowd_chat(db)
|
||||
|
||||
@@ -281,6 +281,31 @@ def test_a_bubble_on_the_way_out_says_which_speaker_it_is(db):
|
||||
assert "2 of 3" in html
|
||||
|
||||
|
||||
def test_the_bubble_that_opened_the_round_says_it_is_first(db):
|
||||
"""The opening reply is stamped once the round begins, so it says `1 of 3`.
|
||||
|
||||
Before that it was the one contribution with no chip at all, which made a
|
||||
two-model round read as an ordinary answer followed by one labelled `2 of 2`.
|
||||
"""
|
||||
chat = _chat(db)
|
||||
html = _bubble(db, chat, phase=crowd_service.PHASE_OUT, index=0, of=3)
|
||||
assert "1 of 3" in html
|
||||
|
||||
|
||||
def test_the_chip_is_translated(db):
|
||||
"""It is prose a person reads, and it was English on a Slovak instance."""
|
||||
from lembas.web import i18n
|
||||
|
||||
chat = _chat(db)
|
||||
i18n.activate("sk")
|
||||
try:
|
||||
html = _bubble(db, chat, phase=crowd_service.PHASE_BACK, index=1, of=3)
|
||||
finally:
|
||||
i18n.activate("en")
|
||||
assert "na ceste späť" in html
|
||||
assert "on the way back" not in html
|
||||
|
||||
|
||||
def test_a_bubble_on_the_way_back_says_so_and_is_quieter(db):
|
||||
chat = _chat(db)
|
||||
html = _bubble(db, chat, phase=crowd_service.PHASE_BACK)
|
||||
|
||||
Reference in New Issue
Block a user