Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Draw a picture, on a ComfyUI you are running

Browse Source 
The last unbuilt capability, and built the way CLAUDE.md said it had to be: a
ToolDef reaching resolve_tools plus a permission and a capability flag, not a new
code path. The only genuinely new UI is one branch in the transcript.

services/images/ is three modules. comfy.py speaks HTTP -- submit, poll /history,
fetch the PNG, /free, and an /object_info discovery for the admin page only.
Polled and not socketed, because holding a connection open for the length of a
generation is the live-connection state the whole ssh.py design forbids, and the
thing being waited for takes tens of seconds anyway. The base URL is exempt from
the SSRF guard by construction, exactly as Connection.base_url and the audio
endpoints are -- said out loud in the docstring, because a default of
127.0.0.1:8188 is precisely the shape that guard exists to refuse and therefore
reads as a hole rather than a decision.

workflow.py fills a template, and the one thing that matters is that it walks the
parsed JSON rather than the text of it. A value that is exactly "{{steps}}"
becomes the number 20; ComfyUI validates types and refuses the string. A
placeholder inside a longer string is still text, which is what makes
"{{prompt}}, masterpiece" work -- and text substitution would additionally mean a
prompt containing a quotation mark produced a document that no longer parses, on
the one input guaranteed to hold arbitrary text. Which node holds the prompt is
the administrator's statement rather than a guess from node types: sniffing for
the first CLIPTextEncode works on the shipped workflow and on nothing else, and
swaps positive for negative the first time somebody reorders them. seed has no
fixed default, because one would make every unspecified generation identical and
make the retry loop redraw the same rejected picture four times.

tool.py is one call, one finished image. Returning every attempt to the
conversation would cost a round each, make the ceiling advisory rather than
enforced, and walk the reader past every reject -- so the reviewer lives inside
the tool and is asked about *bytes*: an attempt about to be discarded should not
leave an Attachment behind, so it sees a downscaled preview built in memory and
only the kept image is written. Anything that goes wrong in review is a keep;
losing a picture because a judging request timed out would be the check
destroying the thing it was checking. The last attempt is kept whatever the
verdict, so a request always produces something. Rejects are recorded, not
stored.

Preserve VRAM unloads the chat's own connection and nothing else, because the
memory being freed belongs to one machine: local llama-swap answers GET /unload,
and a box on the network has no reason to be unloaded when ComfyUI wants memory
here. The swap goes round the review rather than round the tool, which costs two
model loads per retry -- so the two settings are independent and the page warns
when both are on. Nothing loads the LLM back: the reply's next request does, and
that step exists in the description and not in the code, so the code says so.

Two rules elsewhere had to be drawn for the first time. message_payload sends
images only on user turns -- no assistant message had ever carried one, and the
moment one does the multimodal list form on an assistant turn is rejected by
OpenAI and most local runners, breaking every later turn in the chat. And
files.store gained keep_original, because _process_image turns anything without
alpha into JPEG q85 at 1400px: right for a phone photo, a visible loss on the one
output this feature exists to produce.

/image sends the ordinary message with force_tool, which becomes tool_choice for
the first round only -- left in place the reply would draw a picture, be asked
again, and draw another. FORCEABLE_TOOLS is an allow list because the name is
read off a form.

ToolContext gained chat_id, and that fixed a tool nobody had ever successfully
run: _run_scratch_write read context.chat_id on a dataclass with no such field,
so every call raised AttributeError, swallowed by run_tool's blanket except into
"the scratch_write tool failed" -- indistinguishable from a model calling it
wrongly. The test that existed asserted the family and the risk, which are
properties of the declaration rather than of the code.

Verified against the real ComfyUI 0.27.0 on this machine rather than against
documentation: every endpoint shape here was read off it, a generation ran end to
end through the client, the reviewer was shown a matching and a mismatched prompt
and answered KEEP and RETRY correctly, and the unload hook fired for the local
llama-swap and not for the remote box.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-05 14:13:19 +02:00
parent f4bf1bf670
commit 5e75948069
40 changed files with 4081 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+97 -9
View File
@@ -20,7 +20,7 @@ lembas info # paths + counts, useful when confused
lembas secret-key # generate LEMBAS_SECRET_KEY
lembas create-admin # create or promote an admin
pytest # 1529 tests, ~93s
pytest # 1676 tests, ~102s
# PLAN.md tracks what is and is not built
ruff check . # lint (line length 100)
python scripts/build_artwork.py # regenerate artwork (SVG + PWA icons;
@@ -85,6 +85,7 @@ src/lembas/
admin_users.py users, groups, permissions
admin_audio.py speech-to-text and text-to-speech endpoints
admin_search.py web search provider and credentials
admin_images.py the ComfyUI, and the workflow templates on it
admin_prompts.py the prompt fragment editor and its preview
admin_suggestions.py the cards offered on the new-chat screen
admin_tools.py custom HTTP tools and MCP servers
@@ -104,6 +105,9 @@ src/lembas/
services/
llm/openai_client.py httpx streaming + model discovery
search/ ddgs, SearXNG and Firecrawl behind one shape
images/ drawing on a ComfyUI: comfy.py speaks HTTP,
workflow.py fills a template, tool.py ties them
to a chat and decides whether to keep the result
library/ documents, notes, memories, skills, FTS
mcp/ remote MCP servers: framing, transport, rows to tools
agent/ agent chats: the mode table, SSH, the six tools,
@@ -1938,6 +1942,91 @@ have to have been watching to understand. It is the one thing in a tool result
that is genuinely *not* untrusted: it is the reader's own words, so it is stated
as theirs and needs no fence.
**Image generation is a ComfyUI workflow with holes in it, and the holes are the
administrator's statement.** `services/images/` is three modules: `comfy.py`
speaks HTTP, `workflow.py` fills a template, `tool.py` ties them to a chat.
Which node holds the prompt is *declared* with `{{prompt}}` rather than sniffed
by node type — looking for the first `CLIPTextEncode` works on the shipped
workflow and on nothing else, and swaps positive for negative the first time
somebody reorders them.
**Substitution walks the parsed JSON, not the text of it.** A value that is
*exactly* `"{{steps}}"` becomes the number 20; ComfyUI validates types and
refuses the string. A placeholder inside a longer string is still text, which is
what makes `"{{prompt}}, masterpiece"` work. Doing it textually would also mean
a prompt containing a quotation mark produced a document that no longer parses,
on the one input guaranteed to hold arbitrary text. `seed` has no fixed default
— one would make every unspecified generation identical and make the retry loop
redraw the same rejected picture four times.
**One call is one finished image, and the retrying is inside the tool.**
Returning every attempt to the conversation would cost a round each, make the
ceiling advisory rather than enforced, and walk the reader past every reject. So
the reviewer — the admin's chosen vision model, else the chat's own if it has
vision, else nobody — is asked about *bytes* rather than about a row: an attempt
about to be discarded should not leave an `Attachment` behind, so it sees a
downscaled preview built in memory and only the kept image is written. Anything
that goes wrong in review is a **keep**; losing a picture because a judging
request timed out would be the check destroying the thing it was checking. The
last attempt is kept whatever the verdict, so a request always produces
something. Rejected images are not stored — their verdicts are, in `event.text`.
**`task.image_review` is a `GROUP_TASKS` fragment**, so it is editable and
excluded from the harness, exactly like `task.title` and `task.compact` — and
clearing it switches reviewing off, the same way clearing `task.compact` switches
compaction off. It is biased hard towards KEEP on purpose: a reviewer that
retries on taste spends the GPU four times and usually ends up back at the first
image.
**Preserve VRAM unloads the chat's own connection and nothing else.**
`Connection.unload_url` is a column because the memory being freed belongs to one
machine: a local llama-swap answers `GET /unload`, and a box on the network has
no reason to be unloaded when ComfyUI wants memory *here*. Empty means "cannot be
unloaded", which is the honest default — there is no call that works everywhere.
The swap goes round the *review*, not round the tool: unload, generate, free
ComfyUI, ask the reviewer (which loads the LLM again), round again if it said no.
Two model loads per retry, which is why the two settings are independent and the
page says so when both are on. **Nothing loads the LLM back at the end** — the
reply's next request does, and llama-swap loads on demand; that step exists in
the description and not in the code, which is why the code says so.
**A generated image rides on the assistant message, so `message_payload` sends
images only on `user` turns.** No assistant message had ever carried one before,
so the distinction had never been drawn — and the moment one does, the
multimodal list form on an `assistant` turn is rejected by OpenAI and most local
runners, breaking not that turn but every later one in the chat. What follows and
is worth knowing: on a *later* turn the model cannot see the picture it made
(tool results are not replayed either), so "make it bluer" regenerates rather
than edits. Honest for a text-to-image workflow with no img2img path.
**The runner writes the file; only the loop says which turn owns it.**
`event["attachment_id"]` is carried by `generation._run` exactly as
`event["canvas"]` and `event["plan"]` are, because `_persist` is the single
writer. `_bind_attachments` narrows on this chat and on rows still unbound, for
the reason `files.claim` does: the ids arrive on a dict a runner built.
**`files.store(keep_original=True)` skips the resize and the transcode, and
nothing else.** `_process_image` turns anything without alpha into JPEG q85 at
1400px, which is right for a phone photo and a visible loss on generated art.
Pillow still opens it, so a malformed file is still refused and the dimensions
are still measured rather than claimed.
**`/image` forces one tool for one round.** It sends the ordinary message with
`force_tool`, which becomes `tool_choice` — reusing the whole loop rather than
inventing a second generation path. `FORCEABLE_TOOLS` is an allow list because
this is read off a form, and `resolve_tools` still decides whether the tool
exists, so forcing one that was never offered does nothing. `payload.pop(
"tool_choice")` after the first round is load-bearing: left in place the reply
would draw a picture, be asked again, and draw another.
**`ToolContext` gained `chat_id`, and that fixed a tool nobody had ever run.**
`_run_scratch_write` read `context.chat_id` on a dataclass that had no such
field, so **every `scratch_write` call raised `AttributeError`** — swallowed by
`run_tool`'s blanket except into "the scratch_write tool failed", which reads
exactly like a model calling it wrongly. The test that existed asserted the
family and the risk, which are properties of the declaration rather than of the
code. There is one that runs it now.
**A control wired to a method its route does not serve fails silently.** The
agent-mode select posted with `hx-post` against a route that only answers
`PATCH`, so every change returned 405 and the mode never moved — for the whole
@@ -2058,14 +2147,13 @@ notes describe the machine.
## Not built yet
Image generation, and a nav entry marks where it goes. The tool loop in
`services/generation.py` is what a new capability plugs into — a tool is a
`ToolDef` reaching `tools.resolve_tools()` plus a permission and a capability
flag, not a new code path. Its guidance is the same shape: a
`prompts.register_source` yielding one `Fragment` per row puts it in the harness,
on the admin page and in the preview without touching the assembler, the save
handler or a template. Custom HTTP tools, MCP servers and agent chats are the
three worked examples.
Nothing large. The tool loop in `services/generation.py` is what a new
capability plugs into — a tool is a `ToolDef` reaching `tools.resolve_tools()`
plus a permission and a capability flag, not a new code path. Its guidance is
the same shape: a `prompts.register_source` yielding one `Fragment` per row puts
it in the harness, on the admin page and in the preview without touching the
assembler, the save handler or a template. Custom HTTP tools, MCP servers, agent
chats and image generation are the four worked examples.
**Nothing executes on this machine, and that is the design.** Agent chats run
their commands on a host reached over SSH. A local sandbox was designed in