Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Draw a picture, on a ComfyUI you are running

Browse Source 
The last unbuilt capability, and built the way CLAUDE.md said it had to be: a
ToolDef reaching resolve_tools plus a permission and a capability flag, not a new
code path. The only genuinely new UI is one branch in the transcript.

services/images/ is three modules. comfy.py speaks HTTP -- submit, poll /history,
fetch the PNG, /free, and an /object_info discovery for the admin page only.
Polled and not socketed, because holding a connection open for the length of a
generation is the live-connection state the whole ssh.py design forbids, and the
thing being waited for takes tens of seconds anyway. The base URL is exempt from
the SSRF guard by construction, exactly as Connection.base_url and the audio
endpoints are -- said out loud in the docstring, because a default of
127.0.0.1:8188 is precisely the shape that guard exists to refuse and therefore
reads as a hole rather than a decision.

workflow.py fills a template, and the one thing that matters is that it walks the
parsed JSON rather than the text of it. A value that is exactly "{{steps}}"
becomes the number 20; ComfyUI validates types and refuses the string. A
placeholder inside a longer string is still text, which is what makes
"{{prompt}}, masterpiece" work -- and text substitution would additionally mean a
prompt containing a quotation mark produced a document that no longer parses, on
the one input guaranteed to hold arbitrary text. Which node holds the prompt is
the administrator's statement rather than a guess from node types: sniffing for
the first CLIPTextEncode works on the shipped workflow and on nothing else, and
swaps positive for negative the first time somebody reorders them. seed has no
fixed default, because one would make every unspecified generation identical and
make the retry loop redraw the same rejected picture four times.

tool.py is one call, one finished image. Returning every attempt to the
conversation would cost a round each, make the ceiling advisory rather than
enforced, and walk the reader past every reject -- so the reviewer lives inside
the tool and is asked about *bytes*: an attempt about to be discarded should not
leave an Attachment behind, so it sees a downscaled preview built in memory and
only the kept image is written. Anything that goes wrong in review is a keep;
losing a picture because a judging request timed out would be the check
destroying the thing it was checking. The last attempt is kept whatever the
verdict, so a request always produces something. Rejects are recorded, not
stored.

Preserve VRAM unloads the chat's own connection and nothing else, because the
memory being freed belongs to one machine: local llama-swap answers GET /unload,
and a box on the network has no reason to be unloaded when ComfyUI wants memory
here. The swap goes round the review rather than round the tool, which costs two
model loads per retry -- so the two settings are independent and the page warns
when both are on. Nothing loads the LLM back: the reply's next request does, and
that step exists in the description and not in the code, so the code says so.

Two rules elsewhere had to be drawn for the first time. message_payload sends
images only on user turns -- no assistant message had ever carried one, and the
moment one does the multimodal list form on an assistant turn is rejected by
OpenAI and most local runners, breaking every later turn in the chat. And
files.store gained keep_original, because _process_image turns anything without
alpha into JPEG q85 at 1400px: right for a phone photo, a visible loss on the one
output this feature exists to produce.

/image sends the ordinary message with force_tool, which becomes tool_choice for
the first round only -- left in place the reply would draw a picture, be asked
again, and draw another. FORCEABLE_TOOLS is an allow list because the name is
read off a form.

ToolContext gained chat_id, and that fixed a tool nobody had ever successfully
run: _run_scratch_write read context.chat_id on a dataclass with no such field,
so every call raised AttributeError, swallowed by run_tool's blanket except into
"the scratch_write tool failed" -- indistinguishable from a model calling it
wrongly. The test that existed asserted the family and the risk, which are
properties of the declaration rather than of the code.

Verified against the real ComfyUI 0.27.0 on this machine rather than against
documentation: every endpoint shape here was read off it, a generation ran end to
end through the client, the reviewer was shown a matching and a mismatched prompt
and answered KEEP and RETRY correctly, and the unload hook fired for the local
llama-swap and not for the remote box.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-05 14:13:19 +02:00
parent 9f5ff72e32
commit 47d1ddbc3c
38 changed files with 3958 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/prompts.py
+105
View File
@@ -218,6 +218,24 @@ VARIABLES: tuple[Variable, ...] = (
"Skill index",
"Each available skill's name and when to use it, one per line.",
),
Variable(
"image_templates",
"Image templates",
"Each enabled image workflow's name and what it is for, one per line. "
"Empty when none has been set up.",
),
Variable(
"image_models",
"Image checkpoints",
"The checkpoints an administrator has listed on the image generation "
"page, comma separated.",
),
Variable(
"image_instructions",
"Image house rules",
"Whatever an administrator wrote in the Extra instructions box on the "
"image generation page. Empty when they wrote nothing.",
),
Variable(
"knowledge_bases",
"Knowledge bases",
@@ -1001,6 +1019,65 @@ BUILTIN: tuple[Fragment, ...] = (
"since that is all you will see next time."
),
),
Fragment(
key="tool.image",
label="Generating an image",
group=GROUP_TOOLS,
order=243,
families=("image",),
hint="Appears when image generation is offered. The sentence about the "
"picture already being on screen is the one that earns its place: "
"without it the commonest thing a model does next is offer to show you "
"the image, which it has no way of doing and which has already "
"happened.",
default=(
"- You can draw a picture with image_generate. Describe what you want in "
"the prompt as fully as you can — subject, setting, lighting, style — "
"because the prompt is the whole of what the picture is made from.\n"
"- The picture appears in the conversation as soon as the tool returns. "
"It is already on screen: do not offer to show it, link to it or "
"describe how to open it.\n"
"- Only the prompt is required. Everything else has a sensible default, "
"so set a parameter when you have a reason to and leave it out "
"otherwise. Repeat a seed to get the same picture again."
),
),
Fragment(
key="tool.image_choices",
label="Image models and templates",
group=GROUP_TOOLS,
order=244,
families=("image",),
requires=("image_templates",),
variables=("image_templates", "image_models"),
hint="Only once there is at least one workflow to choose between. Split "
"from the fragment above for the reason `tool.skills` is split from "
"`tool.skills_write`: an instance with one template should not be told "
"to weigh up its options, and a list that is not there is worse than no "
"sentence about it.",
default=(
"- The templates you can draw with, and what each is for:\n"
"{{image_templates}}\n"
"- The checkpoints you can name: {{image_models}}\n"
"- Choose the template and checkpoint that suit what is being asked for. "
"If none obviously fits, leave both out and the usual ones are used."
),
),
Fragment(
key="tool.image_instructions",
label="Image generation: house rules",
group=GROUP_TOOLS,
order=245,
families=("image",),
requires=("image_instructions",),
variables=("image_instructions",),
hint="Whatever an administrator wrote in the Extra instructions box on "
"the image generation page. Absent entirely when that box is empty, "
"which is why this is a fragment of its own rather than a paragraph in "
"the one above -- an empty heading saying nothing is worse than no "
"heading.",
default="{{image_instructions}}",
),
Fragment(
key="tool.scratch",
label="The scratch document",
@@ -1325,6 +1402,34 @@ BUILTIN: tuple[Fragment, ...] = (
"Assistant: {{answer}}"
),
),
Fragment(
key="task.image_review",
label="Reviewing a generated image",
group=GROUP_TASKS,
order=405,
hint="A separate one-message request carrying the picture that was just "
"made, asked of a vision model before the reader is shown anything. "
"Clear it to stop reviewing: the first image is then kept, which is "
"what happens anyway when nothing on the instance has vision. The bias "
"towards KEEP is deliberate — a reviewer that retries on taste rather "
"than on faults spends somebody's GPU four times over and usually ends "
"up back at the first image.",
default=(
"You are checking a picture that was just generated against the request "
"it was generated from. Judge only whether it is a competent attempt at "
"what was asked for.\n"
"\n"
"Answer on the first line with one word: KEEP or RETRY. If RETRY, put "
"one short sentence on the second line saying what is wrong.\n"
"\n"
"Say RETRY only for something clearly wrong: the subject that was asked "
"for is missing, the image is mangled or unreadable, or it shows "
"something quite different from the request. Say KEEP for anything that "
"answers the request, including work you would have composed "
"differently. Taste is not a fault, and there is no guarantee the next "
"attempt will be better."
),
),
Fragment(
key="task.compact",
label="Compaction summary",