LLeMbas/docs/notes/image-generation.md

Image generation

Split out of CLAUDE.md -- same document, same rules, kept here because that file is loaded in full on every session and this part is only wanted when you are working on drawing on a ComfyUI. Read it before you do.

Covers services/images/ -- comfy.py, workflow.py, tool.py -- and api/admin_images.py.

Image generation is a ComfyUI workflow with holes in it, and the holes are the administrator's statement. services/images/ is three modules: comfy.py speaks HTTP, workflow.py fills a template, tool.py ties them to a chat. Which node holds the prompt is declared with {{prompt}} rather than sniffed by node type — looking for the first CLIPTextEncode works on the shipped workflow and on nothing else, and swaps positive for negative the first time somebody reorders them.

Substitution walks the parsed JSON, not the text of it. A value that is exactly "{{steps}}" becomes the number 20; ComfyUI validates types and refuses the string. A placeholder inside a longer string is still text, which is what makes "{{prompt}}, masterpiece" work. Doing it textually would also mean a prompt containing a quotation mark produced a document that no longer parses, on the one input guaranteed to hold arbitrary text. seed has no fixed default — one would make every unspecified generation identical and make the retry loop redraw the same rejected picture four times. A negative seed means random, because -1 is what ComfyUI's own interface, A1111 and everything else that has ever asked for a seed use for it, so a model that has read any of them writes it: without that it went through the uint64 wrap and arrived as 18446744073709551615, a perfectly valid fixed seed, so "give me something new" returned the same picture every time.

One call is one finished image, and the retrying is inside the tool. Returning every attempt to the conversation would cost a round each, make the ceiling advisory rather than enforced, and walk the reader past every reject. So the reviewer — the admin's chosen vision model, else the chat's own if it has vision, else nobody — is asked about bytes rather than about a row: an attempt about to be discarded should not leave an Attachment behind, so it sees a downscaled preview built in memory and only the kept image is written. Anything that goes wrong in review is a keep; losing a picture because a judging request timed out would be the check destroying the thing it was checking. The last attempt is kept whatever the verdict, so a request always produces something. Rejected images are not stored — their verdicts are, in event.text.

task.image_review is a GROUP_TASKS fragment, so it is editable and excluded from the harness, exactly like task.title and task.compact — and clearing it switches reviewing off, the same way clearing task.compact switches compaction off. It is biased hard towards KEEP on purpose: a reviewer that retries on taste spends the GPU four times and usually ends up back at the first image.

A failed generation is completed: false for ever, so waiting on that flag hangs the reply. ComfyUI writes its history entry in task_done and nowhere else, so the entry appearing is "finished" — but it sets completed=e.success, which means an out-of-memory, a cancelled job and a broken node all stay incomplete permanently. The first version waited on the flag, so every failure sat for the full 600s timeout and then reported a timeout, when ComfyUI had known within one second and written down exactly what happened. The terminal condition is now a record with a status, and status.messages is read for the last execution_error or execution_interrupted in it, which carries the node and the exception.

Two failures get their own class because they have an obvious next move. OutOfMemory — matched on exception_type, not on the message, which is a paragraph of allocator advice addressed to whoever runs the box — makes the tool tell the model to retry at a named smaller size (worked out from what it actually asked for, because "use a lower resolution" against a request that was already 512x512 is advice nobody can follow) or with a lighter checkpoint. Interrupted is not a fault at all: somebody pressed stop, and the model is told not to simply start it again. Everything else gets the reason and no advice — a model told to "try again" after a broken workflow tries the identical thing, and a suggestion invented for a failure nobody understands is a guess wearing the application's authority.

A tool's parameter descriptions are instructions, and terse ones are why a model sends only the prompt. "cfg: prompt adherence, default 8" tells a model nothing it can act on. Measured against a 4B model on the same request: with the terse descriptions it sent prompt and template and nothing else — meaning 512x512 defaults on an SDXL checkpoint, which is precisely the duplicated-limbs failure the width description now warns about. With descriptions that say what each value does to the picture and when to move it, the same model sent a portrait 1024x1536 and a deliberate sampler. It costs ~3KB of schema per request in a chat that can draw, and it is the difference between having ten parameters and having one. docs/image-generation-instructions.md is the long version, to paste into the admin instructions box for models that need more than the harness can afford to carry.

Preserve VRAM unloads the chat's own connection and nothing else. Connection.unload_url is a column because the memory being freed belongs to one machine: a local llama-swap answers GET /unload, and a box on the network has no reason to be unloaded when ComfyUI wants memory here. Empty means "cannot be unloaded", which is the honest default — there is no call that works everywhere. The swap goes round the review, not round the tool: unload, generate, free ComfyUI, ask the reviewer (which loads the LLM again), round again if it said no. Two model loads per retry, which is why the two settings are independent and the page says so when both are on. Nothing loads the LLM back at the end — the reply's next request does, and llama-swap loads on demand; that step exists in the description and not in the code, which is why the code says so.

A generated image rides on the assistant message, so message_payload sends images only on user turns. No assistant message had ever carried one before, so the distinction had never been drawn — and the moment one does, the multimodal list form on an assistant turn is rejected by OpenAI and most local runners, breaking not that turn but every later one in the chat. What follows and is worth knowing: on a later turn the model cannot see the picture it made (tool results are not replayed either), so "make it bluer" regenerates rather than edits. Honest for a text-to-image workflow with no img2img path.

The runner writes the file; only the loop says which turn owns it. event["attachment_id"] is carried by generation._run exactly as event["canvas"] and event["plan"] are, because _persist is the single writer. _bind_attachments narrows on this chat and on rows still unbound, for the reason files.claim does: the ids arrive on a dict a runner built.

files.store(keep_original=True) skips the resize and the transcode, and nothing else. _process_image turns anything without alpha into JPEG q85 at 1400px, which is right for a phone photo and a visible loss on generated art. Pillow still opens it, so a malformed file is still refused and the dimensions are still measured rather than claimed.

/image forces one tool for one round. It sends the ordinary message with force_tool, which becomes tool_choice — reusing the whole loop rather than inventing a second generation path. FORCEABLE_TOOLS is an allow list because this is read off a form, and resolve_tools still decides whether the tool exists, so forcing one that was never offered does nothing. payload.pop( "tool_choice") after the first round is load-bearing: left in place the reply would draw a picture, be asked again, and draw another.