LLeMbas/docs/notes/image-generation.md

# Image generation

Split out of `CLAUDE.md` -- same document, same rules, kept here because that
file is loaded in full on every session and this part is only wanted when you
are working on drawing on a ComfyUI. Read it before you do.

Covers `services/images/` -- `comfy.py`, `workflow.py`, `tool.py` -- and
`api/admin_images.py`.

**Image generation is a ComfyUI workflow with holes in it, and the holes are the
administrator's statement.** `services/images/` is three modules: `comfy.py`
speaks HTTP, `workflow.py` fills a template, `tool.py` ties them to a chat.
Which node holds the prompt is *declared* with `{{prompt}}` rather than sniffed
by node type — looking for the first `CLIPTextEncode` works on the shipped
workflow and on nothing else, and swaps positive for negative the first time
somebody reorders them.

**Substitution walks the parsed JSON, not the text of it.** A value that is
*exactly* `"{{steps}}"` becomes the number 20; ComfyUI validates types and
refuses the string. A placeholder inside a longer string is still text, which is
what makes `"{{prompt}}, masterpiece"` work. Doing it textually would also mean
a prompt containing a quotation mark produced a document that no longer parses,
on the one input guaranteed to hold arbitrary text. `seed` has no fixed default
— one would make every unspecified generation identical and make the retry loop
redraw the same rejected picture four times. **A negative seed means random**,
because `-1` is what ComfyUI's own interface, A1111 and everything else that has
ever asked for a seed use for it, so a model that has read any of them writes
it: without that it went through the uint64 wrap and arrived as
18446744073709551615, a perfectly valid *fixed* seed, so "give me something new"
returned the same picture every time.

**One call is one finished image, and the retrying is inside the tool.**
Returning every attempt to the conversation would cost a round each, make the
ceiling advisory rather than enforced, and walk the reader past every reject. So
the reviewer — the admin's chosen vision model, else the chat's own if it has
vision, else nobody — is asked about *bytes* rather than about a row: an attempt
about to be discarded should not leave an `Attachment` behind, so it sees a
downscaled preview built in memory and only the kept image is written. Anything
that goes wrong in review is a **keep**; losing a picture because a judging
request timed out would be the check destroying the thing it was checking. The
last attempt is kept whatever the verdict, so a request always produces
something. Rejected images are not stored — their verdicts are, in `event.text`.

**`task.image_review` is a `GROUP_TASKS` fragment**, so it is editable and
excluded from the harness, exactly like `task.title` and `task.compact` — and
clearing it switches reviewing off, the same way clearing `task.compact` switches
compaction off. It is biased hard towards KEEP on purpose: a reviewer that
retries on taste spends the GPU four times and usually ends up back at the first
image.

**A failed generation is `completed: false` for ever, so waiting on that flag
hangs the reply.** ComfyUI writes its history entry in `task_done` and nowhere
else, so the entry appearing *is* "finished" — but it sets `completed=e.success`,
which means an out-of-memory, a cancelled job and a broken node all stay
incomplete permanently. The first version waited on the flag, so every failure
sat for the full 600s timeout and then reported a timeout, when ComfyUI had known
within one second and written down exactly what happened. The terminal condition
is now *a record with a status*, and `status.messages` is read for the last
`execution_error` or `execution_interrupted` in it, which carries the node and
the exception.

Two failures get their own class because they have an obvious next move.
`OutOfMemory` — matched on `exception_type`, not on the message, which is a
paragraph of allocator advice addressed to whoever runs the box — makes the tool
tell the model to retry at a named smaller size (worked out from what it actually
asked for, because "use a lower resolution" against a request that was already
512x512 is advice nobody can follow) or with a lighter checkpoint. `Interrupted`
is not a fault at all: somebody pressed stop, and the model is told not to simply
start it again. **Everything else gets the reason and no advice** — a model told
to "try again" after a broken workflow tries the identical thing, and a
suggestion invented for a failure nobody understands is a guess wearing the
application's authority.

**A tool's parameter descriptions are instructions, and terse ones are why a
model sends only the prompt.** "cfg: prompt adherence, default 8" tells a model
nothing it can act on. Measured against a 4B model on the same request: with the
terse descriptions it sent `prompt` and `template` and nothing else — meaning
512x512 defaults on an SDXL checkpoint, which is precisely the duplicated-limbs
failure the width description now warns about. With descriptions that say what
each value *does to the picture* and when to move it, the same model sent a
portrait 1024x1536 and a deliberate sampler. It costs ~3KB of schema per request
in a chat that can draw, and it is the difference between having ten parameters
and having one. `docs/image-generation-instructions.md` is the long version, to
paste into the admin instructions box for models that need more than the harness
can afford to carry.

**Preserve VRAM unloads the chat's own connection and nothing else.**
`Connection.unload_url` is a column because the memory being freed belongs to one
machine: a local llama-swap answers `GET /unload`, and a box on the network has
no reason to be unloaded when ComfyUI wants memory *here*. Empty means "cannot be
unloaded", which is the honest default — there is no call that works everywhere.
The swap goes round the *review*, not round the tool: unload, generate, free
ComfyUI, ask the reviewer (which loads the LLM again), round again if it said no.
Two model loads per retry, which is why the two settings are independent and the
page says so when both are on. **Nothing loads the LLM back at the end** — the
reply's next request does, and llama-swap loads on demand; that step exists in
the description and not in the code, which is why the code says so.

**A generated image rides on the assistant message, so `message_payload` sends
images only on `user` turns.** No assistant message had ever carried one before,
so the distinction had never been drawn — and the moment one does, the
multimodal list form on an `assistant` turn is rejected by OpenAI and most local
runners, breaking not that turn but every later one in the chat. What follows and
is worth knowing: on a *later* turn the model cannot see the picture it made
(tool results are not replayed either), so "make it bluer" regenerates rather
than edits. Honest for a text-to-image workflow with no img2img path.

**The runner writes the file; only the loop says which turn owns it.**
`event["attachment_id"]` is carried by `generation._run` exactly as
`event["canvas"]` and `event["plan"]` are, because `_persist` is the single
writer. `_bind_attachments` narrows on this chat and on rows still unbound, for
the reason `files.claim` does: the ids arrive on a dict a runner built.

**`files.store(keep_original=True)` skips the resize and the transcode, and
nothing else.** `_process_image` turns anything without alpha into JPEG q85 at
1400px, which is right for a phone photo and a visible loss on generated art.
Pillow still opens it, so a malformed file is still refused and the dimensions
are still measured rather than claimed.

**`/image` forces one tool for one round.** It sends the ordinary message with
`force_tool`, which becomes `tool_choice` — reusing the whole loop rather than
inventing a second generation path. `FORCEABLE_TOOLS` is an allow list because
this is read off a form, and `resolve_tools` still decides whether the tool
exists, so forcing one that was never offered does nothing. `payload.pop(
"tool_choice")` after the first round is load-bearing: left in place the reply
would draw a picture, be asked again, and draw another.

## The defaults an administrator can set

**There were none, for the whole life of the feature.** `workflow.DEFAULTS` was
the only source, so 512×512, `euler` and twenty steps were what every instance
got whatever card it was running on — and 512² on an SDXL checkpoint is exactly
what the tool's own `width` description warns produces duplicated limbs. The two
ways round it were both bad: bake literals into a template where the
placeholders should be, or write prose in the instructions box and hope the
model obeys it.

`resolve(given, settings=…)` is three rungs now, most specific winning:
**`DEFAULTS` → the instance's `default_*` settings → what the model asked for.**
`DEFAULTS` stays underneath as the floor, so an instance that sets nothing
behaves exactly as it did, and improving a floor in code still reaches everyone.

**An empty setting is "no opinion", not zero.** `_number` in `admin_images`
returns `""` for an empty box and `instance_defaults` skips it. Reading it as a
number instead would set every instance to zero steps, which ComfyUI refuses in
a way that looks like a broken model.

**The samplers and schedulers were already being discovered and read by
nothing.** `comfy.discover()` has fetched all three lists since the Test button
existed, and only `checkpoints` was ever used. The pickers are built from the
other two. A stored value that is not in the list is kept as an option anyway,
or opening the page and pressing Save would silently clear a working setting.

**`batch` is a placeholder a model cannot set.** `batch_size` was a literal `1`
in the base template, so an administrator whose card can make four at a time had
no way of saying so. It is absent from `MODEL_SETTABLE`, deliberately: a model
asking for six because it is unsure is the exact cost this must not invite.

**The schema restates the defaults it quotes.** Every "Default 20." in
`SCHEMA` was written when there was one set of defaults in the world.
`_restate_defaults` rewrites each one from what this instance actually resolves
to — a schema saying "Default 512" beside an instance that draws at 1024 is
worse than saying nothing, because the model reasons from it and omits the
parameter, arriving at the right behaviour for the wrong reason or the wrong one
silently. The regex keeps the punctuation it found, since `denoise` says
"Default 1, which is…" and the rest use a full stop.

**The workflow editor's legend shows the resolved value beside each
placeholder.** A list of names answers "what may I write"; the question somebody
has in front of a workflow that came out wrong is "what happens if I leave this
out", and that answer moved the day instance defaults arrived. It is resolved
through the same call a generation makes, so the two cannot disagree. The legend
also states the two names that are not ComfyUI's own — `{{model}}` fills
`ckpt_name` and `{{sampler}}` fills `sampler_name` — which is the mistake that
costs an afternoon.