Jaroslav Beneš
fa8c8ab5e8
`hx-get=""` is not "fetch nothing". htmx looks for the attribute, not for a value, so the empty one the canvas rendered before a chat existed was a real request for the empty path -- which the browser resolves against the current document. Opening the canvas on the new-chat screen fetched the new-chat screen and swapped the whole site into the panel. The attribute is omitted now, and a test refuses an empty verb anywhere on the page. Which panels can exist is the server's answer; which are offered is the browser's. Both need an agent chat on a chosen connection, and before a chat exists those are controls in the composer -- so answering with the first profile offered a terminal on an ordinary chat with nothing selected. They follow `lembas:agent-target` now, and an open panel whose target goes away is closed rather than left showing one machine under another's name. `.tabs__body` is only sometimes the scroller: true where the tabs are a bounded flex child, false under the admin layout, where the page scrolls instead. So setting its scrollTop on every tab change had never once run on /admin/prompts, silently, while the reader was dragged to the bottom of a document that had just got shorter. The rule names the position now, and the handler finds the container that actually scrolls. The two top borders come off. They were what made the misalignment at the bottom of the shell visible; `--footer-height` stays, because two ends at different heights are visible without a line to prove it. The top of the shell keeps its line -- there, everything is `--header-height` and aligns by construction. And one version. pyproject carried its own copy and had drifted three minors from the one everything actually reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
127 lines
8.0 KiB
Markdown
127 lines
8.0 KiB
Markdown
# Image generation
|
|
|
|
Split out of `CLAUDE.md` -- same document, same rules, kept here because that
|
|
file is loaded in full on every session and this part is only wanted when you
|
|
are working on drawing on a ComfyUI. Read it before you do.
|
|
|
|
Covers `services/images/` -- `comfy.py`, `workflow.py`, `tool.py` -- and
|
|
`api/admin_images.py`.
|
|
|
|
**Image generation is a ComfyUI workflow with holes in it, and the holes are the
|
|
administrator's statement.** `services/images/` is three modules: `comfy.py`
|
|
speaks HTTP, `workflow.py` fills a template, `tool.py` ties them to a chat.
|
|
Which node holds the prompt is *declared* with `{{prompt}}` rather than sniffed
|
|
by node type — looking for the first `CLIPTextEncode` works on the shipped
|
|
workflow and on nothing else, and swaps positive for negative the first time
|
|
somebody reorders them.
|
|
|
|
**Substitution walks the parsed JSON, not the text of it.** A value that is
|
|
*exactly* `"{{steps}}"` becomes the number 20; ComfyUI validates types and
|
|
refuses the string. A placeholder inside a longer string is still text, which is
|
|
what makes `"{{prompt}}, masterpiece"` work. Doing it textually would also mean
|
|
a prompt containing a quotation mark produced a document that no longer parses,
|
|
on the one input guaranteed to hold arbitrary text. `seed` has no fixed default
|
|
— one would make every unspecified generation identical and make the retry loop
|
|
redraw the same rejected picture four times. **A negative seed means random**,
|
|
because `-1` is what ComfyUI's own interface, A1111 and everything else that has
|
|
ever asked for a seed use for it, so a model that has read any of them writes
|
|
it: without that it went through the uint64 wrap and arrived as
|
|
18446744073709551615, a perfectly valid *fixed* seed, so "give me something new"
|
|
returned the same picture every time.
|
|
|
|
**One call is one finished image, and the retrying is inside the tool.**
|
|
Returning every attempt to the conversation would cost a round each, make the
|
|
ceiling advisory rather than enforced, and walk the reader past every reject. So
|
|
the reviewer — the admin's chosen vision model, else the chat's own if it has
|
|
vision, else nobody — is asked about *bytes* rather than about a row: an attempt
|
|
about to be discarded should not leave an `Attachment` behind, so it sees a
|
|
downscaled preview built in memory and only the kept image is written. Anything
|
|
that goes wrong in review is a **keep**; losing a picture because a judging
|
|
request timed out would be the check destroying the thing it was checking. The
|
|
last attempt is kept whatever the verdict, so a request always produces
|
|
something. Rejected images are not stored — their verdicts are, in `event.text`.
|
|
|
|
**`task.image_review` is a `GROUP_TASKS` fragment**, so it is editable and
|
|
excluded from the harness, exactly like `task.title` and `task.compact` — and
|
|
clearing it switches reviewing off, the same way clearing `task.compact` switches
|
|
compaction off. It is biased hard towards KEEP on purpose: a reviewer that
|
|
retries on taste spends the GPU four times and usually ends up back at the first
|
|
image.
|
|
|
|
**A failed generation is `completed: false` for ever, so waiting on that flag
|
|
hangs the reply.** ComfyUI writes its history entry in `task_done` and nowhere
|
|
else, so the entry appearing *is* "finished" — but it sets `completed=e.success`,
|
|
which means an out-of-memory, a cancelled job and a broken node all stay
|
|
incomplete permanently. The first version waited on the flag, so every failure
|
|
sat for the full 600s timeout and then reported a timeout, when ComfyUI had known
|
|
within one second and written down exactly what happened. The terminal condition
|
|
is now *a record with a status*, and `status.messages` is read for the last
|
|
`execution_error` or `execution_interrupted` in it, which carries the node and
|
|
the exception.
|
|
|
|
Two failures get their own class because they have an obvious next move.
|
|
`OutOfMemory` — matched on `exception_type`, not on the message, which is a
|
|
paragraph of allocator advice addressed to whoever runs the box — makes the tool
|
|
tell the model to retry at a named smaller size (worked out from what it actually
|
|
asked for, because "use a lower resolution" against a request that was already
|
|
512x512 is advice nobody can follow) or with a lighter checkpoint. `Interrupted`
|
|
is not a fault at all: somebody pressed stop, and the model is told not to simply
|
|
start it again. **Everything else gets the reason and no advice** — a model told
|
|
to "try again" after a broken workflow tries the identical thing, and a
|
|
suggestion invented for a failure nobody understands is a guess wearing the
|
|
application's authority.
|
|
|
|
**A tool's parameter descriptions are instructions, and terse ones are why a
|
|
model sends only the prompt.** "cfg: prompt adherence, default 8" tells a model
|
|
nothing it can act on. Measured against a 4B model on the same request: with the
|
|
terse descriptions it sent `prompt` and `template` and nothing else — meaning
|
|
512x512 defaults on an SDXL checkpoint, which is precisely the duplicated-limbs
|
|
failure the width description now warns about. With descriptions that say what
|
|
each value *does to the picture* and when to move it, the same model sent a
|
|
portrait 1024x1536 and a deliberate sampler. It costs ~3KB of schema per request
|
|
in a chat that can draw, and it is the difference between having ten parameters
|
|
and having one. `docs/image-generation-instructions.md` is the long version, to
|
|
paste into the admin instructions box for models that need more than the harness
|
|
can afford to carry.
|
|
|
|
**Preserve VRAM unloads the chat's own connection and nothing else.**
|
|
`Connection.unload_url` is a column because the memory being freed belongs to one
|
|
machine: a local llama-swap answers `GET /unload`, and a box on the network has
|
|
no reason to be unloaded when ComfyUI wants memory *here*. Empty means "cannot be
|
|
unloaded", which is the honest default — there is no call that works everywhere.
|
|
The swap goes round the *review*, not round the tool: unload, generate, free
|
|
ComfyUI, ask the reviewer (which loads the LLM again), round again if it said no.
|
|
Two model loads per retry, which is why the two settings are independent and the
|
|
page says so when both are on. **Nothing loads the LLM back at the end** — the
|
|
reply's next request does, and llama-swap loads on demand; that step exists in
|
|
the description and not in the code, which is why the code says so.
|
|
|
|
**A generated image rides on the assistant message, so `message_payload` sends
|
|
images only on `user` turns.** No assistant message had ever carried one before,
|
|
so the distinction had never been drawn — and the moment one does, the
|
|
multimodal list form on an `assistant` turn is rejected by OpenAI and most local
|
|
runners, breaking not that turn but every later one in the chat. What follows and
|
|
is worth knowing: on a *later* turn the model cannot see the picture it made
|
|
(tool results are not replayed either), so "make it bluer" regenerates rather
|
|
than edits. Honest for a text-to-image workflow with no img2img path.
|
|
|
|
**The runner writes the file; only the loop says which turn owns it.**
|
|
`event["attachment_id"]` is carried by `generation._run` exactly as
|
|
`event["canvas"]` and `event["plan"]` are, because `_persist` is the single
|
|
writer. `_bind_attachments` narrows on this chat and on rows still unbound, for
|
|
the reason `files.claim` does: the ids arrive on a dict a runner built.
|
|
|
|
**`files.store(keep_original=True)` skips the resize and the transcode, and
|
|
nothing else.** `_process_image` turns anything without alpha into JPEG q85 at
|
|
1400px, which is right for a phone photo and a visible loss on generated art.
|
|
Pillow still opens it, so a malformed file is still refused and the dimensions
|
|
are still measured rather than claimed.
|
|
|
|
**`/image` forces one tool for one round.** It sends the ordinary message with
|
|
`force_tool`, which becomes `tool_choice` — reusing the whole loop rather than
|
|
inventing a second generation path. `FORCEABLE_TOOLS` is an allow list because
|
|
this is read off a form, and `resolve_tools` still decides whether the tool
|
|
exists, so forcing one that was never offered does nothing. `payload.pop(
|
|
"tool_choice")` after the first round is load-bearing: left in place the reply
|
|
would draw a picture, be asked again, and draw another.
|