Say what actually failed, and tell the model how to use the thing

Two problems, both found by looking rather than by guessing.

ComfyUI writes its history entry in task_done and nowhere else, so the entry
appearing IS "finished" -- but it sets completed=e.success, which means an
out-of-memory, a cancelled job and a broken node all stay completed:false for
ever. await_images waited on that flag. So every failure sat for the full 600s
timeout and then reported a timeout, when ComfyUI had known within one second and
written down the node, the exception type and the message. Proved by causing both
against the real instance: an OOM now raises in 1.0s and an interrupt in 4.0s,
each naming the node.

The terminal condition is a record with a status, and status.messages is read for
the last execution_error or execution_interrupted. OutOfMemory and Interrupted
are their own classes because they are the two failures with an obvious next
move: the first tells the model to retry at a named smaller size -- worked out
from what it actually asked for, since "use a lower resolution" against a request
that was already 512x512 is advice nobody can follow -- or with a lighter
checkpoint; the second says somebody pressed stop, so do not simply start again.
Everything else gets the reason and no advice, because a model told to try again
after a broken workflow tries the identical thing.

The OOM message is cut to its first sentence. The rest is allocator advice --
PYTORCH_CUDA_ALLOC_CONF, fragmentation notes -- addressed to whoever runs the box
and meaningless to a model, in a tool result that is already a failure.

Second: the parameters were described in the register of a reference table, and
"cfg: prompt adherence, default 8" tells a model nothing it can act on. Measured
on a 4B model, same request, same everything else: with the old wording it sent
prompt and template and nothing more -- so 512x512 on an SDXL checkpoint, which
is exactly the duplicated-limbs failure the width description now warns about.
With descriptions that say what each value does to the picture and when to move
it, the same model sent a portrait 1024x1536 and a deliberate sampler. ~3KB of
schema per request in a chat that can draw, and the difference between having ten
parameters and having one.

docs/image-generation-instructions.md is the long version for the admin
instructions box, for models that need more than the harness can afford to carry
on every request in every chat.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-05 14:55:18 +02:00
parent ca7eb6cedb
commit 60e7d0d599
9 changed files with 632 additions and 40 deletions
+140
View File
@@ -0,0 +1,140 @@
# Extra instructions for image generation
Paste the block below into **Admin Image generation Extra instructions**.
It reaches every model on the instance, above whatever each chat's own system
prompt says, and it appears only when the image tool is actually offered.
It is longer than the built-in guidance on purpose. The built-in fragment has to
suit every instance and is kept short because it costs tokens on every request
in every chat that can draw; this is yours to make as long as your models need.
**Small models need more of it.** A 4B model left to itself passes the request
through verbatim — "draw me a cat" becomes the prompt "draw me a cat" — and
leaves ten parameters at their defaults for ever. Most of what follows exists to
stop that.
Trim it if your models are large enough not to need it: every line of it is sent
on every request in every chat where image generation is on.
Two things it deliberately does **not** cover, because LLeMbas already tells the
model and repeating them wastes the window:
- the parameter ranges and defaults — those are in the tool's own schema
- that the picture is already on screen — that is in the built-in fragment
---
```text
WRITING THE PROMPT
Never send the request as the prompt. "a cat" is a request; the prompt is what
you write from it. Expand it into a description, in this order:
subject, what it is doing, setting, lighting, composition, style and medium
Comma-separated phrases, not a sentence. Concrete nouns and adjectives. Twenty
to sixty words is the useful range: below that the model invents everything you
left out, and much above it the later words stop having any effect.
weak: a cat
better: a ginger tabby cat asleep on a windowsill, curled up, potted herbs
beside it, low afternoon sun through old glass, warm rim light,
shallow depth of field, 50mm photograph
Say the medium explicitly — photograph, oil painting, pencil sketch, 3D render,
watercolour, screen print. Without it you get an averaged, plasticky look that
belongs to no medium at all.
For a photograph, naming a lens and light does most of the work: 35mm, 85mm
portrait, golden hour, overcast, backlit, studio softbox.
For an illustration, name the tradition rather than a living artist: art
nouveau, ukiyo-e, mid-century children's book, technical cutaway diagram.
Do not write instructions in the prompt. "make sure there are exactly two
people" is not understood. Describe the result: "two people".
NEGATIVE PROMPTS
Plain nouns and adjectives for things that must not appear:
"blurry, low quality, extra fingers, deformed hands, text, watermark, signature".
Never phrase it as an instruction. "no text" contains the word text and puts
text in the picture. The negative prompt is a list of things to avoid, not a
sentence to obey.
Add "extra fingers, deformed hands" whenever hands are visible, and
"extra limbs, fused bodies" for more than one person.
SIZE
Choose the aspect ratio for the subject, then keep the total near what the
checkpoint expects.
portrait of a person 512x768 (or 832x1216 on an SDXL checkpoint)
landscape or interior 768x512 (or 1216x832)
square, product, icon 512x512 (or 1024x1024)
Going far above what a checkpoint was trained for does not add detail: it adds
second heads, extra limbs and repeated horizons. If you want more detail, add
detail to the prompt.
CHOOSING A CHECKPOINT AND A TEMPLATE
Read the descriptions you were given and pick by what the picture needs. When
nothing obviously fits, leave both out — the chat's usual ones are used, and a
wrong guess costs a whole generation.
WHEN TO CHANGE THE OTHER PARAMETERS
drafting, or making several to compare steps 10-12
the result looks harsh or over-saturated cfg 4-6
the subject is being ignored cfg 9-11, and simplify the prompt
fine texture matters steps 35-45, sampler dpmpp_2m,
scheduler karras
Otherwise leave them alone. Changing three at once teaches you nothing about
which one helped.
CHANGING A PICTURE YOU HAVE ALREADY MADE
You are told the seed of every image you generate. To change one thing and keep
the rest, send the same seed with an edited prompt. To get something completely
different, omit the seed or send -1.
Note that you cannot see a picture again on a later turn, so decide what to
change from what you wrote, not from what you remember seeing.
WHEN IT FAILS
Out of video memory: generate again at about half the width and height, or with
a lighter checkpoint. Do not resend the same request — it will fail the same
way.
Cancelled: somebody stopped it deliberately. Say so and ask before starting
another.
Anything else: say what failed and what you were trying to draw. Do not retry
the identical request more than once.
AFTERWARDS
The picture is already in the conversation. Say in one or two lines what you
made and what you would change — the checkpoint, the size and the seed are
shown, so do not repeat them.
```
---
## A shorter version
For a large model, or an instance where the window is tight:
```text
Write the prompt as a description, never as the request you were given:
subject, action, setting, lighting, style and medium, comma-separated,
twenty to sixty words. Always name the medium. Use the negative prompt for
things to avoid, as plain nouns ("blurry, extra fingers, text") and never as
an instruction. Choose the aspect ratio for the subject — taller for a
person, wider for a place — and keep the total near what the checkpoint
expects. Change the other parameters only for a reason. If it runs out of
video memory, retry once at half the size or with a lighter checkpoint.
```