Say what actually failed, and tell the model how to use the thing
Two problems, both found by looking rather than by guessing. ComfyUI writes its history entry in task_done and nowhere else, so the entry appearing IS "finished" -- but it sets completed=e.success, which means an out-of-memory, a cancelled job and a broken node all stay completed:false for ever. await_images waited on that flag. So every failure sat for the full 600s timeout and then reported a timeout, when ComfyUI had known within one second and written down the node, the exception type and the message. Proved by causing both against the real instance: an OOM now raises in 1.0s and an interrupt in 4.0s, each naming the node. The terminal condition is a record with a status, and status.messages is read for the last execution_error or execution_interrupted. OutOfMemory and Interrupted are their own classes because they are the two failures with an obvious next move: the first tells the model to retry at a named smaller size -- worked out from what it actually asked for, since "use a lower resolution" against a request that was already 512x512 is advice nobody can follow -- or with a lighter checkpoint; the second says somebody pressed stop, so do not simply start again. Everything else gets the reason and no advice, because a model told to try again after a broken workflow tries the identical thing. The OOM message is cut to its first sentence. The rest is allocator advice -- PYTORCH_CUDA_ALLOC_CONF, fragmentation notes -- addressed to whoever runs the box and meaningless to a model, in a tool result that is already a failure. Second: the parameters were described in the register of a reference table, and "cfg: prompt adherence, default 8" tells a model nothing it can act on. Measured on a 4B model, same request, same everything else: with the old wording it sent prompt and template and nothing more -- so 512x512 on an SDXL checkpoint, which is exactly the duplicated-limbs failure the width description now warns about. With descriptions that say what each value does to the picture and when to move it, the same model sent a portrait 1024x1536 and a deliberate sampler. ~3KB of schema per request in a chat that can draw, and the difference between having ten parameters and having one. docs/image-generation-instructions.md is the long version for the admin instructions box, for models that need more than the harness can afford to carry on every request in every chat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -393,3 +393,97 @@ async def test_an_unload_that_fails_does_not_stop_the_generation(
|
||||
|
||||
outcome = await image_tool.run(_context(db, user_id, configured), {"prompt": "x"})
|
||||
assert outcome.event["status"] == "ok"
|
||||
|
||||
|
||||
# --- What the model is told when it fails --------------------------------------
|
||||
async def test_running_out_of_memory_tells_the_model_what_to_do(
|
||||
db, user_id, configured, fake, monkeypatch
|
||||
):
|
||||
"""A bare "out of memory" gets the same request sent again, which fails the
|
||||
same way. The numbers are concrete because "use a lower resolution" against
|
||||
a request that was already 512x512 is advice nobody can follow."""
|
||||
|
||||
async def oom(config, wf):
|
||||
raise comfy.OutOfMemory("ComfyUI ran out of video memory in KSampler.")
|
||||
|
||||
monkeypatch.setattr(comfy, "submit", oom)
|
||||
outcome = await image_tool.run(
|
||||
_context(db, user_id, configured), {"prompt": "x", "width": 1024, "height": 1024}
|
||||
)
|
||||
|
||||
assert outcome.event["status"] == "error"
|
||||
assert "512x512" in outcome.content, "a size it can actually try"
|
||||
assert "1024x1024" in outcome.content, "and what it just asked for"
|
||||
assert "lighter checkpoint" in outcome.content
|
||||
assert "unchanged" in outcome.content
|
||||
|
||||
|
||||
async def test_a_cancelled_generation_is_not_retried_blindly(
|
||||
db, user_id, configured, fake, monkeypatch
|
||||
):
|
||||
"""Somebody pressed stop. Starting it again is arguing with them."""
|
||||
|
||||
async def stopped(config, wf):
|
||||
raise comfy.Interrupted("The image was cancelled on the ComfyUI side.")
|
||||
|
||||
monkeypatch.setattr(comfy, "submit", stopped)
|
||||
outcome = await image_tool.run(_context(db, user_id, configured), {"prompt": "x"})
|
||||
|
||||
assert "do not simply start it again" in outcome.content
|
||||
|
||||
|
||||
async def test_an_ordinary_failure_gets_no_invented_advice(
|
||||
db, user_id, configured, fake, monkeypatch
|
||||
):
|
||||
"""A model told to "try again" after a broken workflow tries the identical
|
||||
thing, and a suggestion invented for a failure nobody understands is a guess
|
||||
wearing the application's authority."""
|
||||
|
||||
async def broken(config, wf):
|
||||
raise comfy.ComfyError("ComfyUI could not finish the workflow in VAEDecode.")
|
||||
|
||||
monkeypatch.setattr(comfy, "submit", broken)
|
||||
outcome = await image_tool.run(_context(db, user_id, configured), {"prompt": "x"})
|
||||
|
||||
assert "VAEDecode" in outcome.content
|
||||
assert "smaller size" not in outcome.content
|
||||
assert "Try once more" not in outcome.content
|
||||
|
||||
|
||||
async def test_preserve_vram_frees_comfyui_even_when_it_failed(
|
||||
db, user_id, configured, fake, monkeypatch
|
||||
):
|
||||
"""It failed *inside* the far side, so its models are still resident and the
|
||||
language model is still unloaded. Without this the reply cannot even get far
|
||||
enough to say what happened."""
|
||||
settings_store.update(db, {"preserve_vram": True}, key=settings_store.IMAGES)
|
||||
db.commit()
|
||||
|
||||
async def oom(config, wf):
|
||||
raise comfy.OutOfMemory("out of video memory")
|
||||
|
||||
monkeypatch.setattr(comfy, "submit", oom)
|
||||
monkeypatch.setattr(image_tool, "_unload_llm", _noop)
|
||||
|
||||
await image_tool.run(_context(db, user_id, configured), {"prompt": "x"})
|
||||
assert fake.frees >= 1
|
||||
|
||||
|
||||
async def _noop(context):
|
||||
return True
|
||||
|
||||
|
||||
# --- Telling a model how to use the thing --------------------------------------
|
||||
def test_every_parameter_says_when_to_move_it(db, user_id, configured):
|
||||
""" "cfg: prompt adherence, default 8" tells a model nothing it can act on,
|
||||
and the observable result is a model that sends the prompt alone and leaves
|
||||
ten parameters at their defaults for ever."""
|
||||
schema = image_tool.schema_for(db, settings_store.images(db))
|
||||
for name in ("steps", "cfg", "width", "height", "sampler", "scheduler", "denoise", "negative"):
|
||||
description = schema["properties"][name]["description"]
|
||||
assert len(description) > 80, f"{name} is described too thinly to act on"
|
||||
|
||||
assert "never" in schema["properties"]["negative"]["description"].lower(), (
|
||||
"the negative prompt's one real trap: phrasing it as an instruction"
|
||||
)
|
||||
assert "portrait" in schema["properties"]["width"]["description"]
|
||||
|
||||
Reference in New Issue
Block a user