Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Say what actually failed, and tell the model how to use the thing

Browse Source 
Two problems, both found by looking rather than by guessing.

ComfyUI writes its history entry in task_done and nowhere else, so the entry
appearing IS "finished" -- but it sets completed=e.success, which means an
out-of-memory, a cancelled job and a broken node all stay completed:false for
ever. await_images waited on that flag. So every failure sat for the full 600s
timeout and then reported a timeout, when ComfyUI had known within one second and
written down the node, the exception type and the message. Proved by causing both
against the real instance: an OOM now raises in 1.0s and an interrupt in 4.0s,
each naming the node.

The terminal condition is a record with a status, and status.messages is read for
the last execution_error or execution_interrupted. OutOfMemory and Interrupted
are their own classes because they are the two failures with an obvious next
move: the first tells the model to retry at a named smaller size -- worked out
from what it actually asked for, since "use a lower resolution" against a request
that was already 512x512 is advice nobody can follow -- or with a lighter
checkpoint; the second says somebody pressed stop, so do not simply start again.
Everything else gets the reason and no advice, because a model told to try again
after a broken workflow tries the identical thing.

The OOM message is cut to its first sentence. The rest is allocator advice --
PYTORCH_CUDA_ALLOC_CONF, fragmentation notes -- addressed to whoever runs the box
and meaningless to a model, in a tool result that is already a failure.

Second: the parameters were described in the register of a reference table, and
"cfg: prompt adherence, default 8" tells a model nothing it can act on. Measured
on a 4B model, same request, same everything else: with the old wording it sent
prompt and template and nothing more -- so 512x512 on an SDXL checkpoint, which
is exactly the duplicated-limbs failure the width description now warns about.
With descriptions that say what each value does to the picture and when to move
it, the same model sent a portrait 1024x1536 and a deliberate sampler. ~3KB of
schema per request in a chat that can draw, and the difference between having ten
parameters and having one.

docs/image-generation-instructions.md is the long version for the admin
instructions box, for models that need more than the harness can afford to carry
on every request in every chat.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-05 14:55:18 +02:00
parent b2a05e0351
commit 178742501d
6 changed files with 447 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/images/tool.py
+114 -11
View File
@@ -61,9 +61,19 @@ SCHEMA: dict[str, Any] = {
"type": "string",
"description": "What to draw. Describe the subject, the setting and the style.",
},
# Every description below says what the value *does to the picture* and
# when to move it, not what it is called. A model that is told "cfg:
# prompt adherence, default 8" has been told nothing it can act on, and
# the observable result is a model that sends the prompt alone and
# leaves ten parameters at their defaults for ever.
"negative": {
"type": "string",
"description": "What to keep out of the picture. Defaults to 'text, watermark'.",
"description": (
"Comma-separated things to keep OUT of the picture, as plain nouns and "
"adjectives: 'blurry, extra fingers, text, watermark'. Not a sentence, "
"and never phrased as an instruction — 'do not add text' puts *text* in "
"the picture. Defaults to 'text, watermark'."
),
},
"template": {
"type": "string",
@@ -71,22 +81,80 @@ SCHEMA: dict[str, Any] = {
},
"model": {
"type": "string",
"description": "Which checkpoint to draw with. Omit to use this chat's usual one.",
"description": (
"Which checkpoint to draw with. Pick by what it is good at; omit to use "
"this chat's usual one."
),
},
"seed": {
"type": "integer",
"description": (
"Omit it, or pass -1, for a new random image. Repeat a seed you were "
"told about to get the same image again."
"told about to get that same image again — which is how you change one "
"thing about a picture and keep the rest."
),
},
"steps": {
"type": "integer",
"description": (
"How long to refine, 1-150. Default 20. Around 20-30 for most things; "
"8-12 for a quick draft or when several are wanted; 40+ only for fine "
"detail, and past about 50 it stops improving and only costs time."
),
},
"cfg": {
"type": "number",
"description": (
"How literally to follow the prompt, 0-30. Default 8. 3-6 gives the "
"model room and looks more natural; 7-9 is the usual range; 12+ forces "
"the words through and starts to look burnt and over-saturated. Lower "
"it if the picture looks harsh, raise it if the subject is being "
"ignored."
),
},
"width": {
"type": "integer",
"description": (
"Pixels, 64-2048, a multiple of 8. Default 512. Use the size the "
"checkpoint was trained for — about 512 for SD1.5, about 1024 for SDXL "
"— and change the ratio rather than the total: 512x768 for a portrait, "
"768x512 for a landscape. Going far above what the checkpoint expects "
"produces duplicated limbs and repeated horizons, not more detail."
),
},
"height": {
"type": "integer",
"description": (
"Pixels, 64-2048, a multiple of 8. Default 512. See width: the aspect "
"ratio is the thing to choose, and taller than wide suits a person, "
"wider than tall suits a place."
),
},
"sampler": {
"type": "string",
"description": (
"How the image is solved. Default euler. 'euler' is safe and fast; "
"'dpmpp_2m' is a good general improvement; 'dpmpp_2m_sde' for more "
"texture; 'ddim' for a clean flat look. Leave it out unless you have a "
"reason."
),
},
"scheduler": {
"type": "string",
"description": (
"How the steps are spaced. Default normal. 'karras' pairs well with the "
"dpmpp samplers and usually helps at low step counts; 'normal' "
"otherwise. Leave it out unless you are also setting the sampler."
),
},
"denoise": {
"type": "number",
"description": (
"How much of the starting noise to replace, 0-1. Default 1, which is "
"what you want for a picture drawn from nothing. Lower values only mean "
"something for a workflow that starts from an existing image."
),
},
"steps": {"type": "integer", "description": "Sampling steps. Default 20."},
"cfg": {"type": "number", "description": "Prompt adherence. Default 8."},
"width": {"type": "integer", "description": "Pixels. Default 512."},
"height": {"type": "integer", "description": "Pixels. Default 512."},
"sampler": {"type": "string", "description": "Sampler name. Default euler."},
"scheduler": {"type": "string", "description": "Scheduler name. Default normal."},
"denoise": {"type": "number", "description": "0 to 1. Default 1."},
},
"required": ["prompt"],
}
@@ -371,6 +439,9 @@ async def run(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
attempts: list[Attempt] = []
kept: tuple[bytes, dict[str, Any]] | None = None
# What the last attempt actually asked for, so a failure can name concrete
# numbers back at the model rather than saying "try something smaller".
params_used: dict[str, Any] = workflow.resolve(given)
try:
for number in range(1, tries + 1):
@@ -378,6 +449,7 @@ async def run(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
await _unload_llm(context)
params = workflow.resolve({**given, "seed": args.get("seed") if number == 1 else None})
params_used = params
refs = await comfy.await_images(
config, await comfy.submit(config, workflow.fill(template, params))
)
@@ -402,9 +474,12 @@ async def run(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
break
except comfy.ComfyError as exc:
if preserve:
# It failed *inside* the far side, so its models are still resident
# and the language model is still unloaded. Freeing here is what
# lets the reply carry on and say what happened.
await comfy.free(config)
return ToolOutcome(
f"The image could not be generated: {exc.message}",
f"The image could not be generated: {exc.message}{_advice(exc, params_used)}",
{**event, "status": "error", "error": exc.message},
)
@@ -454,6 +529,34 @@ async def run(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
)
def _advice(exc: comfy.ComfyError, params: dict[str, Any]) -> str:
"""What to do about a failure, when there is something to do about it.
Only for the two that have an obvious next move. Everything else gets the
reason and nothing else -- a model told to "try again" after a broken
workflow will try the identical thing, and a suggestion invented for a
failure nobody understands is a guess wearing the application's authority.
The numbers are concrete on purpose. "Use a lower resolution" against a
request that was already 512x512 is advice that cannot be followed, so the
halved size is worked out here where the request is known.
"""
if isinstance(exc, comfy.Interrupted):
return (
" Somebody stopped it deliberately, so do not simply start it again — say so and ask."
)
if not isinstance(exc, comfy.OutOfMemory):
return ""
width, height = int(params.get("width") or 512), int(params.get("height") or 512)
smaller = f"{max(256, width // 2)}x{max(256, height // 2)}"
return (
f" Try once more at a smaller size — {smaller} instead of {width}x{height} — "
"or with a lighter checkpoint if one is offered. Do not repeat the same "
"request unchanged; it will run out of memory again."
)
def _pick(rows: list[Any], wanted: str, chat_default: str, values: dict[str, Any]) -> Any:
"""The workflow to use: asked for, then the chat's, then the instance's."""
by_slug = {row.slug: row for row in rows}