Draw a picture, on a ComfyUI you are running

The last unbuilt capability, and built the way CLAUDE.md said it had to be: a
ToolDef reaching resolve_tools plus a permission and a capability flag, not a new
code path. The only genuinely new UI is one branch in the transcript.

services/images/ is three modules. comfy.py speaks HTTP -- submit, poll /history,
fetch the PNG, /free, and an /object_info discovery for the admin page only.
Polled and not socketed, because holding a connection open for the length of a
generation is the live-connection state the whole ssh.py design forbids, and the
thing being waited for takes tens of seconds anyway. The base URL is exempt from
the SSRF guard by construction, exactly as Connection.base_url and the audio
endpoints are -- said out loud in the docstring, because a default of
127.0.0.1:8188 is precisely the shape that guard exists to refuse and therefore
reads as a hole rather than a decision.

workflow.py fills a template, and the one thing that matters is that it walks the
parsed JSON rather than the text of it. A value that is exactly "{{steps}}"
becomes the number 20; ComfyUI validates types and refuses the string. A
placeholder inside a longer string is still text, which is what makes
"{{prompt}}, masterpiece" work -- and text substitution would additionally mean a
prompt containing a quotation mark produced a document that no longer parses, on
the one input guaranteed to hold arbitrary text. Which node holds the prompt is
the administrator's statement rather than a guess from node types: sniffing for
the first CLIPTextEncode works on the shipped workflow and on nothing else, and
swaps positive for negative the first time somebody reorders them. seed has no
fixed default, because one would make every unspecified generation identical and
make the retry loop redraw the same rejected picture four times.

tool.py is one call, one finished image. Returning every attempt to the
conversation would cost a round each, make the ceiling advisory rather than
enforced, and walk the reader past every reject -- so the reviewer lives inside
the tool and is asked about *bytes*: an attempt about to be discarded should not
leave an Attachment behind, so it sees a downscaled preview built in memory and
only the kept image is written. Anything that goes wrong in review is a keep;
losing a picture because a judging request timed out would be the check
destroying the thing it was checking. The last attempt is kept whatever the
verdict, so a request always produces something. Rejects are recorded, not
stored.

Preserve VRAM unloads the chat's own connection and nothing else, because the
memory being freed belongs to one machine: local llama-swap answers GET /unload,
and a box on the network has no reason to be unloaded when ComfyUI wants memory
here. The swap goes round the review rather than round the tool, which costs two
model loads per retry -- so the two settings are independent and the page warns
when both are on. Nothing loads the LLM back: the reply's next request does, and
that step exists in the description and not in the code, so the code says so.

Two rules elsewhere had to be drawn for the first time. message_payload sends
images only on user turns -- no assistant message had ever carried one, and the
moment one does the multimodal list form on an assistant turn is rejected by
OpenAI and most local runners, breaking every later turn in the chat. And
files.store gained keep_original, because _process_image turns anything without
alpha into JPEG q85 at 1400px: right for a phone photo, a visible loss on the one
output this feature exists to produce.

/image sends the ordinary message with force_tool, which becomes tool_choice for
the first round only -- left in place the reply would draw a picture, be asked
again, and draw another. FORCEABLE_TOOLS is an allow list because the name is
read off a form.

ToolContext gained chat_id, and that fixed a tool nobody had ever successfully
run: _run_scratch_write read context.chat_id on a dataclass with no such field,
so every call raised AttributeError, swallowed by run_tool's blanket except into
"the scratch_write tool failed" -- indistinguishable from a model calling it
wrongly. The test that existed asserted the family and the risk, which are
properties of the declaration rather than of the code.

Verified against the real ComfyUI 0.27.0 on this machine rather than against
documentation: every endpoint shape here was read off it, a generation ran end to
end through the client, the reviewer was shown a matching and a mismatched prompt
and answered KEEP and RETRY correctly, and the unload hook fired for the local
llama-swap and not for the remote box.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-05 14:13:19 +02:00
co-authored by Claude Opus 5
parent 9f5ff72e32
commit 47d1ddbc3c
38 changed files with 3958 additions and 18 deletions
+18
View File
@@ -452,6 +452,24 @@
overflow-y: auto;
}
/* What a generation drew, inside its own tool block.
Bounded by height rather than by width: a portrait at 1024x1536 sized to the
column would push everything under it off the screen, and this is the block's
supporting evidence rather than the thing itself -- the picture proper is an
attachment on the bubble above. `width: auto` with a max height is what keeps
a landscape and a portrait both looking deliberate. */
.tool-image { display: block; margin: 0; }
.tool-image img {
display: block;
max-width: 100%;
max-height: 20rem;
width: auto;
height: auto;
border-radius: var(--radius-sm);
background: var(--bg-sunken);
}
/* A diff, from a write or an update.
The same visual language as .tool-result__text above -- both answer "what did
+42
View File
@@ -168,6 +168,48 @@
else window.lembas.toggleTheme();
}
},
{
name: "image",
summary: "Draw a picture",
argument: "what to draw",
/* Offered wherever there is a chat, not only where image generation is
switched on. `available()` filters `run()` as well as the menu, so a
command hidden here stops being a command and gets *sent as a message*
-- and "/image a red bicycle" arriving as prose is worse than being
told the feature is off. The server answers either way. */
when: function () { return !!chat(); },
run: function (rest) {
var wanted = (rest || "").trim();
if (!wanted) return note("Say what to draw: /image a red bicycle in the rain.");
var thread = el("#thread");
if (!thread) return;
var body = new FormData();
body.append("content", wanted);
/* The whole of what this command is. The turn goes through the ordinary
path -- same route, same bubbles, same stream -- and carries one extra
field that makes the first round call the image tool instead of
deciding whether to. The words still say what is wanted, so an
endpoint that ignores tool_choice steers on those alone. */
body.append("force_tool", "image_generate");
fetch("/api/chats/" + chat() + "/messages", {
method: "POST",
body: body,
credentials: "same-origin"
})
.then(function (r) { return r.text(); })
.then(function (html) {
thread.insertAdjacentHTML("beforeend", html);
/* Without this the assistant bubble's sse-connect is inert markup
and the reply never starts -- the same reason /compact processes
the thread it swapped in. */
if (window.htmx) window.htmx.process(thread);
if (window.lembas.scrollThread) window.lembas.scrollThread(true);
})
.catch(function () { note("Could not start the image.", "error"); });
}
},
{ name: "new", summary: "Start a new chat", run: function () { window.location = "/chat"; } },
{
name: "temp",
@@ -72,6 +72,25 @@
</p>
</div>
<div class="field">
<label class="field__label" for="unload-{{ connection.id }}">Unload URL</label>
<div class="btn-row">
<input class="input input--mono" id="unload-{{ connection.id }}" name="unload_url"
value="{{ connection.unload_url }}" placeholder="No unload call"
style="flex: 1; min-width: 0">
<select class="select" name="unload_method" style="flex: none">
<option value="POST" {{ 'selected' if connection.unload_method != 'GET' }}>POST</option>
<option value="GET" {{ 'selected' if connection.unload_method == 'GET' }}>GET</option>
</select>
</div>
<p class="field__hint">
Only used by image generation's Preserve VRAM, to free this endpoint's
memory while ComfyUI works. llama-swap answers <code>GET /unload</code>.
Leave it empty for anything on another machine — its memory is not the
memory being freed, and it would be an unexplained request.
</p>
</div>
<div class="field">
<label class="checkbox">
<input type="checkbox" name="enabled" value="true"
@@ -0,0 +1,24 @@
{% from "_macros.html" import icon %}
{#
What ComfyUI said when it was asked what it has.
Everything here came off that machine and is escaped accordingly -- a
checkpoint filename is a filename somebody chose, not a value this
application generated.
#}
<div class="alert {{ 'alert--error' if message_kind == 'error' else 'alert--success' }}"
style="margin-top: var(--sp-3)">
{{ icon("warning" if message_kind == "error" else "check", "icon--sm") }}
<span>{{ message }}</span>
</div>
{% if checkpoints %}
<p class="field__hint">
{% if kept %}
Your list above was left alone. What ComfyUI has:
{% else %}
Filled in above:
{% endif %}
</p>
<pre class="tool-result__text">{{ checkpoints | join("\n") }}</pre>
{% endif %}
@@ -43,6 +43,10 @@
{{ icon("globe", "icon--sm") }}
<span class="nav-item__label">Web search</span>
</a>
<a class="nav-item {{ 'is-active' if section == 'images' }}" href="/admin/images">
{{ icon("image", "icon--sm") }}
<span class="nav-item__label">Image generation</span>
</a>
<a class="nav-item {{ 'is-active' if section == 'tools' }}" href="/admin/tools">
{{ icon("link", "icon--sm") }}
<span class="nav-item__label">Tools</span>
+274
View File
@@ -0,0 +1,274 @@
{% extends "admin/_layout.html" %}
{% from "_macros.html" import icon %}
{% set section = "images" %}
{% block title %}Image generation - LLeMbas{% endblock %}
{% block heading %}Image generation{% endblock %}
{% block admin_content %}
<p class="admin-lede">
Lets a model draw a picture and show it in the conversation, on a ComfyUI you
are running. It is offered as a tool the model chooses to call, so nothing
changes for a conversation that never asks for one — and it is only offered
once there is a ComfyUI, a workflow and at least one checkpoint, because a
tool that fails on its first call is worse than a tool nobody was given.
</p>
{% if saved %}
<div class="alert alert--success">{{ icon("check", "icon--sm") }} <span>{{ saved }}</span></div>
{% endif %}
<form method="post" action="/admin/images">
<section class="card">
<h2 class="card__title">
Image generation
{% if values.enabled %}<span class="badge badge--success">on</span>
{% else %}<span class="badge">off</span>{% endif %}
</h2>
<div class="field">
<label class="checkbox">
<input type="checkbox" name="enabled" value="true" {{ 'checked' if values.enabled }}>
<span>Offer image generation to models that support tools</span>
</label>
<p class="field__hint">
Who may use it is a permission — <code>tools.image</code> under
Groups &amp; permissions. Which models may is a checkbox on each model.
</p>
</div>
</section>
<section class="card">
<h2 class="card__title">ComfyUI</h2>
<div class="field">
<label class="field__label" for="base_url">Base URL</label>
<input class="input input--mono" id="base_url" name="base_url" type="url"
placeholder="http://127.0.0.1:8188" value="{{ values.base_url }}">
<p class="field__hint">
Where ComfyUI is listening. An address on this machine or this network is
fine here and is not checked against the request-forgery rules — you
typed it, unlike an address a model asks for.
</p>
</div>
<div class="field">
<label class="field__label" for="api_key">API key</label>
<input class="input input--mono" id="api_key" name="api_key" type="password"
autocomplete="off" placeholder="No key set"
value="{{ unchanged if values.api_key_encrypted else '' }}">
<p class="field__hint">
{% if values.api_key_encrypted %}
Currently <code>{{ masked }}</code>. Leave the dots alone to keep it, or
clear the field to remove it.
{% else %}
Only needed if something sits in front of ComfyUI. Encrypted at rest.
{% endif %}
</p>
</div>
<div class="field">
<label class="field__label" for="timeout">Timeout (seconds)</label>
<input class="input" id="timeout" name="timeout" type="number"
min="10" max="3600" step="10" value="{{ values.timeout | int }}">
<p class="field__hint">
How long to wait for one image, queue included. Far longer than any other
timeout here, because the thing being waited for genuinely takes that long.
</p>
</div>
</section>
<section class="card">
<h2 class="card__title">Checkpoints</h2>
<div class="field">
<label class="field__label" for="checkpoints">Available checkpoints</label>
<textarea class="textarea input--mono" id="checkpoints" name="checkpoints" rows="4"
spellcheck="false"
placeholder="sd_xl_base_1.0.safetensors">{{ checkpoints_text }}</textarea>
<p class="field__hint">
One filename per line, exactly as ComfyUI spells it. This is the list the
model chooses from, so leaving out a checkpoint is how you stop it being
used. Press Test below to read them off ComfyUI — that fills this in the
first time and never overwrites it afterwards.
</p>
</div>
<div class="btn-row">
<button class="btn" type="button"
hx-post="/admin/images/test" hx-target="#images-test-result">
{{ icon("refresh", "icon--sm") }} Test &amp; read what it has
</button>
</div>
<div id="images-test-result"></div>
</section>
<section class="card">
<h2 class="card__title">Checking the result</h2>
<div class="field">
<label class="checkbox">
<input type="checkbox" name="review_enabled" value="true"
{{ 'checked' if values.review_enabled }}>
<span>Look at each image before showing it, and try again if it is wrong</span>
</label>
<p class="field__hint">
A vision model is shown the picture and the request it came from, and says
keep or retry. Only clearly wrong images are retried — a missing subject, a
mangled picture — because taste is not a fault and the next attempt is not
promised to be better. The reader sees only the image that was kept.
</p>
</div>
<div class="field">
<label class="field__label" for="review_model_id">Model that reviews</label>
<select class="select" id="review_model_id" name="review_model_id">
<option value="">The chat's own model, when it has vision</option>
{% for model in vision_models %}
{% if model.capabilities_json.get("vision") %}
<option value="{{ model.id }}"
{{ 'selected' if values.review_model_id == model.id }}>
{{ model.label }}
</option>
{% endif %}
{% endfor %}
</select>
<p class="field__hint">
Only models marked as having vision are listed. If there is nothing to ask,
the first image is kept and nothing fails.
</p>
</div>
<div class="field">
<label class="field__label" for="max_tries">Attempts per image</label>
<input class="input" id="max_tries" name="max_tries" type="number"
min="1" max="10" value="{{ values.max_tries }}">
<p class="field__hint">
Including the first. The last attempt is kept whatever the review says, so
a request always produces a picture.
</p>
</div>
</section>
<section class="card">
<h2 class="card__title">Memory</h2>
<div class="field">
<label class="checkbox">
<input type="checkbox" name="preserve_vram" value="true"
{{ 'checked' if values.preserve_vram }}>
<span>Preserve VRAM: unload the language model while ComfyUI works</span>
</label>
<p class="field__hint">
For a machine that cannot hold both at once. Before generating, the chat's
own connection is asked to unload — set an unload URL on it under
Connections, or nothing happens. Afterwards ComfyUI is asked to free its
own models, and the language model loads again by itself on the next
request.
</p>
</div>
{% if values.preserve_vram and values.review_enabled %}
<div class="alert">
{{ icon("warning", "icon--sm") }}
<span>
Both are on, so every retry costs two model loads — one to review, one to
generate again. That is the slow combination; consider two attempts rather
than {{ values.max_tries }}.
</span>
</div>
{% endif %}
</section>
<section class="card">
<h2 class="card__title">Extra instructions</h2>
<div class="field">
<label class="field__label visually-hidden" for="instructions">Extra instructions</label>
<textarea class="textarea" id="instructions" name="instructions" rows="4"
placeholder="Prefer the SDXL template for anything photographic.">{{ values.instructions }}</textarea>
<p class="field__hint">
Added to what every model is told about image generation, on this instance.
Where “always put <em>text, watermark</em> in the negative prompt” lives.
Leave it empty and nothing is added at all.
</p>
</div>
</section>
<div class="btn-row">
<button class="btn btn--primary" type="submit">Save changes</button>
</div>
</form>
<section class="card">
<h2 class="card__title">Workflows</h2>
<p class="field__hint">
A workflow is a ComfyUI graph exported with <strong>Export (API)</strong>, with
<code>{{ '{{prompt}}' }}</code> and the other placeholders where the values go.
The model picks between them by their descriptions, so write the description
for a reader who cannot see the graph.
</p>
<div class="btn-row">
<a class="btn btn--primary" href="/admin/images/workflows/new">
{{ icon("plus", "icon--sm") }} Add a workflow
</a>
</div>
{% if not workflows %}
<div class="empty" style="padding: var(--sp-8) 0">
{{ icon("image", "empty__mark") }}
<p class="empty__text">
No workflows yet. One is needed before anything can be drawn; the default
one is filled in for you when you add the first.
</p>
</div>
{% else %}
<div class="model-rows">
{% for item in workflows %}
<div class="model-row {{ 'is-off' if not item.enabled }}">
<div class="model-row__main">
<span class="model-row__name">
{{ item.name }}
{% if not item.enabled %}<span class="badge badge--danger">disabled</span>{% endif %}
{% if values.default_workflow_id == item.id %}
<span class="badge">default</span>
{% endif %}
</span>
<code class="model-row__id">{{ item.slug }}</code>
{% if item.description %}
<p class="text-xs faint">{{ item.description }}</p>
{% endif %}
</div>
<div class="model-row__actions">
<a class="btn btn--sm" href="/admin/images/workflows/{{ item.id }}/edit">Edit</a>
</div>
</div>
{% endfor %}
</div>
<form method="post" action="/admin/images" class="field" style="margin-top: var(--sp-4)">
{# Every other value is resubmitted with it, because this posts to the same
handler as the form above and an absent field reads as cleared. #}
<input type="hidden" name="enabled" value="{{ 'true' if values.enabled }}">
<input type="hidden" name="base_url" value="{{ values.base_url }}">
<input type="hidden" name="api_key" value="{{ unchanged if values.api_key_encrypted }}">
<input type="hidden" name="timeout" value="{{ values.timeout }}">
<input type="hidden" name="checkpoints" value="{{ checkpoints_text }}">
<input type="hidden" name="review_enabled" value="{{ 'true' if values.review_enabled }}">
<input type="hidden" name="review_model_id" value="{{ values.review_model_id }}">
<input type="hidden" name="max_tries" value="{{ values.max_tries }}">
<input type="hidden" name="preserve_vram" value="{{ 'true' if values.preserve_vram }}">
<input type="hidden" name="instructions" value="{{ values.instructions }}">
<label class="field__label" for="default_workflow_id">Default workflow</label>
<div class="btn-row">
<select class="select" id="default_workflow_id" name="default_workflow_id">
<option value="">The first one</option>
{% for item in workflows %}
<option value="{{ item.id }}"
{{ 'selected' if values.default_workflow_id == item.id }}>{{ item.name }}</option>
{% endfor %}
</select>
<button class="btn" type="submit">Set default</button>
</div>
<p class="field__hint">
Used when the model names no template and the chat has no preference.
</p>
</form>
{% endif %}
</section>
{% endblock %}
@@ -0,0 +1,101 @@
{% extends "admin/_layout.html" %}
{% from "_macros.html" import icon %}
{% set section = "images" %}
{% block title %}{{ workflow.name or "New workflow" }} - LLeMbas{% endblock %}
{% block heading %}{{ workflow.name or "New workflow" }}{% endblock %}
{% block admin_content %}
<nav class="crumbs">
<a class="crumbs__back" href="/admin/images">
{{ icon("chevron-right", "icon--sm crumbs__icon") }} Image generation
</a>
</nav>
{% if error %}
<div class="alert alert--error">{{ icon("warning", "icon--sm") }} <span>{{ error }}</span></div>
{% endif %}
<form method="post"
action="{{ '/admin/images/workflows' if is_new else '/admin/images/workflows/' ~ workflow.id }}"
class="form-grid">
<section class="card">
<h2 class="card__title">What it is</h2>
<div class="field">
<label class="field__label" for="name">Name</label>
<input class="input" id="name" name="name" required maxlength="120"
value="{{ workflow.name }}" placeholder="SDXL, photographic">
<p class="field__hint">Shown to you, in the list.</p>
</div>
<div class="field">
<label class="field__label" for="slug">Name the model uses</label>
<input class="input input--mono" id="slug" name="slug" required maxlength="48"
value="{{ workflow.slug }}" placeholder="sdxl-photo">
<p class="field__hint">
Lowercase letters, digits, hyphens and underscores. This is what the model
writes when it picks this workflow.
</p>
</div>
<div class="field">
<label class="field__label" for="description">Description</label>
<textarea class="textarea" id="description" name="description" rows="3"
placeholder="Photographic and slow. Best for people, interiors and product shots at 1024px.">{{ workflow.description }}</textarea>
<p class="field__hint">
The only thing the model has to choose with, so say what this is
<em>for</em> rather than what it contains. It never sees the graph.
</p>
</div>
<div class="field">
<label class="checkbox">
<input type="checkbox" name="enabled" value="true" {{ 'checked' if workflow.enabled }}>
<span>Offer this workflow</span>
</label>
</div>
</section>
<section class="card">
<h2 class="card__title">The workflow</h2>
<div class="field">
<label class="field__label" for="workflow">ComfyUI API format</label>
<textarea class="textarea input--mono" id="workflow" name="workflow" rows="20"
spellcheck="false">{{ workflow_text }}</textarea>
<p class="field__hint">
Export this from ComfyUI with <strong>Export (API)</strong>, not Save — the
two formats are different and only the API one can be submitted.
</p>
<p class="field__hint">
Then put a placeholder where each value goes. A placeholder that is the
<em>whole</em> value keeps its type, so
<code>"steps": {{ '{{steps}}' }}</code> sends the number 20 rather than the
text “20”; one inside a longer string is substituted as text, so
<code>"{{ '{{prompt}}' }}, masterpiece"</code> works. Anything you leave out
takes its default.
</p>
<p class="field__hint">
Available:
{% for name in placeholders %}<code>{{ '{{' ~ name ~ '}}' }}</code>{{ ", " if not loop.last }}{% endfor %}.
<code>{{ '{{prompt}}' }}</code> is required — without it every image would
be the same.
</p>
</div>
</section>
<div class="btn-row">
<button class="btn btn--primary" type="submit">
{{ "Add workflow" if is_new else "Save changes" }}
</button>
<a class="btn btn--ghost" href="/admin/images">Back</a>
{% if not is_new %}
<button class="btn btn--danger" type="submit" formnovalidate
formaction="/admin/images/workflows/{{ workflow.id }}/delete"
data-confirm-button="Delete the workflow “{{ workflow.name }}”? Chats that used it fall back to the default.">
Delete
</button>
{% endif %}
</div>
</form>
{% endblock %}
@@ -106,10 +106,32 @@
{% if event.error %}
<p class="tool-activity__error">{{ event.error }}</p>
{% elif not event.results and not event.text %}
{% elif not event.results and not event.text and not event.image %}
<p class="tool-activity__error">Nothing was found.</p>
{% endif %}
{% if event.image and event.image.id %}
{#
What was drawn, in the block that drew it. The picture is also an
attachment on this bubble, so the reader sees it without opening
anything; this copy is what makes the block make sense on its own once
it is expanded, beside the seed and the attempts.
The address is built here from an id and is never taken from the event.
That is the same rule the result links a few lines down follow, and it
matters more here: an event is a dict a runner wrote, and a `src` taken
from one would be a model-supplied URL fetched by the reader's browser.
`/api/files/{id}/content` is owner-checked on the way out.
#}
<a class="tool-image" href="/api/files/{{ event.image.id }}/content"
target="_blank" rel="noopener">
<img src="/api/files/{{ event.image.id }}/content" alt="The generated image"
loading="lazy"
{% if event.image.width %}width="{{ event.image.width }}"{% endif %}
{% if event.image.height %}height="{{ event.image.height }}"{% endif %}>
</a>
{% endif %}
{% if event.text %}
<pre class="tool-result__text">{{ event.text }}</pre>
{% endif %}