e16bede85b36e05816ba092ff57c473a999dcf21
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jaroslav Beneš
|
5e75948069 |
Draw a picture, on a ComfyUI you are running
The last unbuilt capability, and built the way CLAUDE.md said it had to be: a
ToolDef reaching resolve_tools plus a permission and a capability flag, not a new
code path. The only genuinely new UI is one branch in the transcript.
services/images/ is three modules. comfy.py speaks HTTP -- submit, poll /history,
fetch the PNG, /free, and an /object_info discovery for the admin page only.
Polled and not socketed, because holding a connection open for the length of a
generation is the live-connection state the whole ssh.py design forbids, and the
thing being waited for takes tens of seconds anyway. The base URL is exempt from
the SSRF guard by construction, exactly as Connection.base_url and the audio
endpoints are -- said out loud in the docstring, because a default of
127.0.0.1:8188 is precisely the shape that guard exists to refuse and therefore
reads as a hole rather than a decision.
workflow.py fills a template, and the one thing that matters is that it walks the
parsed JSON rather than the text of it. A value that is exactly "{{steps}}"
becomes the number 20; ComfyUI validates types and refuses the string. A
placeholder inside a longer string is still text, which is what makes
"{{prompt}}, masterpiece" work -- and text substitution would additionally mean a
prompt containing a quotation mark produced a document that no longer parses, on
the one input guaranteed to hold arbitrary text. Which node holds the prompt is
the administrator's statement rather than a guess from node types: sniffing for
the first CLIPTextEncode works on the shipped workflow and on nothing else, and
swaps positive for negative the first time somebody reorders them. seed has no
fixed default, because one would make every unspecified generation identical and
make the retry loop redraw the same rejected picture four times.
tool.py is one call, one finished image. Returning every attempt to the
conversation would cost a round each, make the ceiling advisory rather than
enforced, and walk the reader past every reject -- so the reviewer lives inside
the tool and is asked about *bytes*: an attempt about to be discarded should not
leave an Attachment behind, so it sees a downscaled preview built in memory and
only the kept image is written. Anything that goes wrong in review is a keep;
losing a picture because a judging request timed out would be the check
destroying the thing it was checking. The last attempt is kept whatever the
verdict, so a request always produces something. Rejects are recorded, not
stored.
Preserve VRAM unloads the chat's own connection and nothing else, because the
memory being freed belongs to one machine: local llama-swap answers GET /unload,
and a box on the network has no reason to be unloaded when ComfyUI wants memory
here. The swap goes round the review rather than round the tool, which costs two
model loads per retry -- so the two settings are independent and the page warns
when both are on. Nothing loads the LLM back: the reply's next request does, and
that step exists in the description and not in the code, so the code says so.
Two rules elsewhere had to be drawn for the first time. message_payload sends
images only on user turns -- no assistant message had ever carried one, and the
moment one does the multimodal list form on an assistant turn is rejected by
OpenAI and most local runners, breaking every later turn in the chat. And
files.store gained keep_original, because _process_image turns anything without
alpha into JPEG q85 at 1400px: right for a phone photo, a visible loss on the one
output this feature exists to produce.
/image sends the ordinary message with force_tool, which becomes tool_choice for
the first round only -- left in place the reply would draw a picture, be asked
again, and draw another. FORCEABLE_TOOLS is an allow list because the name is
read off a form.
ToolContext gained chat_id, and that fixed a tool nobody had ever successfully
run: _run_scratch_write read context.chat_id on a dataclass with no such field,
so every call raised AttributeError, swallowed by run_tool's blanket except into
"the scratch_write tool failed" -- indistinguishable from a model calling it
wrongly. The test that existed asserted the family and the risk, which are
properties of the declaration rather than of the code.
Verified against the real ComfyUI 0.27.0 on this machine rather than against
documentation: every endpoint shape here was read off it, a generation ran end to
end through the client, the reviewer was shown a matching and a mismatched prompt
and answered KEEP and RETRY correctly, and the unload hook fired for the local
llama-swap and not for the remote box.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Jaroslav Beneš
|
8a3a225fea |
Two selects that never wrote anything, and a queue
The approval card in Auto mode and the missing /effort were one bug. Both
selects hung their hx-patch on an empty sibling form reached by form="…",
and htmx binds a trigger to the annotated element: change fires on the
select and bubbles to its ancestors, which a sibling is not. The live rows
read agent_mode=manual and params_json={} while the browser showed Auto and
Effort: high. policy.py was never involved.
The verb moves onto the control; the empty form stays as value scoping,
which is the half of the CLAUDE.md note that was right. conftest gains
control_named so a test asserts the element carrying the name carries the
verb, rather than asserting the markup that was there throughout.
The composer's highlight was a third instance of the same carelessness in
CSS: .tok-mention is written for the transcript and scoped to nothing, so
the mirror painted its token in accent-coloured monospace over the
textarea's own text. Scoped under .msg; the mirror restates transparency
and font rather than inheriting them, and bleeds by box-shadow.
/effort is now offered before the first prompt and _new_chat reads it.
/index re-walks the project directory on demand, file_write drops the
listing it just invalidated, and the index ladder falls through to SFTP on
a host that refuses exec instead of returning nothing.
A second message during a reply is queued rather than starting a second
concurrent generation: a real Message row with queued set, so it survives a
restart and can be withdrawn. _drain hands one on at the end of a reply,
_inject takes one in at a tool-round boundary so an agent can be steered
mid-task. Stop leaves the queue undelivered. The terminal's Auto toggle
becomes off/copy/send, and send posts straight to the chat without touching
the composer.
@ now offers notes, skills, this chat's attachments and a URL to fetch; a
knowledge base attaches as a reference rather than a copy. copy_document
carries provenance, which was the one attach path that dropped it.
Also fixes an unrelated live bug: the round loop compared against the
global MAX_ROUNDS of 3 while sizing itself from the agent budget of 40, so
agent replies stopped after three rounds and reported forty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Jaroslav Beneš
|
b6cea42631 |
A directory the model knows about, and @ to name a file in it
An agent chat used to open with the model knowing the name of a machine and nothing about what was on it, so the first two rounds of every reply went on finding out. It now gets a listing: one read-only command, `git ls-files` where that works and `find` otherwise, falling back to an SFTP walk that always does. git first because a repository already carries somebody's considered list of what is not part of the project, and reproducing it by hand is how an index ends up mostly build output. The listing is budgeted rather than dumped. A tree of a thousand files is worse than no tree -- it costs the window on every request forever and buries the four names that mattered -- so directories that will not fit are shown as a count and the model is told to open one itself. Collapsing picks the deepest and largest first: by saving alone it would take `src/` before `src/web/static/vendor/`, because it contains it, and lose every name worth having. Read from a cache and never fetched. `harness.context_variables` is synchronous and sits on the request path; the walk happens in the generation setup, which is async and already doing network work, with a short wait. A chat whose first reply outruns its first walk simply has no listing that turn and the fragment disappears rather than appearing as an empty heading. Then `@`, over the same index and over the library, and `/` for commands with an Alt-based keyboard for the same jobs. A mentioned file arrives as contents, not a reference -- a small model asked to call file_read often does not bother -- and it arrives with its absolute path and the machine it came from, because a model handed `main.py` cannot tell which of four it is and cannot name it back when asked to change something. The rule that matters for `/`: a message that merely starts with a slash still sends. `//` escapes and an unrecognised command is posted as written. Swallowing somebody's message is a much worse failure than an unknown command. Two exceptions to Manual mode now, not one. Browsing and indexing are a person acting, not a model, so neither passes through policy.py -- the same argument the terminal panel rests on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jaroslav Beneš
|
09eecbdd9a |
Temporary chats
A clock in the top-right starts one. It is never listed in the sidebar and is swept a day after the last thing said in it. A real row rather than something held in the browser, because a reload, a crash or a background tab all look identical from here -- "delete when you navigate away" would lose conversations people meant to keep. The flag rides in the URL (/chat?temporary=1) rather than in JavaScript, so it survives a reload and can be bookmarked, and the composer carries it as a hidden field beside model_id. Keep clears the flag. Without a way out, a conversation that turns out to matter is destroyed a day later with no recourse, and people would find that out exactly once. archived was filtered in three places and temporary mirrors all three, plus Folder.visible_chats. It also skips the unread flag in _persist: there is no sidebar row for the dot to land on, and the toast would name a chat nobody can navigate to. The sweep measures age from the newest message, not from the chat row. created_at would destroy a conversation still in use at hour 23, and updated_at does not move when a message is inserted -- onupdate fires on an UPDATE of the chat, and adding a message is not one. It runs at startup beside the existing upload sweep. Deleting a chat cascades its rows but leaves the files on disk; only the orphan sweep unlinks anything, and it looks only at uploads that were never attached. files.remove_files_for_chats() closes that for the new sweep. The same hole in delete_chat is pre-existing and left for its own change, which can now call the same helper. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jaroslav Beneš
|
21001f2eb8 |
Knowledge, notes, memory and skills, and a harness to make them used
Four places a model can reach for, differing in who writes a record and how it gets in front of the model. **Knowledge** is uploaded by a person and searched by the model. It goes through `services/files.py:prepare` — the same pipeline as a chat attachment — so the same PDF produces the same text whichever way it arrived, and `Document` carries the same content columns as `Attachment` for the same reason. **Notes** are written by the model and edited by you. Too long to inject, so they are searched. **Memory** is short facts, and every one of them goes into every request. That single decision is where the rest of its design comes from: records are capped short, the block has a budget, there is no search tool because the model is already looking at them, and they are not shareable — a record about a person is not content to hand round. **Skills** are saved procedures. Only the name and description are injected; the body is fetched when the model decides one applies, which is what makes a hundred skills affordable. A model may write and revise its own — the safety story is not a gate but a record: every revision is kept, attributed and revertible. A model that has just read a hostile page can save a skill that outlives the conversation, and the honest mitigation is that it is visible and undoable rather than that it was prevented. **The harness** is why any of it gets used. A model handed a tools array ignores it and answers from recall, because nothing in the request suggests otherwise. `services/harness.py` assembles a preamble from what this chat actually has: when to reach for each tool, the memories, the skill index. This is an exception to "system prompts are precedence, not concatenation", and a deliberate one. That rule governs the three *authored* layers and is untouched — exactly one still wins. The harness is a different axis: it describes the machinery rather than the behaviour, nobody authored it, and there is nothing for it to disagree with. It is prepended to whichever authored prompt won, in one system message, since several endpoints reject a second. Supporting changes: - **Sharing**, in one helper. `visible_to()` is the only definition of who can see a library item and every listing and tool goes through it. Sharing grants *reading*; two people editing one note with no history and no merge is worse than copying it. **Administrators do not bypass this** — they bypass permissions elsewhere because an admin can grant themselves those anyway, but reading somebody's private notes is a different act. - **FTS5**, created by `db/migrations.py:ensure_fts` with the triggers an external-content index needs. Idempotent, like the column sync beside it. Terms are ANDed and then ORed: the caller is usually a model writing a whole question, and requiring every word loses the match on one absent term. - **The attach button is a menu** — file, image, a web page, or a document from the library. Attaching a document copies it, because history must not change when a document is edited later. - **A URL fetcher with an SSRF guard.** This server can reach the router, the other services on the box and LLeMbas itself, and the address can come from a model. Private ranges are refused *after resolution* and redirects are followed by hand so every hop is checked. An admin can open it deliberately. - **Model capabilities split** into protocol support and a toggle per built-in tool. Rows predating the split have no `tool_*` keys, and absent counts as on when `tools` is on — otherwise an upgrade silently takes web search away from every model already configured for it. Also fixes the test fixture, which built the schema with `create_all` and so ran against a database without the FTS tables production has; it now runs `sync_schema`, the same path startup takes. 430 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Jaroslav Beneš
|
bdce2764b1 |
File attachments: images for vision, PDFs and text into the prompt
Drag, paste or pick a file in the composer. Images go to vision models as multimodal content parts; PDFs and text files have their content extracted and placed in the prompt. Verified end to end against gemma4-e4b-q8 on llama-swap: given a drawing and a text file, it named the red square and blue circle and read the number out of the document. Type is decided by inspecting the bytes, never the filename or the browser's Content-Type -- a .png full of text is stored as text. Images are downscaled to 1400px and re-encoded: a phone photo is several megabytes of base64, which is slow and a large slice of the context window. PDF text is extracted once, at upload, and stored; re-extracting per request would let a reply change because a parser was upgraded. Design points worth keeping: - Images are only sent to models an administrator has marked `vision`. This is not graceful degradation -- most endpoints reject the entire request rather than ignoring an image part. A plain text turn stays a plain string for the same reason: the list form 400s on endpoints that do not implement it. - Images reach the model as base64 data URIs, not links. A local endpoint has no route back to LLeMbas, and a hosted one has no credentials for it. - Non-images are served Content-Disposition: attachment with nosniff, so an uploaded .html can never execute in this origin. Stored names are random; the uploader's name is a label and never a path. - Uploads are unbound until the message is sent, which is what lets a file be removed beforehand. claim() only takes unclaimed rows owned by the sender, so a forged id cannot pull in someone else's file. Abandoned uploads are swept at startup. - A scanned PDF says so rather than silently contributing nothing, and truncation is declared to the model in the document tag so it can admit it did not see page 400. - "Here, look at this" with no words is a legitimate turn, so a message is only empty when it carries neither text nor files. Also fixes auto-titling, which read message["content"] as a string and would have broken on the first multimodal turn. 186 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |