File attachments: images for vision, PDFs and text into the prompt

Drag, paste or pick a file in the composer. Images go to vision models
as multimodal content parts; PDFs and text files have their content
extracted and placed in the prompt. Verified end to end against
gemma4-e4b-q8 on llama-swap: given a drawing and a text file, it named
the red square and blue circle and read the number out of the document.

Type is decided by inspecting the bytes, never the filename or the
browser's Content-Type -- a .png full of text is stored as text. Images
are downscaled to 1400px and re-encoded: a phone photo is several
megabytes of base64, which is slow and a large slice of the context
window. PDF text is extracted once, at upload, and stored; re-extracting
per request would let a reply change because a parser was upgraded.

Design points worth keeping:

- Images are only sent to models an administrator has marked `vision`.
  This is not graceful degradation -- most endpoints reject the entire
  request rather than ignoring an image part. A plain text turn stays a
  plain string for the same reason: the list form 400s on endpoints that
  do not implement it.
- Images reach the model as base64 data URIs, not links. A local
  endpoint has no route back to LLeMbas, and a hosted one has no
  credentials for it.
- Non-images are served Content-Disposition: attachment with nosniff, so
  an uploaded .html can never execute in this origin. Stored names are
  random; the uploader's name is a label and never a path.
- Uploads are unbound until the message is sent, which is what lets a
  file be removed beforehand. claim() only takes unclaimed rows owned by
  the sender, so a forged id cannot pull in someone else's file.
  Abandoned uploads are swept at startup.
- A scanned PDF says so rather than silently contributing nothing, and
  truncation is declared to the model in the document tag so it can
  admit it did not see page 400.
- "Here, look at this" with no words is a legitimate turn, so a message
  is only empty when it carries neither text nor files.

Also fixes auto-titling, which read message["content"] as a string and
would have broken on the first multimodal turn.

186 tests, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-07-21 12:19:59 +02:00
parent d6c87ac811
commit bdce2764b1
19 changed files with 1601 additions and 20 deletions
+30 -6
View File
@@ -19,7 +19,7 @@ lembas info # paths + counts, useful when confused
lembas secret-key # generate LEMBAS_SECRET_KEY
lembas create-admin # create or promote an admin
pytest # 143 tests, ~5s
pytest # 186 tests, ~7s
ruff check . # lint (line length 100)
python scripts/build_artwork.py # regenerate all SVG artwork
python scripts/fetch_vendor.py # verify vendored JS against the lockfile
@@ -75,6 +75,7 @@ src/lembas/
admin.py connections + instance settings
admin_models.py model ordering, defaults, images, access
admin_users.py users, groups, permissions
files.py upload, serve, remove attachments
preferences.py per-user theme, default model, password
db/
base.py Base, UUID/Timestamp mixins
@@ -87,6 +88,7 @@ src/lembas/
chat.py request building, endpoint resolution, titles
markdown.py markdown-it + pygments + nh3
crypto.py Fernet encrypt/decrypt/mask
files.py attachment validation, images, PDF/text extraction
reasoning.py splits thinking from the answer
settings_store.py runtime instance settings
uploads.py validated image storage
@@ -167,6 +169,27 @@ llama-swap, vLLM) or `<think>` tags inline in `content` (Ollama and friends).
the tags arrive split across chunks. Reasoning is stored in `Message.reasoning`
and is deliberately **not** replayed as context on the next turn.
**Attachments are typed by their bytes, not their name.** `services/files.py`
sniffs magic numbers; a `.png` full of text is stored as text. Images are
downscaled and re-encoded (a phone photo is megabytes of base64), PDFs have
their text extracted **once at upload** — re-extracting per request would let a
reply change because a parser was upgraded.
**Images only go to models marked `vision`.** Sending content parts to an
endpoint without multimodal support is not graceful degradation; most reject
the whole request. `build_request()` checks the capability and falls back to a
plain string. A plain text turn must *stay* a plain string for the same reason.
**Attachments are served, never linked.** Images reach the model as base64 data
URIs: a local endpoint has no route back to LLeMbas and a hosted one has no
credentials. Non-images are served `Content-Disposition: attachment` with
`nosniff`, so an uploaded `.html` cannot execute in this origin.
**Uploads are unbound until the message is sent.** `Attachment.message_id` is
null in the composer; `files.claim()` binds them, and only unclaimed rows owned
by that user, so a forged id cannot pull in someone else's file. Abandoned ones
are swept at startup.
**JSON columns need reassignment.** `user.settings_json["theme"] = x` on a
plain dict is not detected. The columns use `MutableDict` (`db/types.py`), but
the safe habit is `obj.field = {**obj.field, "k": v}`.
@@ -215,9 +238,10 @@ notes describe the machine.
## Not built yet
File upload / vision / PDFs, built-in tools + admin tool settings, custom tools
and MCP, agentic execution (local subprocess and SSH connection profiles), image
generation. Nav entries mark where each one goes.
Built-in tools + admin tool settings, custom tools and MCP, agentic execution
(local subprocess and SSH connection profiles), image generation. Nav entries
mark where each one goes.
`Model.capabilities_json` already carries `vision` and `tools` flags that
nothing reads yet — they are admin overrides waiting for those features.
`Model.capabilities_json` carries a `tools` flag nothing reads yet. No OCR:
a scanned PDF is stored with an explanatory `extraction_error` rather than
silently contributing nothing.