File attachments: images for vision, PDFs and text into the prompt
Drag, paste or pick a file in the composer. Images go to vision models as multimodal content parts; PDFs and text files have their content extracted and placed in the prompt. Verified end to end against gemma4-e4b-q8 on llama-swap: given a drawing and a text file, it named the red square and blue circle and read the number out of the document. Type is decided by inspecting the bytes, never the filename or the browser's Content-Type -- a .png full of text is stored as text. Images are downscaled to 1400px and re-encoded: a phone photo is several megabytes of base64, which is slow and a large slice of the context window. PDF text is extracted once, at upload, and stored; re-extracting per request would let a reply change because a parser was upgraded. Design points worth keeping: - Images are only sent to models an administrator has marked `vision`. This is not graceful degradation -- most endpoints reject the entire request rather than ignoring an image part. A plain text turn stays a plain string for the same reason: the list form 400s on endpoints that do not implement it. - Images reach the model as base64 data URIs, not links. A local endpoint has no route back to LLeMbas, and a hosted one has no credentials for it. - Non-images are served Content-Disposition: attachment with nosniff, so an uploaded .html can never execute in this origin. Stored names are random; the uploader's name is a label and never a path. - Uploads are unbound until the message is sent, which is what lets a file be removed beforehand. claim() only takes unclaimed rows owned by the sender, so a forged id cannot pull in someone else's file. Abandoned uploads are swept at startup. - A scanned PDF says so rather than silently contributing nothing, and truncation is declared to the model in the document tag so it can admit it did not see page 400. - "Here, look at this" with no words is a legitimate turn, so a message is only empty when it carries neither text nor files. Also fixes auto-titling, which read message["content"] as a string and would have broken on the first multimodal turn. 186 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -19,7 +19,7 @@ lembas info # paths + counts, useful when confused
|
||||
lembas secret-key # generate LEMBAS_SECRET_KEY
|
||||
lembas create-admin # create or promote an admin
|
||||
|
||||
pytest # 143 tests, ~5s
|
||||
pytest # 186 tests, ~7s
|
||||
ruff check . # lint (line length 100)
|
||||
python scripts/build_artwork.py # regenerate all SVG artwork
|
||||
python scripts/fetch_vendor.py # verify vendored JS against the lockfile
|
||||
@@ -75,6 +75,7 @@ src/lembas/
|
||||
admin.py connections + instance settings
|
||||
admin_models.py model ordering, defaults, images, access
|
||||
admin_users.py users, groups, permissions
|
||||
files.py upload, serve, remove attachments
|
||||
preferences.py per-user theme, default model, password
|
||||
db/
|
||||
base.py Base, UUID/Timestamp mixins
|
||||
@@ -87,6 +88,7 @@ src/lembas/
|
||||
chat.py request building, endpoint resolution, titles
|
||||
markdown.py markdown-it + pygments + nh3
|
||||
crypto.py Fernet encrypt/decrypt/mask
|
||||
files.py attachment validation, images, PDF/text extraction
|
||||
reasoning.py splits thinking from the answer
|
||||
settings_store.py runtime instance settings
|
||||
uploads.py validated image storage
|
||||
@@ -167,6 +169,27 @@ llama-swap, vLLM) or `<think>` tags inline in `content` (Ollama and friends).
|
||||
the tags arrive split across chunks. Reasoning is stored in `Message.reasoning`
|
||||
and is deliberately **not** replayed as context on the next turn.
|
||||
|
||||
**Attachments are typed by their bytes, not their name.** `services/files.py`
|
||||
sniffs magic numbers; a `.png` full of text is stored as text. Images are
|
||||
downscaled and re-encoded (a phone photo is megabytes of base64), PDFs have
|
||||
their text extracted **once at upload** — re-extracting per request would let a
|
||||
reply change because a parser was upgraded.
|
||||
|
||||
**Images only go to models marked `vision`.** Sending content parts to an
|
||||
endpoint without multimodal support is not graceful degradation; most reject
|
||||
the whole request. `build_request()` checks the capability and falls back to a
|
||||
plain string. A plain text turn must *stay* a plain string for the same reason.
|
||||
|
||||
**Attachments are served, never linked.** Images reach the model as base64 data
|
||||
URIs: a local endpoint has no route back to LLeMbas and a hosted one has no
|
||||
credentials. Non-images are served `Content-Disposition: attachment` with
|
||||
`nosniff`, so an uploaded `.html` cannot execute in this origin.
|
||||
|
||||
**Uploads are unbound until the message is sent.** `Attachment.message_id` is
|
||||
null in the composer; `files.claim()` binds them, and only unclaimed rows owned
|
||||
by that user, so a forged id cannot pull in someone else's file. Abandoned ones
|
||||
are swept at startup.
|
||||
|
||||
**JSON columns need reassignment.** `user.settings_json["theme"] = x` on a
|
||||
plain dict is not detected. The columns use `MutableDict` (`db/types.py`), but
|
||||
the safe habit is `obj.field = {**obj.field, "k": v}`.
|
||||
@@ -215,9 +238,10 @@ notes describe the machine.
|
||||
|
||||
## Not built yet
|
||||
|
||||
File upload / vision / PDFs, built-in tools + admin tool settings, custom tools
|
||||
and MCP, agentic execution (local subprocess and SSH connection profiles), image
|
||||
generation. Nav entries mark where each one goes.
|
||||
Built-in tools + admin tool settings, custom tools and MCP, agentic execution
|
||||
(local subprocess and SSH connection profiles), image generation. Nav entries
|
||||
mark where each one goes.
|
||||
|
||||
`Model.capabilities_json` already carries `vision` and `tools` flags that
|
||||
nothing reads yet — they are admin overrides waiting for those features.
|
||||
`Model.capabilities_json` carries a `tools` flag nothing reads yet. No OCR:
|
||||
a scanned PDF is stored with an explanatory `extraction_error` rather than
|
||||
silently contributing nothing.
|
||||
|
||||
Reference in New Issue
Block a user