File attachments: images for vision, PDFs and text into the prompt

Drag, paste or pick a file in the composer. Images go to vision models as multimodal content parts; PDFs and text files have their content extracted and placed in the prompt. Verified end to end against gemma4-e4b-q8 on llama-swap: given a drawing and a text file, it named the red square and blue circle and read the number out of the document. Type is decided by inspecting the bytes, never the filename or the browser's Content-Type -- a .png full of text is stored as text. Images are downscaled to 1400px and re-encoded: a phone photo is several megabytes of base64, which is slow and a large slice of the context window. PDF text is extracted once, at upload, and stored; re-extracting per request would let a reply change because a parser was upgraded. Design points worth keeping: - Images are only sent to models an administrator has marked `vision`. This is not graceful degradation -- most endpoints reject the entire request rather than ignoring an image part. A plain text turn stays a plain string for the same reason: the list form 400s on endpoints that do not implement it. - Images reach the model as base64 data URIs, not links. A local endpoint has no route back to LLeMbas, and a hosted one has no credentials for it. - Non-images are served Content-Disposition: attachment with nosniff, so an uploaded .html can never execute in this origin. Stored names are random; the uploader's name is a label and never a path. - Uploads are unbound until the message is sent, which is what lets a file be removed beforehand. claim() only takes unclaimed rows owned by the sender, so a forged id cannot pull in someone else's file. Abandoned uploads are swept at startup. - A scanned PDF says so rather than silently contributing nothing, and truncation is declared to the model in the document tag so it can admit it did not see page 400. - "Here, look at this" with no words is a legitimate turn, so a message is only empty when it carries neither text nor files. Also fixes auto-titling, which read message["content"] as a string and would have broken on the first multimodal turn. 186 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 12:19:59 +02:00
parent 1d3f6c450b
commit d90195015c
18 changed files with 1571 additions and 14 deletions
@@ -16,6 +16,7 @@ from lembas.db.models import ROLE_ASSISTANT, ROLE_USER, Chat, Message, User
 from lembas.db.session import session_scope
 from lembas.security import permissions
 from lembas.services import chat as chat_service
+from lembas.services import files as files_service
 from lembas.services import sse
 from lembas.services.llm.openai_client import (
    LLMError,
@@ -66,7 +67,8 @@ async def post_message(
    db: Db,
    user: RequiredUser,
    chat_id: str,
-    content: str = Form(...),
+    content: str = Form(""),
+    file_ids: list[str] = Form(default=[]),
 ) -> Response:
    """Persist the user's turn and hand back the pair of bubbles.

@@ -77,10 +79,15 @@ async def post_message(
    chat = _owned_chat(db, chat_id, user.id)

    content = content.strip()
-    if not content:
+    # "Here, look at this" with no words is a legitimate turn, so an empty
+    # message is only empty when it carries nothing at all.
+    if not content and not file_ids:
        return Response(status_code=status.HTTP_204_NO_CONTENT)

    user_message = chat_service.create_message(db, chat, ROLE_USER, content)
+    if file_ids:
+        files_service.claim(db, ids=file_ids, user_id=user.id, message_id=user_message.id)
+        db.refresh(user_message)
    assistant_message = chat_service.create_message(
        db, chat, ROLE_ASSISTANT, "", complete_=False, model_id=chat.model_id
    )
@@ -130,6 +137,19 @@ async def stream_message(
    )


+def _plain_text(content: str | list) -> str:
+    """The text of a message payload, whether it is a string or content parts."""
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        return " ".join(
+            part.get("text", "")
+            for part in content
+            if isinstance(part, dict) and part.get("type") == "text"
+        ).strip()
+    return ""
+
+
 async def _generate(chat_id: str, message_id: str) -> AsyncIterator[str]:
    """Drive one completion and frame it as SSE.

@@ -159,8 +179,14 @@ async def _generate(chat_id: str, message_id: str) -> AsyncIterator[str]:
        try:
            endpoint, model_id = chat_service.resolve_endpoint(db, chat)
            payload = chat_service.build_request(db, chat, upto=message)
+            # A multimodal turn's content is a list of parts, not a string, so
+            # the text has to be picked out before it can title a chat.
            first_user_text = next(
-                (m["content"] for m in reversed(payload["messages"]) if m["role"] == ROLE_USER),
+                (
+                    _plain_text(m["content"])
+                    for m in reversed(payload["messages"])
+                    if m["role"] == ROLE_USER
+                ),
                "",
            )

@@ -0,0 +1,117 @@
+"""Uploading, serving and removing chat attachments."""
+
+from __future__ import annotations
+
+import logging
+
+from fastapi import APIRouter, Depends, File, HTTPException, Request, Response, UploadFile, status
+from fastapi.responses import FileResponse
+
+from lembas.api.deps import Db, RequiredUser, require_permission
+from lembas.db.models import Attachment
+from lembas.services import files as files_service
+from lembas.web.templating import templates
+
+log = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/files", tags=["files"])
+
+
+def _owned(db: Db, attachment_id: str, user_id: str) -> Attachment:
+    attachment = db.get(Attachment, attachment_id)
+    if attachment is None or attachment.user_id != user_id:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "That file no longer exists.")
+    return attachment
+
+
+@router.post("", dependencies=[Depends(require_permission("files.upload"))])
+async def upload(
+    request: Request,
+    db: Db,
+    user: RequiredUser,
+    file: UploadFile = File(...),
+    chat_id: str = "",
+) -> Response:
+    """Accept one file and return the chip that represents it in the composer.
+
+    The attachment is stored immediately but left unbound: it only joins a
+    message when that message is sent. That is what lets a file be removed
+    before sending, and what the orphan sweep later cleans up.
+    """
+    payload = await file.read()
+
+    try:
+        attachment = files_service.store(
+            db,
+            user_id=user.id,
+            chat_id=chat_id or None,
+            payload=payload,
+            filename=file.filename or "file",
+        )
+    except files_service.FileError as exc:
+        # 200 with an error chip rather than a 4xx: htmx swaps the response
+        # body either way, and an error the user can read beats a silent
+        # failure in the console.
+        return templates.TemplateResponse(
+            request,
+            "chat/_attachment_error.html",
+            {"request": request, "filename": file.filename or "file", "error": str(exc)},
+        )
+
+    return templates.TemplateResponse(
+        request,
+        "chat/_attachment_chip.html",
+        {"request": request, "attachment": attachment},
+    )
+
+
+@router.delete("/{attachment_id}")
+async def remove(db: Db, user: RequiredUser, attachment_id: str) -> Response:
+    """Detach a file before it has been sent."""
+    attachment = _owned(db, attachment_id, user.id)
+    if attachment.message_id is not None:
+        # Deleting it now would rewrite a conversation that has already been
+        # sent to a model and read by the user.
+        raise HTTPException(
+            status.HTTP_409_CONFLICT, "That file is part of a sent message."
+        )
+    files_service.delete(db, attachment)
+    return Response(status_code=status.HTTP_200_OK)
+
+
+@router.get("/{attachment_id}/content")
+async def content(db: Db, user: RequiredUser, attachment_id: str) -> Response:
+    """Serve an attachment back to its owner."""
+    attachment = _owned(db, attachment_id, user.id)
+    path = files_service.stored_path(attachment.stored_name)
+    if path is None:
+        raise HTTPException(status.HTTP_404_NOT_FOUND, "That file is no longer on disk.")
+
+    # inline for images so they render in the thread; attachment for everything
+    # else so a text/html upload can never be executed in this origin.
+    disposition = "inline" if attachment.is_image else "attachment"
+    return FileResponse(
+        path,
+        media_type=attachment.media_type if attachment.is_image else "application/octet-stream",
+        headers={
+            "Content-Disposition": f'{disposition}; filename="{attachment.filename}"',
+            "Cache-Control": "private, max-age=604800",
+            # Belt and braces: even for images, never let a browser sniff its
+            # way to treating the bytes as something executable.
+            "X-Content-Type-Options": "nosniff",
+        },
+    )
+
+
+@router.get("/{attachment_id}/text")
+async def extracted_text(db: Db, user: RequiredUser, attachment_id: str) -> Response:
+    """The text a document contributed to the prompt.
+
+    Worth being able to see: a PDF that extracted badly explains a strange
+    reply, and there is otherwise no way to tell what the model was given.
+    """
+    attachment = _owned(db, attachment_id, user.id)
+    return Response(
+        attachment.extracted_text or attachment.extraction_error,
+        media_type="text/plain; charset=utf-8",
+    )