PWA, one send/stop button, audio in and out, web search as a tool

Four pieces of work.

**Installable.** A manifest carrying the instance name, PWA icons rasterised
from the existing mark at design time, a service worker and a themed offline
page. The worker caches the shell only and bails out on /api/, /auth/, /admin/
and anything accepting text/event-stream -- passing a reply stream through a
worker turns it into one delivery at the end, or nothing. It is served from
GET /sw.js rather than the static mount because a worker's scope is the path it
came from.

**Send and Stop are one button.** They were two, and the hidden one was never
hidden: `.btn` is display: inline-flex, which outranks the browser's own
`[hidden] { display: none }`, so Stop sat permanently beside Send. app.css now
forces the attribute to win -- every control toggled with `hidden` depended on
that -- and the composer renders one button carrying both icons, with ui.js
flipping data-composer-action and the type with it.

**Audio.** Speech to text and text to speech against any OpenAI-shaped
/v1/audio/* endpoint: dictate into the composer, have a reply read out.
Instance settings in Admin, per-reader overrides in Settings, with the voice
list discovered from the server where it offers one. Recorded audio is capped
and never written to disk -- it is not an attachment, it has no owner, and
nothing would ever sweep it.

**Web search, as a tool.** This is the tool loop PLAN.md described as the real
work: one reply is now a bounded sequence of requests rather than one. The model
asks, the tool runs, the result goes back and it is asked again, up to three
rounds. Providers are DuckDuckGo (no setup), SearXNG and Firecrawl.

Two decisions worth stating. Tools are only offered to models flagged `tools`,
because an endpoint without support rejects the whole request rather than
ignoring the array -- the same reason images only reach models flagged
`vision`. And tool results are not replayed as context on the next turn, for the
same reasons reasoning is not: the answer already contains what the model made
of them, and replaying stale results into every later request wastes the window
and reliably sends a small model into a search loop. The sources stay visible in
the transcript instead.

Search results are untrusted third-party text and are treated as such: escaped,
and only http/https URLs rendered as links.

338 tests, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-07-21 17:56:50 +02:00
parent de178837b8
commit 7456525d19
63 changed files with 4597 additions and 140 deletions
+183
View File
@@ -0,0 +1,183 @@
"""Dictation and read-aloud.
Both directions go through the server rather than from the browser to the audio
endpoint directly, for the same reason model requests do: the endpoint is often
on a private address the browser cannot reach, and its API key must never leave
this process.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Depends, File, HTTPException, UploadFile, status
from fastapi.responses import PlainTextResponse, Response, StreamingResponse
from sqlalchemy.orm import Session as DBSession
from lembas.api.deps import Db, RequiredUser, require_permission
from lembas.db.models import Chat, Message, User
from lembas.services import audio as audio_service
from lembas.services import settings_store
from lembas.services.llm.openai_client import LLMError
from lembas.services.markdown import speakable_text
log = logging.getLogger(__name__)
router = APIRouter(prefix="/api/audio", tags=["audio"])
# A minute of speech is well under a megabyte in any browser codec; this is a
# ceiling on nonsense, not a budget. Recorded audio is held in memory and never
# written to disk: it is not an attachment, has no owner and nothing would ever
# sweep it up.
MAX_AUDIO_BYTES = 25 * 1024 * 1024
def _user_audio(user: User) -> dict:
return dict((user.settings_json or {}).get("audio") or {})
def resolve_voice(config: dict, user: User) -> str:
"""The voice a given user should be read to in.
Their own choice, then the instance default, then whatever the endpoint
picks. Not validated against the discovered list: a voice can disappear
when a server is reconfigured, and falling back beats failing.
"""
return (_user_audio(user).get("voice") or config.get("tts_voice") or "").strip()
def resolve_speed(config: dict, user: User) -> float:
"""The playback speed for this user, in the range every endpoint accepts.
Key presence decides which layer wins, not truthiness: chained `or` would
make a stored speed of 0 fall through to the default instead of being
clamped, which is a different answer for no stated reason.
"""
preferences = _user_audio(user)
if "speed" in preferences:
raw = preferences["speed"]
elif "tts_speed" in config:
raw = config["tts_speed"]
else:
return 1.0
try:
chosen = float(raw)
except (TypeError, ValueError):
return 1.0
# Clamped rather than dropped, unlike the sampling parameters: a speed of 0
# is not a slower reading, it is silence.
return min(max(chosen, 0.25), 4.0)
@router.post(
"/transcribe", dependencies=[Depends(require_permission("audio.transcribe"))]
)
async def transcribe(
db: Db, user: RequiredUser, file: UploadFile = File(...)
) -> Response:
"""Turn a recording into text for the composer.
Returns plain text, not HTML: the caller assigns it to a textarea's value,
where it is never parsed as markup.
"""
config = settings_store.audio(db)
if not config.get("stt_enabled"):
raise HTTPException(
status.HTTP_404_NOT_FOUND, "Dictation is not enabled on this instance."
)
data = await file.read(MAX_AUDIO_BYTES + 1)
if len(data) > MAX_AUDIO_BYTES:
raise HTTPException(
status.HTTP_413_CONTENT_TOO_LARGE, "That recording is too long."
)
if not data:
raise HTTPException(status.HTTP_400_BAD_REQUEST, "The recording was empty.")
language = (_user_audio(user).get("language") or config.get("stt_language") or "").strip()
try:
text = await audio_service.transcribe(
audio_service.endpoint_for(config, "stt"),
data=data,
filename=file.filename or "speech.webm",
content_type=file.content_type or "audio/webm",
model=config.get("stt_model") or "whisper-1",
language=language,
)
except LLMError as exc:
log.info("transcription failed: %s", exc.message)
raise HTTPException(status.HTTP_502_BAD_GATEWAY, exc.message) from exc
return PlainTextResponse(text)
@router.get(
"/speech/{chat_id}/{message_id}",
dependencies=[Depends(require_permission("audio.listen"))],
)
async def speech(db: Db, user: RequiredUser, chat_id: str, message_id: str) -> Response:
"""Read one message aloud."""
config = settings_store.audio(db)
if not config.get("tts_enabled"):
raise HTTPException(
status.HTTP_404_NOT_FOUND, "Read-aloud is not enabled on this instance."
)
message = _owned_message(db, chat_id, message_id, user)
text = speakable_text(message.content)
if not text:
raise HTTPException(status.HTTP_404_NOT_FOUND, "There is nothing to read out.")
try:
media_type, stream = await audio_service.speak(
audio_service.endpoint_for(config, "tts"),
text,
model=config.get("tts_model") or "tts-1",
voice=resolve_voice(config, user),
fmt=config.get("tts_format") or "mp3",
speed=resolve_speed(config, user),
)
except LLMError as exc:
log.info("speech failed: %s", exc.message)
raise HTTPException(status.HTTP_502_BAD_GATEWAY, exc.message) from exc
return StreamingResponse(
stream,
media_type=media_type,
# Not cached: the voice can change under the reader between plays, and
# a message can be regenerated at the same URL.
headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"},
)
async def available_voices(config: dict, *, refresh: bool = False) -> tuple[list[str], str]:
"""Discovered voices and, if discovery failed, why.
Returns rather than raises: a settings page whose voice list could not be
fetched should still render, with the reason next to an empty list.
"""
if not config.get("tts_enabled") or not (config.get("tts_base_url") or "").strip():
return [], ""
try:
return await audio_service.voices(
audio_service.endpoint_for(config, "tts"), refresh=refresh
), ""
except LLMError as exc:
return [], exc.message
def _owned_message(db: DBSession, chat_id: str, message_id: str, user: User) -> Message:
"""The message, if it belongs to a chat this user owns.
404 rather than 403 throughout, matching api/chats.py: whether a given id
exists is not information these endpoints hand out.
"""
chat = db.get(Chat, chat_id)
if chat is None or chat.user_id != user.id:
raise HTTPException(status.HTTP_404_NOT_FOUND, "That chat no longer exists.")
message = db.get(Message, message_id)
if message is None or message.chat_id != chat.id:
raise HTTPException(status.HTTP_404_NOT_FOUND, "That message no longer exists.")
return message