PWA, one send/stop button, audio in and out, web search as a tool

Four pieces of work.

**Installable.** A manifest carrying the instance name, PWA icons rasterised
from the existing mark at design time, a service worker and a themed offline
page. The worker caches the shell only and bails out on /api/, /auth/, /admin/
and anything accepting text/event-stream -- passing a reply stream through a
worker turns it into one delivery at the end, or nothing. It is served from
GET /sw.js rather than the static mount because a worker's scope is the path it
came from.

**Send and Stop are one button.** They were two, and the hidden one was never
hidden: `.btn` is display: inline-flex, which outranks the browser's own
`[hidden] { display: none }`, so Stop sat permanently beside Send. app.css now
forces the attribute to win -- every control toggled with `hidden` depended on
that -- and the composer renders one button carrying both icons, with ui.js
flipping data-composer-action and the type with it.

**Audio.** Speech to text and text to speech against any OpenAI-shaped
/v1/audio/* endpoint: dictate into the composer, have a reply read out.
Instance settings in Admin, per-reader overrides in Settings, with the voice
list discovered from the server where it offers one. Recorded audio is capped
and never written to disk -- it is not an attachment, it has no owner, and
nothing would ever sweep it.

**Web search, as a tool.** This is the tool loop PLAN.md described as the real
work: one reply is now a bounded sequence of requests rather than one. The model
asks, the tool runs, the result goes back and it is asked again, up to three
rounds. Providers are DuckDuckGo (no setup), SearXNG and Firecrawl.

Two decisions worth stating. Tools are only offered to models flagged `tools`,
because an endpoint without support rejects the whole request rather than
ignoring the array -- the same reason images only reach models flagged
`vision`. And tool results are not replayed as context on the next turn, for the
same reasons reasoning is not: the answer already contains what the model made
of them, and replaying stale results into every later request wastes the window
and reliably sends a small model into a search loop. The sources stay visible in
the transcript instead.

Search results are untrusted third-party text and are treated as such: escaped,
and only http/https URLs rendered as links.

338 tests, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

src/lembas/services/search/__init__.py

@@ -0,0 +1,108 @@
+"""Web search providers.
+
+One shape in, one shape out: a query and a limit go in, a list of SearchResult
+comes back, and which service answered is a setting rather than a code path any
+caller has to know about.
+
+Everything here returns *untrusted third-party text*. A title or snippet from a
+search result is exactly as much attacker-controlled as model output, and gets
+the same treatment: escaped on the way into a page, and only http/https URLs
+rendered as links.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from typing import Any
+
+from lembas.services.search import ddg, firecrawl, searxng
+from lembas.services.search.base import SearchError, SearchResult
+
+log = logging.getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class Provider:
+    key: str
+    label: str
+    description: str
+    # Whether an administrator has to configure something before it works.
+    needs_setup: bool
+
+
+PROVIDERS: tuple[Provider, ...] = (
+    Provider(
+        "ddgs",
+        "DuckDuckGo",
+        "No account, no key, no server to run. Rate limited if used heavily.",
+        False,
+    ),
+    Provider(
+        "searxng",
+        "SearXNG",
+        "Your own metasearch instance. Needs its JSON format enabled.",
+        True,
+    ),
+    Provider(
+        "firecrawl",
+        "Firecrawl",
+        "Hosted search API. Needs an account and a key.",
+        True,
+    ),
+)
+
+_RUNNERS = {"ddgs": ddg.search, "searxng": searxng.search, "firecrawl": firecrawl.search}
+
+
+def provider(key: str) -> Provider:
+    return next((p for p in PROVIDERS if p.key == key), PROVIDERS[0])
+
+
+def availability(key: str) -> str:
+    """Why a provider cannot be used, or "" when it can.
+
+    Checked before a search is attempted so the admin screen can say what is
+    wrong while it is being configured, rather than the first chat to try it
+    being where the problem surfaces.
+    """
+    if key == "ddgs" and not ddg.is_available():
+        return (
+            "The ddgs package is not installed. Install it with: "
+            'pip install "lembas[search]"'
+        )
+    return ""
+
+
+async def run(
+    config: dict[str, Any], query: str, *, limit: int | None = None
+) -> list[SearchResult]:
+    """Search with whichever provider is configured.
+
+    Raises SearchError with something worth reading; every provider translates
+    its own failures rather than letting an httpx exception escape.
+    """
+    query = " ".join(query.split())[:400]
+    if not query:
+        raise SearchError("There was nothing to search for.")
+
+    key = config.get("provider") or "ddgs"
+    problem = availability(key)
+    if problem:
+        raise SearchError(problem)
+
+    runner = _RUNNERS.get(key)
+    if runner is None:
+        raise SearchError(f"Unknown search provider '{key}'.")
+
+    count = limit or int(config.get("max_results") or 5)
+    # A model that asks for fifty results is asking for a prompt nobody can
+    # afford; the administrator's number is the ceiling either way.
+    count = min(max(count, 1), int(config.get("max_results") or 5))
+
+    results = await runner(config, query, count)
+    log.info("web search (%s) for %r: %d results", key, query[:60], len(results))
+    return results[:count]
+
+
+__all__ = ["PROVIDERS", "Provider", "SearchError", "SearchResult", "availability", "run"]

src/lembas/services/search/base.py

@@ -0,0 +1,62 @@
+"""What every search provider produces, and how it fails."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+from urllib.parse import urlparse
+
+# A snippet is context, not an article. Longer than this and a handful of
+# results crowds out the conversation they were meant to inform.
+MAX_SNIPPET = 400
+
+
+class SearchError(Exception):
+    """A search failure with a message fit to show a user.
+
+    Same contract as LLMError in the chat client: one exception type, always
+    carrying text that can be put on screen without editing.
+    """
+
+    def __init__(self, message: str) -> None:
+        super().__init__(message)
+        self.message = message
+
+
+@dataclass(frozen=True)
+class SearchResult:
+    title: str
+    url: str
+    snippet: str
+
+    @property
+    def host(self) -> str:
+        try:
+            return urlparse(self.url).netloc or self.url
+        except ValueError:
+            return self.url
+
+    @property
+    def is_linkable(self) -> bool:
+        """Whether this result's URL may be rendered as a link.
+
+        Only http and https. A search provider is an untrusted source, and a
+        javascript: or data: URL arriving in a result and being turned into an
+        anchor is the obvious way this feature would be abused.
+        """
+        try:
+            return urlparse(self.url).scheme in ("http", "https")
+        except ValueError:
+            return False
+
+
+def clean(title: Any, url: Any, snippet: Any) -> SearchResult | None:
+    """Normalise one provider's row, or None if there is nothing usable in it."""
+    url = str(url or "").strip()
+    if not url:
+        return None
+    return SearchResult(
+        title=" ".join(str(title or "").split())[:300] or url,
+        url=url[:2000],
+        snippet=" ".join(str(snippet or "").split())[:MAX_SNIPPET],
+    )

src/lembas/services/search/ddg.py

@@ -0,0 +1,88 @@
+"""DuckDuckGo, via the ddgs package.
+
+The default provider because it is the only one that works with no account, no
+key and no server to run: enabling web search should not also be a
+configuration exercise.
+
+Optional at install time -- see the `search` extra in pyproject.toml -- so the
+import is guarded and its absence is reported as something to install rather
+than as a crash.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+
+from lembas.services.search.base import SearchError, SearchResult, clean
+
+try:  # pragma: no cover - exercised by whether the extra is installed
+    from ddgs import DDGS
+
+    _IMPORT_ERROR = ""
+except ImportError as exc:  # pragma: no cover
+    DDGS = None
+    _IMPORT_ERROR = str(exc)
+
+
+def is_available() -> bool:
+    return DDGS is not None
+
+
+def _blocking_search(query: str, count: int, region: str, safesearch: str) -> list[dict[str, Any]]:
+    with DDGS() as client:
+        return list(
+            client.text(
+                query,
+                region=region or "wt-wt",
+                safesearch=safesearch or "moderate",
+                max_results=count,
+            )
+        )
+
+
+async def search(config: dict[str, Any], query: str, count: int) -> list[SearchResult]:
+    if not is_available():
+        raise SearchError(
+            'The ddgs package is not installed. Install it with: pip install "lembas[search]"'
+        )
+
+    try:
+        # ddgs is synchronous. Run it on a thread: blocking the event loop here
+        # would stall every other chat in the process, including the one that
+        # asked for the search.
+        rows = await asyncio.wait_for(
+            asyncio.to_thread(
+                _blocking_search,
+                query,
+                count,
+                str(config.get("region") or "wt-wt"),
+                str(config.get("safesearch") or "moderate"),
+            ),
+            timeout=float(config.get("timeout") or 20.0),
+        )
+    except TimeoutError as exc:
+        raise SearchError("DuckDuckGo did not answer in time.") from exc
+    except Exception as exc:  # noqa: BLE001 - the library raises its own types
+        # Rate limiting is the common failure and worth naming, because the fix
+        # is to wait rather than to change anything.
+        detail = str(exc)
+        if "ratelimit" in detail.lower() or "202" in detail:
+            raise SearchError(
+                "DuckDuckGo is rate limiting this instance. Try again shortly, "
+                "or configure SearXNG instead."
+            ) from exc
+        raise SearchError(f"DuckDuckGo search failed: {detail[:200]}") from exc
+
+    results = []
+    for row in rows:
+        # ddgs renamed its fields across versions; both spellings are read so
+        # an upgrade does not silently return empty snippets.
+        result = clean(
+            row.get("title"),
+            row.get("href") or row.get("url") or row.get("link"),
+            row.get("body") or row.get("description") or row.get("snippet"),
+        )
+        if result is not None:
+            results.append(result)
+    return results

src/lembas/services/search/firecrawl.py

@@ -0,0 +1,75 @@
+"""Firecrawl's hosted search API.
+
+The paid option, and the only one of the three that needs a key. Included
+because it answers with cleaned page content rather than a search engine's
+snippet, which is materially better material for a model to read.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import httpx
+
+from lembas.services.crypto import decrypt
+from lembas.services.search.base import SearchError, SearchResult, clean
+
+DEFAULT_BASE_URL = "https://api.firecrawl.dev"
+
+
+async def search(config: dict[str, Any], query: str, count: int) -> list[SearchResult]:
+    api_key = decrypt(str(config.get("firecrawl_api_key_encrypted") or ""))
+    if not api_key:
+        raise SearchError("No Firecrawl API key has been configured.")
+
+    base_url = str(config.get("firecrawl_base_url") or DEFAULT_BASE_URL).strip().rstrip("/")
+
+    try:
+        async with httpx.AsyncClient(timeout=float(config.get("timeout") or 20.0)) as client:
+            response = await client.post(
+                f"{base_url}/v1/search",
+                headers={
+                    "Authorization": f"Bearer {api_key}",
+                    "Content-Type": "application/json",
+                },
+                json={"query": query, "limit": count},
+            )
+    except httpx.RequestError as exc:
+        raise SearchError(f"Could not reach {base_url}: {exc}") from exc
+
+    if response.status_code == 401:
+        raise SearchError("Firecrawl rejected the API key.")
+    if response.status_code == 402:
+        raise SearchError("The Firecrawl account is out of credit.")
+
+    try:
+        payload = response.json()
+    except ValueError as exc:
+        raise SearchError(f"Firecrawl returned HTTP {response.status_code}.") from exc
+
+    if response.status_code >= 400 or (
+        isinstance(payload, dict) and payload.get("success") is False
+    ):
+        detail = payload.get("error") if isinstance(payload, dict) else ""
+        raise SearchError(str(detail) or f"Firecrawl returned HTTP {response.status_code}.")
+
+    rows = payload.get("data") if isinstance(payload, dict) else None
+    # Newer responses nest the list under data.web; older ones put it directly
+    # in data. Both are read so an API revision does not empty the results.
+    if isinstance(rows, dict):
+        rows = rows.get("web")
+    if not isinstance(rows, list):
+        raise SearchError("Firecrawl returned a response in an unexpected shape.")
+
+    results = []
+    for row in rows[:count]:
+        if not isinstance(row, dict):
+            continue
+        result = clean(
+            row.get("title"),
+            row.get("url"),
+            row.get("description") or row.get("markdown") or row.get("content"),
+        )
+        if result is not None:
+            results.append(result)
+    return results

src/lembas/services/search/searxng.py

@@ -0,0 +1,72 @@
+"""SearXNG, a self-hosted metasearch instance.
+
+The right answer for anyone already running one: no third party sees the
+queries, and it aggregates several engines. It needs one thing switched on
+first, which a stock install does not have, so that case is detected and named
+rather than reported as "search failed".
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+import httpx
+
+from lembas.services.search.base import SearchError, SearchResult, clean
+
+# What a stock settings.yml is missing. Worth quoting exactly: it is the whole
+# fix, and hunting for it in the documentation takes longer than reading it.
+JSON_DISABLED = (
+    "This SearXNG instance will not answer in JSON. Add \"- json\" under "
+    "search.formats in its settings.yml and restart it."
+)
+
+
+async def search(config: dict[str, Any], query: str, count: int) -> list[SearchResult]:
+    base_url = str(config.get("searxng_base_url") or "").strip().rstrip("/")
+    if not base_url:
+        raise SearchError("No SearXNG instance has been configured.")
+
+    params = {
+        "q": query,
+        "format": "json",
+        "categories": "general",
+        "safesearch": {"off": "0", "moderate": "1", "strict": "2"}.get(
+            str(config.get("safesearch") or "moderate"), "1"
+        ),
+    }
+
+    try:
+        async with httpx.AsyncClient(
+            timeout=float(config.get("timeout") or 20.0), follow_redirects=True
+        ) as client:
+            response = await client.get(f"{base_url}/search", params=params)
+    except httpx.RequestError as exc:
+        raise SearchError(f"Could not reach {base_url}: {exc}") from exc
+
+    # 403 on an otherwise working instance means the JSON format is not in the
+    # allowed list -- SearXNG refuses the format rather than the request.
+    if response.status_code == 403:
+        raise SearchError(JSON_DISABLED)
+    if response.status_code >= 400:
+        raise SearchError(f"{base_url} returned HTTP {response.status_code}.")
+
+    try:
+        payload = response.json()
+    except ValueError as exc:
+        # An HTML page where JSON was asked for is the same misconfiguration
+        # wearing a different status code.
+        raise SearchError(JSON_DISABLED) from exc
+
+    rows = payload.get("results") if isinstance(payload, dict) else None
+    if not isinstance(rows, list):
+        raise SearchError("SearXNG returned a response in an unexpected shape.")
+
+    results = []
+    for row in rows[:count]:
+        if not isinstance(row, dict):
+            continue
+        result = clean(row.get("title"), row.get("url"), row.get("content"))
+        if result is not None:
+            results.append(result)
+    return results