An update you can ask for, and a boundary that stays where it was

The button cannot do the work, and that is the whole design. The service runs as
an unprivileged account, cannot restart itself, and should not be able to: a web
application that can restart its own service is one whose worst day is much
worse. So /admin/updates writes a file, and an opt-in systemd .path unit runs
deploy/update.sh as root.

Three properties hold it up, and each is a thing that could have been got wrong.
The request file carries nothing that reaches a command line -- no branch, no
ref, no arguments -- because the branch is baked into the unit at install time,
so pressing the button is always "deploy the branch this host was configured
with" and can never be "deploy something else". It is off unless somebody passes
INSTALL_UPDATE_HELPER=1, and re-running the installer without it removes both
units and the marker. And without the helper the page says so and prints the
manual command rather than writing a file nothing is watching, which would be a
button that reports success and does nothing.

The card that says all of this is rendered whether or not there is anything to
apply. It was inside the "there is an update" branch first, so an administrator
could not discover the helper was missing until the day they needed it, which is
the worst possible moment.

Opening the page makes no network request; Check is the one thing that fetches.
And it shows the log between, not a count: "3 behind" is a number somebody has to
go and look up, while the subjects are what decides whether this is worth
restarting for right now.

Docker is one stage, because there is nothing to build -- no Node, no compiled
assets. It bakes no secret key (one in an image is one every copy shares, and
rotating it makes stored API keys unreadable), no data, and no .git, so
/admin/updates inside a container correctly reports that it was not installed
from a checkout. Compose publishes on loopback and refuses to start without a
key. TLS in front is a constraint rather than a recommendation: the service
worker and the microphone both require HTTPS or localhost.

The image was built and run before this was committed, which is how the missing
COPY of LICENSE was found -- pyproject declares it and the build backend reads
it, so the failure reads like a packaging problem and is one line.

deploy/lxc-install.sh creates an unprivileged Debian container and runs the
existing installer inside it. A wrapper, not a second install path: a parallel
installer is two things to keep correct and one of them rots.

/healthz opens the database rather than only proving the socket is listening -- a
process that is up with a database it cannot open answers every page with a 500
-- and says nothing about what is here, being reachable without signing in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-06 17:56:18 +02:00
parent 1b8c9f948c
commit ddad585e4b
17 changed files with 1279 additions and 2 deletions
+31
View File
@@ -0,0 +1,31 @@
# What must never reach the image.
#
# The first two blocks are the ones that matter: a `data/` directory copied in
# would bake somebody's database, their uploads and their encrypted API keys
# into an image, and a `.env` would bake the key that decrypts them.
data/
*.db
*.db-wal
*.db-shm
.env
.env.*
lembas.env
# `.git` is excluded and that has a consequence worth knowing: /admin/updates
# reads it to say what is running, so inside a container that page says "not
# installed from a checkout" and offers nothing. That is correct -- a container
# is updated by pulling a new image, not by resetting a checkout inside it.
.git/
.github/
.venv/
venv/
__pycache__/
*.pyc
.pytest_cache/
.ruff_cache/
htmlcov/
.coverage
dist/
build/
*.egg-info/
+69
View File
@@ -0,0 +1,69 @@
# LLeMbas in a container.
#
# One stage, on purpose. There is nothing to build: no Node, no compiled assets,
# no wheel worth producing separately — the vendored browser libraries are
# committed and the templates are read at runtime. A multi-stage build here
# would be ceremony that saves nothing and hides where the files came from.
#
# **This image is not a deployment on its own.** It serves plain HTTP and expects
# a TLS reverse proxy in front, and that is a constraint rather than a
# preference: a service worker and a microphone both require HTTPS or localhost,
# so over plain http on a LAN address the app installs as nothing and cannot
# dictate. See deploy/README.md.
FROM python:3.12-slim
# `bash` and `git` earn their place: `git` is what /admin/updates reads to say
# what is running, and its absence there is reported rather than crashed on.
# `curl` is the healthcheck below. Everything else stays out.
RUN apt-get update \
&& apt-get install --no-install-recommends -y git curl \
&& rm -rf /var/lib/apt/lists/*
# A real account rather than root, and made before the install so the layers it
# owns are its own. 10001 rather than the first free id: a bind-mounted volume
# on the host is easier to reason about when the id is stated.
RUN useradd --create-home --uid 10001 --shell /usr/sbin/nologin lembas
WORKDIR /app
# The dependency install is its own layer, keyed on the files that decide it, so
# editing a template does not re-resolve the whole tree.
#
# LICENSE is in the list because `pyproject.toml` declares `license = { file =
# "LICENSE" }` and the build backend reads it -- without it the install fails
# with "License file does not exist", which reads like a packaging problem and
# is a missing COPY. README.md is there for the same reason (`readme = `).
COPY pyproject.toml README.md LICENSE ./
COPY src/lembas/__init__.py src/lembas/__init__.py
RUN pip install --no-cache-dir -e ".[search,ssh]"
COPY . .
# Again, because the first install ran against a source tree with one file in
# it. Cheap: everything is already resolved and cached above.
RUN pip install --no-cache-dir --no-deps -e "." \
&& chown -R lembas:lembas /app
# The database, the uploads and the encryption at rest all live here. Declared
# so that running without `-v` still works and says where the data went, rather
# than losing it silently at the first `docker rm`.
ENV LEMBAS_DATA_DIR=/data \
LEMBAS_HOST=0.0.0.0 \
LEMBAS_PORT=8080 \
PYTHONUNBUFFERED=1
RUN install -d -o lembas -g lembas /data
VOLUME ["/data"]
# **No secret key is baked in.** One in an image is one every copy of the image
# shares, and rotating it signs everybody out *and* makes stored upstream API
# keys unreadable. Without LEMBAS_SECRET_KEY the app generates a temporary one
# and warns loudly at startup, which is the right failure: it works for a look
# and cannot be mistaken for a deployment.
USER lembas
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
CMD curl -fsS http://127.0.0.1:8080/healthz || exit 1
CMD ["lembas", "serve"]
+70 -1
View File
@@ -45,7 +45,37 @@ Everything is overridable from the environment:
| `HOME_DIR` | `/home/lembas` | that account's home |
| `PREFIX` | `/srv/lembas` | install root (bind mount of `HOME_DIR`) |
| `REPO_URL` | this checkout's `origin` | so a fork deploys itself |
| `LEMBAS_BRANCH` | `main` | branch to deploy |
| `LEMBAS_BRANCH` | `main` | branch to deploy, and what `/admin/updates` compares against |
| `INSTALL_UPDATE_HELPER` | `0` | `1` lets the web interface deploy that branch as root |
## Updating from the web interface
`/admin/updates` says what is running, what is on the branch and what changed
between. **Checking** reaches the remote; opening the page does not.
The button is opt-in, and the reason is a boundary rather than caution:
```bash
INSTALL_UPDATE_HELPER=1 SITE_HOST=chat.example ./deploy/install.sh
```
That installs `lembas-update.path` and `lembas-update.service`. The web
interface writes `$PREFIX/data/update-requested`; the path unit notices and the
service runs `update.sh` **as root**.
**What that grants.** Anybody who can administer this web interface can then
deploy whatever is on the configured branch and restart the service. That is the
point of it, and it is why it is not the default.
**What it deliberately does not grant.** The request file carries nothing that
reaches a command line — no branch, no ref, no arguments. The branch is baked
into the unit at install time from `LEMBAS_BRANCH`, so the button is always
"deploy the branch this host was configured with" and never "deploy something
else". Re-running the installer without the flag removes both units and the
marker, and the page goes back to printing the manual command.
Without the helper the page says so and shows `sudo …/deploy/update.sh`, which is
the same honest degradation the SSH and search extras have.
## Deploying a change
@@ -59,6 +89,45 @@ reinstalls dependencies and restarts, printing the commits it pulled. The hard
reset is deliberate: nothing is ever edited in place there, so there is no local
work to preserve and no conflicts to resolve.
## In a container
A `Dockerfile` and a `docker-compose.yml` are in the repository root.
```bash
echo "LEMBAS_SECRET_KEY=$(python -c 'import secrets;print(secrets.token_urlsafe(48))')" > .env
docker compose up -d
```
It publishes on `127.0.0.1:8080` and expects **a TLS reverse proxy in front**.
That is a constraint, not a preference: a service worker and a microphone both
require HTTPS or localhost, so over plain http on a LAN address the app cannot be
installed and cannot dictate — and the session cookie is deliberately not marked
`secure`, so an attacker on that network could steal a session.
Three things about the image:
- **No secret key is baked in**, and compose refuses to start without one. A key
in an image is a key every copy of that image shares, and rotating it signs
everybody out *and* makes stored upstream API keys unreadable.
- **`.git` is excluded**, so `/admin/updates` inside a container says it was not
installed from a checkout and offers nothing. That is correct: a container is
updated by pulling a new image.
- **One replica.** The generation registry, the stop mechanism, the terminal
sessions and the schedule ticker are all in-process — two would mean two
tickers and every schedule firing twice.
## On Proxmox
```bash
CTID=140 SITE_HOST=chat.example ./deploy/lxc-install.sh
```
Run on the Proxmox host. It creates an **unprivileged** Debian container,
installs the dependencies, and runs `deploy/install.sh` inside it — the same
installer, so a fix there reaches this without anybody remembering. Unprivileged
is not a default to change: nothing LLeMbas does needs privilege, because agent
chats run their commands over SSH on some *other* machine.
## Operating it
```bash
+44
View File
@@ -26,6 +26,11 @@ SERVICE_USER="${SERVICE_USER:-lembas}"
HOME_DIR="${HOME_DIR:-/home/lembas}"
PREFIX="${PREFIX:-/srv/lembas}"
BRANCH="${LEMBAS_BRANCH:-main}"
# Whether to install the units that let the web interface update this host.
# Off, and off on a re-run that does not ask for it: it grants anybody who can
# administer the web UI the ability to deploy the branch, as root. See the
# "Updating from the web interface" section of deploy/README.md.
INSTALL_UPDATE_HELPER="${INSTALL_UPDATE_HELPER:-0}"
# Default to wherever this checkout came from, so a fork deploys itself.
REPO_URL="${REPO_URL:-$(git -C "$HERE" remote get-url origin 2>/dev/null || true)}"
@@ -43,6 +48,11 @@ echo " host : https://$SITE_HOST -> 127.0.0.1:$APP_PORT"
echo " user : $SERVICE_USER ($HOME_DIR)"
echo " prefix : $PREFIX"
echo " repo : $REPO_URL ($BRANCH)"
if [[ "$INSTALL_UPDATE_HELPER" == "1" ]]; then
echo " updates : web interface may deploy $BRANCH as root (helper units)"
else
echo " updates : by hand only ($PREFIX/app/deploy/update.sh)"
fi
echo "== service user =="
# --system: no ageing, no mail spool. Home under /home, not /var/lib, so the
@@ -101,6 +111,11 @@ LEMBAS_PORT=$APP_PORT
LEMBAS_LOG_LEVEL=info
LEMBAS_ALLOW_SIGNUP=true
LEMBAS_DEFAULT_THEME=moria
# Which branch /admin/updates compares against. Deployment configuration, not
# an instance setting: it decides what code runs here, and a value a web
# administrator could edit would turn "you may deploy the branch" into "you may
# deploy anything".
LEMBAS_UPDATE_BRANCH=$BRANCH
EOF
sudo chown "$SERVICE_USER:$SERVICE_USER" "$ENV_FILE"
sudo chmod 600 "$ENV_FILE"
@@ -120,6 +135,35 @@ sed -e "s|__PREFIX__|$PREFIX|g" -e "s|__SERVICE_USER__|$SERVICE_USER|g" \
sha256sum "$HERE/lembas.service" | cut -d' ' -f1 | sudo tee "$PREFIX/.unit-applied" >/dev/null
sudo systemctl daemon-reload
echo "== update helper =="
# Two units and a marker. The marker is what the web interface reads to decide
# whether to offer the button at all -- a file rather than `systemctl
# is-enabled`, because that would be a subprocess on every page render to answer
# a question that changes once.
UPDATE_MARKER="$PREFIX/data/.update-helper"
if [[ "$INSTALL_UPDATE_HELPER" == "1" ]]; then
for unit in lembas-update.path lembas-update.service; do
sed -e "s|__PREFIX__|$PREFIX|g" \
-e "s|__SERVICE_USER__|$SERVICE_USER|g" \
-e "s|__UPDATE_BRANCH__|$BRANCH|g" \
"$HERE/$unit" | sudo tee "/etc/systemd/system/$unit" >/dev/null
done
sudo systemctl daemon-reload
sudo systemctl enable --now lembas-update.path
sudo touch "$UPDATE_MARKER"
sudo chown "$SERVICE_USER:$SERVICE_USER" "$UPDATE_MARKER"
echo " installed. The web interface can now deploy $BRANCH and restart."
else
# Removed rather than left, so turning it off is re-running without the flag
# rather than remembering three commands. The button then says so and prints
# the manual one, which is the honest degradation.
sudo systemctl disable --now lembas-update.path 2>/dev/null || true
sudo rm -f /etc/systemd/system/lembas-update.path \
/etc/systemd/system/lembas-update.service "$UPDATE_MARKER"
sudo systemctl daemon-reload
echo " not installed (INSTALL_UPDATE_HELPER=1 to allow updating from the web UI)"
fi
echo "== self-signed cert for $SITE_HOST =="
sudo mkdir -p /etc/nginx/ssl
if [[ ! -f "/etc/nginx/ssl/$SITE_HOST.crt" ]]; then
+23
View File
@@ -0,0 +1,23 @@
# Watches for an update request written by the web interface.
#
# install.sh substitutes __PREFIX__ and writes the result to
# /etc/systemd/system/lembas-update.path. Installed only when the installer is
# run with INSTALL_UPDATE_HELPER=1 — see deploy/README.md for what that decision
# means.
#
# `PathExists` rather than `PathChanged`: the service deletes the file as its
# first act, so the unit re-arms itself and a second request fires again. With
# `PathChanged` a request written while the service was running would be missed.
[Unit]
Description=Watch for a LLeMbas update request
# Only while the thing being updated is meant to be running. Stopping lembas on
# purpose should not leave a watcher that restarts it.
PartOf=lembas.service
[Path]
PathExists=__PREFIX__/data/update-requested
Unit=lembas-update.service
[Install]
WantedBy=multi-user.target
+37
View File
@@ -0,0 +1,37 @@
# Runs deploy/update.sh when the web interface asks for it.
#
# install.sh substitutes __PREFIX__, __SERVICE_USER__ and __UPDATE_BRANCH__ and
# writes the result to /etc/systemd/system/lembas-update.service.
#
# **What this grants.** Installing it means anybody who can administer the web
# interface can deploy whatever is on the configured branch, as root, and
# restart the service. That is the point of it, and it is why it is opt-in and
# why the installer says so out loud rather than doing it by default.
#
# **What it deliberately does not grant.** The request file carries nothing that
# reaches this command line: no branch, no ref, no arguments. The branch is
# baked in below, from the installer's environment, so pressing the button is
# "deploy the branch this host was configured with" and can never be "deploy
# something else".
[Unit]
Description=Apply a requested LLeMbas update
# Not `After=lembas.service`: this restarts it, and an ordering dependency on
# the thing being restarted is how a one-shot ends up waiting for itself.
[Service]
Type=oneshot
# Deleted first, always. The path unit re-arms on the file existing, so leaving
# it in place would run this again the moment the service came back -- an
# update loop with no obvious cause. `-` so a failure to delete does not stop
# the update, and `ExecStartPre` so it happens even if the script itself fails.
ExecStartPre=-/usr/bin/rm -f __PREFIX__/data/update-requested
Environment=SERVICE_USER=__SERVICE_USER__
Environment=PREFIX=__PREFIX__
Environment=LEMBAS_BRANCH=__UPDATE_BRANCH__
ExecStart=/bin/bash __PREFIX__/app/deploy/update.sh
# The script's own failure path prints the journal and exits non-zero, which is
# what makes `systemctl status lembas-update` say what went wrong.
StandardOutput=journal
StandardError=journal
TimeoutStartSec=600
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env bash
# Create a Debian LXC container on a Proxmox host and install LLeMbas in it.
#
# A **wrapper around what already works**, not a second install path. It makes a
# container, puts the dependencies in it, and runs `deploy/install.sh` inside --
# which is the same script, doing the same things, so a fix to the installer
# reaches this without anybody remembering. A parallel installer would be two
# things to keep correct and one of them would rot.
#
# Run this on the Proxmox host, as root:
#
# CTID=140 SITE_HOST=chat.example ./deploy/lxc-install.sh
#
# Everything is overridable:
#
# CTID next free id the container's id
# CT_HOSTNAME lembas hostname inside it
# CT_STORAGE local-lvm where the rootfs goes
# CT_TEMPLATE debian-12 template, matched against pveam list
# CT_DISK 12 GB
# CT_CORES 2
# CT_MEMORY 4096 MB
# CT_BRIDGE vmbr0
# CT_IP dhcp or 192.168.1.50/24
# CT_GATEWAY (unset) required when CT_IP is static
# REPO_URL this checkout's origin
# SITE_HOST lembas.local
#
# **Unprivileged, and that is not a default to change lightly.** Nothing LLeMbas
# does needs privilege: agent chats run their commands over SSH on some *other*
# machine, which is the whole isolation story. A privileged container would give
# up the host's protection to buy nothing.
set -euo pipefail
CT_HOSTNAME="${CT_HOSTNAME:-lembas}"
CT_STORAGE="${CT_STORAGE:-local-lvm}"
CT_TEMPLATE="${CT_TEMPLATE:-debian-12}"
CT_DISK="${CT_DISK:-12}"
CT_CORES="${CT_CORES:-2}"
CT_MEMORY="${CT_MEMORY:-4096}"
CT_BRIDGE="${CT_BRIDGE:-vmbr0}"
CT_IP="${CT_IP:-dhcp}"
CT_GATEWAY="${CT_GATEWAY:-}"
SITE_HOST="${SITE_HOST:-lembas.local}"
BRANCH="${LEMBAS_BRANCH:-main}"
HERE="$(dirname "$(readlink -f "$0")")"
REPO_URL="${REPO_URL:-$(git -C "$HERE" remote get-url origin 2>/dev/null || true)}"
if ! command -v pct >/dev/null; then
echo "pct not found. Run this on a Proxmox host." >&2
exit 1
fi
if [[ -z "$REPO_URL" ]]; then
echo "Could not determine REPO_URL. Set it explicitly." >&2
exit 1
fi
CTID="${CTID:-$(pvesh get /cluster/nextid)}"
# The template has to be on the host before a container can be made from it.
# Matched by prefix rather than pinned to a filename, because the point release
# in it moves and a hard-coded name would break on a host that downloaded a
# different one.
echo "== template =="
template=$(pveam list local 2>/dev/null | awk -v want="$CT_TEMPLATE" '$1 ~ want {print $1}' | head -1)
if [[ -z "$template" ]]; then
available=$(pveam available --section system | awk -v want="$CT_TEMPLATE" '$2 ~ want {print $2}' | tail -1)
if [[ -z "$available" ]]; then
echo "No template matching '$CT_TEMPLATE'. Try: pveam available --section system" >&2
exit 1
fi
echo " downloading $available"
pveam download local "$available"
template="local:vztmpl/$available"
fi
echo " $template"
echo "== container $CTID =="
if pct status "$CTID" >/dev/null 2>&1; then
echo " $CTID already exists, using it"
else
net="name=eth0,bridge=$CT_BRIDGE,ip=$CT_IP"
[[ -n "$CT_GATEWAY" ]] && net="$net,gw=$CT_GATEWAY"
pct create "$CTID" "$template" \
--hostname "$CT_HOSTNAME" \
--cores "$CT_CORES" \
--memory "$CT_MEMORY" \
--rootfs "$CT_STORAGE:$CT_DISK" \
--net0 "$net" \
--unprivileged 1 \
--features nesting=1 \
--onboot 1
echo " created"
fi
pct start "$CTID" 2>/dev/null || true
# `pct exec` returns before the container's own network is up, and the very next
# thing this does is apt-get. Waiting on DNS resolving rather than on a fixed
# sleep, because a fixed sleep is either too short on a slow host or wasted on a
# fast one.
echo "== waiting for the network =="
for _ in $(seq 1 30); do
pct exec "$CTID" -- getent hosts deb.debian.org >/dev/null 2>&1 && break
sleep 2
done
echo "== dependencies =="
pct exec "$CTID" -- bash -lc '
set -e
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq --no-install-recommends \
git python3 python3-venv python3-pip nginx openssl sudo ca-certificates
'
echo "== checkout =="
pct exec "$CTID" -- bash -lc "
set -e
rm -rf /tmp/lembas-src
git clone --quiet --branch '$BRANCH' '$REPO_URL' /tmp/lembas-src
"
# The same installer this repository ships, run inside. Everything it decides --
# the service user, the prefix, the unit, the vhost, the self-signed certificate
# -- it decides there, so this script has no opinions to keep in step with it.
echo "== install =="
pct exec "$CTID" -- bash -lc "
set -e
SITE_HOST='$SITE_HOST' LEMBAS_BRANCH='$BRANCH' REPO_URL='$REPO_URL' \
bash /tmp/lembas-src/deploy/install.sh
"
address=$(pct exec "$CTID" -- hostname -I 2>/dev/null | awk '{print $1}')
echo
echo "LLeMbas is installed in container $CTID."
echo " address : ${address:-unknown}"
echo " site : https://$SITE_HOST (self-signed; accept the warning)"
echo
echo "Point '$SITE_HOST' at ${address:-the container} in your DNS or hosts file,"
echo "then create the first account -- it becomes the administrator."
+57
View File
@@ -0,0 +1,57 @@
# LLeMbas, and nothing else.
#
# Deliberately no reverse proxy in here. Which one to use, where the certificate
# comes from and what else the host already serves are all decisions this file
# cannot make -- and baking one in would mean anybody who already runs Caddy or
# Traefik has to unpick it first. What this does is publish on loopback, which is
# what a proxy on the same host proxies to.
#
# **TLS is not optional in practice.** The service worker and the microphone both
# require HTTPS or localhost, so over plain http on a LAN address the app cannot
# be installed and cannot dictate. See deploy/README.md.
services:
lembas:
build: .
image: lembas:latest
restart: unless-stopped
environment:
# Generate once and keep it: rotating this signs every user out *and*
# makes stored upstream API keys unreadable, because they are encrypted
# with it. `lembas secret-key` prints one.
#
# Required with no default on purpose. A compose file with a key in it is
# a key in everybody's git history, and one that quietly generated a
# temporary one would lose every stored credential on the next restart.
LEMBAS_SECRET_KEY: ${LEMBAS_SECRET_KEY:?set LEMBAS_SECRET_KEY in .env}
LEMBAS_DATA_DIR: /data
LEMBAS_HOST: 0.0.0.0
LEMBAS_PORT: 8080
LEMBAS_LOG_LEVEL: ${LEMBAS_LOG_LEVEL:-info}
LEMBAS_ALLOW_SIGNUP: ${LEMBAS_ALLOW_SIGNUP:-true}
# 127.0.0.1 rather than 0.0.0.0: the session cookie is deliberately not
# marked `secure` so a localhost install can sign anybody in at all, which
# means a network attacker on plain http could steal a session. Publishing
# this on a LAN interface without a proxy in front is the one configuration
# that turns that from a note into a problem.
ports:
- "127.0.0.1:8080:8080"
volumes:
# The database, the uploads, the encryption at rest. A named volume rather
# than a bind mount so it survives `docker compose down` -- `down -v` is
# the command that deletes it, and that asymmetry is the point.
- lembas-data:/data
# One worker, and that is not a shortcut. The generation registry, the stop
# mechanism, the terminal sessions and the schedule ticker are all
# in-process; two of these would mean two tickers and every schedule firing
# twice. Scaling this service is not supported -- see PLAN.md's first known
# limit.
deploy:
replicas: 1
volumes:
lembas-data:
+1 -1
View File
@@ -1,3 +1,3 @@
"""LLeMbas - a Middle-earth themed web UI for OpenAI-compatible LLM endpoints."""
__version__ = "0.9.6"
__version__ = "0.9.7"
+80
View File
@@ -0,0 +1,80 @@
"""What is running here, and getting to what is not.
Read `services/updates.py` first — the reason the button writes a file rather
than doing the work is there, and it is the whole design.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Request, Response, status
from fastapi.responses import RedirectResponse
from lembas.api.deps import AdminUser, Db
from lembas.services import updates as updates_service
from lembas.web.templating import render
log = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/updates", tags=["admin-updates"])
def _page(request: Request, state, saved: str = "") -> Response:
return render(
request,
"admin/updates.html",
{
"state": state,
"command": updates_service.manual_command(),
"saved": saved,
},
)
@router.get("")
async def updates_page(request: Request, db: Db, user: AdminUser, saved: str = ""):
"""No network on a page load.
`read(fetch=False)` compares against whatever the last fetch left behind, so
opening this is a few git reads off the local disk. A page that reached the
remote every time it was rendered would be one somebody stops opening.
"""
return _page(request, updates_service.read(), saved)
@router.post("/check")
async def check(request: Request, db: Db, user: AdminUser) -> Response:
"""Ask the remote what is there. The one place this touches the network."""
state = updates_service.read(fetch=True)
log.info("%s checked for updates", user.email)
return _page(request, state)
@router.post("/apply")
async def apply(db: Db, user: AdminUser) -> Response:
"""Write the request the helper is watching for.
Refused when the helper is not installed rather than written and left to sit
there: a file nothing is watching is a button that reports success and does
nothing, which is the failure this codebase keeps cataloguing.
"""
if not updates_service.helper_installed():
return RedirectResponse(
"/admin/updates?saved=The+update+helper+is+not+installed+on+this+host.",
status_code=status.HTTP_303_SEE_OTHER,
)
problem = updates_service.request_update(user.email)
message = problem or "Update requested. The service will restart in a moment."
return RedirectResponse(
f"/admin/updates?saved={message.replace(' ', '+')}",
status_code=status.HTTP_303_SEE_OTHER,
)
@router.post("/cancel")
async def cancel(db: Db, user: AdminUser) -> Response:
updates_service.clear_request()
return RedirectResponse(
"/admin/updates?saved=Request+withdrawn.", status_code=status.HTTP_303_SEE_OTHER
)
+26
View File
@@ -435,6 +435,32 @@ async def home(user: RequiredUser):
# has by definition no server to ask who is looking at it.
@router.get("/healthz", include_in_schema=False)
async def healthz() -> Response:
"""Is the process up and can it reach its database.
Unauthenticated, like the three below, and for a fourth reason: a
healthcheck that needed a session would be a healthcheck nothing could run.
It says nothing about *what* is here -- no version, no counts -- because it
is reachable without signing in and a health endpoint is a common place to
leak the first fact an attacker wants.
The query is what makes it worth having. A process that is up with a
database it cannot open answers every page with a 500, and a check that only
proved the socket was listening would call that healthy.
"""
from sqlalchemy import text
from lembas.db.session import session_scope
try:
with session_scope() as db:
db.execute(text("SELECT 1"))
except Exception: # noqa: BLE001 - the answer is the status code
return JSONResponse({"status": "error"}, status_code=503)
return JSONResponse({"status": "ok"})
@router.get("/manifest.webmanifest", include_in_schema=False)
async def manifest(db: Db) -> Response:
"""The web app manifest.
+7
View File
@@ -38,6 +38,13 @@ class Settings(BaseSettings):
session_ttl: int = 60 * 60 * 24 * 30
request_timeout: float = 300.0
# Which branch `/admin/updates` compares against and the helper deploys.
# Deployment configuration and deliberately not an instance setting: it
# decides what code runs on this machine, and a value a web administrator
# could edit would turn "you may deploy the branch" into "you may deploy
# anything". `deploy/install.sh` writes it beside the rest.
update_branch: str = "main"
@model_validator(mode="after")
def _generate_secret_if_absent(self) -> Settings:
# A generated key lets `lembas serve` work with no configuration at all,
+2
View File
@@ -25,6 +25,7 @@ from lembas.api import (
admin_search,
admin_suggestions,
admin_tools,
admin_updates,
admin_users,
agents,
audio,
@@ -198,6 +199,7 @@ def create_app() -> FastAPI:
app.include_router(sharing.router)
app.include_router(admin.router)
app.include_router(admin_users.router)
app.include_router(admin_updates.router)
app.include_router(admin_models.router)
app.include_router(admin_audio.router)
app.include_router(admin_branding.router)
+273
View File
@@ -0,0 +1,273 @@
"""What is running here, what is available, and how to get from one to the other.
## Why this exists
Updating meant a shell on the box. That is fine for whoever built it and wrong
for everybody else: an instance somebody installed from `deploy/install.sh` and
handed over has an administrator who can configure every part of it and cannot
tell whether it is three releases behind.
So this answers three questions what is running, what is on the branch, and
what changed between and offers a button for the fourth.
## The button cannot do the work, and that is the design
The service runs as an unprivileged account. It cannot restart itself, it cannot
write outside its own prefix, and it should not be able to: a web application
that can restart its own service is one whose worst day is much worse.
So the button **writes a file**, and an opt-in systemd `.path` unit watching that
file runs `deploy/update.sh` as root. Three things follow, and all three are
deliberate:
- **The request file carries nothing.** No branch, no ref, no arguments. The
helper runs the update script with the branch *it* was configured with, so an
administrator of the web UI cannot choose what gets deployed only *that* the
configured branch gets deployed.
- **It is opt-in.** Without the helper the page says so and shows the command,
which is the honest degradation and the same shape the SSH and search extras
already have. Installing it is a decision with a consequence worth stating:
anybody who can administer the web UI can then deploy whatever is on that
branch.
- **Nothing here runs a shell.** Every subprocess is a fixed argv with
`shell=False`, and the only variable in any of them is the branch, which comes
from the environment rather than from a request.
## On "nothing executes on this machine"
That rule is about *agent chats* a model's commands, on a host somebody chose.
This is an administrator pressing a button to run `git`, with no model anywhere
near it and no input from a request in the argv. Worth stating rather than
leaving somebody to notice the tension.
"""
from __future__ import annotations
import logging
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
from lembas import __version__
from lembas.config import settings
log = logging.getLogger(__name__)
# Long enough for a fetch over a slow link, short enough that a page does not
# appear to hang. A fetch that takes longer than this is a network problem, and
# saying so is more use than waiting.
GIT_TIMEOUT = 30.0
# The file the button writes. Inside the data directory, which is the one place
# the service account can write, and named so that finding it in a backup says
# what it was.
REQUEST_NAME = "update-requested"
# Written by `install.sh` when the helper is installed. A marker rather than
# asking systemd, because `systemctl is-enabled` means a subprocess on every
# page render to answer a question that changes once.
MARKER_NAME = ".update-helper"
@dataclass(frozen=True)
class Commit:
sha: str
subject: str
when: str = ""
@property
def short(self) -> str:
return self.sha[:7]
@dataclass(frozen=True)
class State:
"""Everything the page shows, resolved in one go."""
version: str = __version__
checkout: str = ""
branch: str = "main"
head: Commit | None = None
remote: Commit | None = None
behind: list[Commit] = field(default_factory=list)
dirty: bool = False
helper: bool = False
requested: bool = False
error: str = ""
@property
def is_git(self) -> bool:
return bool(self.checkout)
@property
def up_to_date(self) -> bool:
return bool(self.head and self.remote and self.head.sha == self.remote.sha)
def checkout_dir() -> Path | None:
"""The git checkout this is running from, or None.
`src/lembas/__init__.py` upwards twice is the repository root for an editable
install, which is what every deployment here is. A wheel installed into
site-packages has no `.git` above it and answers None, which the page reads
as "this was not installed from a checkout" rather than as an error.
"""
root = Path(__file__).resolve().parents[3]
return root if (root / ".git").exists() else None
def _git(args: list[str], *, cwd: Path, timeout: float = GIT_TIMEOUT) -> tuple[int, str]:
"""One git command, with no shell anywhere near it.
`shell=False` and a fixed argv, so nothing here can be made to run something
else by what it is given. Returns the code and the output together because
every caller wants both and none of them wants an exception: git failing is
a thing to report on the page, not a 500.
"""
try:
done = subprocess.run( # noqa: S603 - fixed argv, no shell, admin-only
["git", *args],
cwd=str(cwd),
capture_output=True,
text=True,
timeout=timeout,
check=False,
)
except FileNotFoundError:
return 127, "git is not installed on this machine."
except subprocess.TimeoutExpired:
return 124, f"git took longer than {int(timeout)}s."
except Exception as exc: # noqa: BLE001 - a broken git is a message, not a crash
log.debug("git %s failed", args, exc_info=True)
return 1, str(exc)
return done.returncode, (done.stdout or done.stderr or "").strip()
def _commit(line: str) -> Commit | None:
"""One line of `--format=%H\\x1f%s\\x1f%cI`.
Split on a unit separator rather than a space: a commit subject contains
spaces, and every other separator anybody reaches for is one a subject may
also contain.
"""
parts = line.split("\x1f")
if len(parts) < 2 or not parts[0]:
return None
return Commit(sha=parts[0], subject=parts[1], when=parts[2] if len(parts) > 2 else "")
FORMAT = "--format=%H\x1f%s\x1f%cI"
def branch_name() -> str:
return settings.update_branch or "main"
def helper_installed() -> bool:
return (settings.data_dir / MARKER_NAME).exists()
def request_path() -> Path:
return settings.data_dir / REQUEST_NAME
def pending() -> bool:
return request_path().exists()
def read(*, fetch: bool = False) -> State:
"""What is running and what is available.
`fetch` is the network half and is off by default: this is called to render a
page, and a page that made a network request on every load would be one
somebody stops opening. The Check button passes it.
"""
branch = branch_name()
root = checkout_dir()
base = State(branch=branch, helper=helper_installed(), requested=pending())
if root is None:
return base
code, head_line = _git(["log", "-1", FORMAT], cwd=root)
if code != 0:
return State(**{**base.__dict__, "checkout": str(root), "error": head_line})
code, status = _git(["status", "--porcelain"], cwd=root)
dirty = bool(status.strip()) if code == 0 else False
error = ""
if fetch:
code, output = _git(["fetch", "--quiet", "origin", branch], cwd=root)
if code != 0:
error = f"Could not reach the remote: {output}"
remote_line = ""
code, output = _git(["log", "-1", FORMAT, f"origin/{branch}"], cwd=root)
if code == 0:
remote_line = output
behind: list[Commit] = []
if remote_line:
code, output = _git(["log", FORMAT, f"HEAD..origin/{branch}"], cwd=root)
if code == 0 and output:
behind = [c for line in output.splitlines() if (c := _commit(line))]
return State(
checkout=str(root),
branch=branch,
head=_commit(head_line),
remote=_commit(remote_line) if remote_line else None,
behind=behind,
dirty=dirty,
helper=helper_installed(),
requested=pending(),
error=error,
)
def request_update(who: str = "") -> str:
"""Ask the helper to update. Returns a message fit to show.
The file is written with the requester's email in it **for the log**, and
that is all it carries. Nothing in it reaches the helper's command line: the
helper runs the update script with its own configured branch, so being able
to press this button is not being able to choose what is deployed.
"""
path = request_path()
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(f"requested by {who or 'an administrator'}\n", encoding="utf-8")
except OSError as exc:
log.warning("could not write the update request: %s", exc)
return f"Could not write {path}: {exc}"
log.info("update requested by %s", who or "an administrator")
return ""
def clear_request() -> None:
"""Forget a request. For the page's Cancel, and for tests."""
request_path().unlink(missing_ok=True)
def manual_command() -> str:
"""What to run by hand when the helper is not installed."""
root = checkout_dir()
return f"sudo {root}/deploy/update.sh" if root else "sudo ./deploy/update.sh"
__all__ = [
"GIT_TIMEOUT",
"MARKER_NAME",
"REQUEST_NAME",
"Commit",
"State",
"branch_name",
"checkout_dir",
"clear_request",
"helper_installed",
"manual_command",
"pending",
"read",
"request_path",
"request_update",
]
@@ -86,6 +86,10 @@
{{ icon("user", "icon--sm") }}
<span class="nav-item__label">Users</span>
</a>
<a class="nav-item {{ 'is-active' if section == 'updates' }}" href="/admin/updates">
{{ icon("refresh", "icon--sm") }}
<span class="nav-item__label">Updates</span>
</a>
<a class="nav-item {{ 'is-active' if section == 'groups' }}" href="/admin/groups">
{{ icon("users", "icon--sm") }}
<span class="nav-item__label">Groups &amp; permissions</span>
+175
View File
@@ -0,0 +1,175 @@
{% extends "admin/_layout.html" %}
{% from "_macros.html" import icon %}
{% set section = "updates" %}
{% block title %}Updates - {{ brand.name }}{% endblock %}
{% block heading %}Updates{% endblock %}
{% block admin_content %}
<p class="admin-lede">
What is running here, what is on <code>{{ state.branch }}</code>, and what
changed between. Checking reaches the remote; opening this page does not.
</p>
{% if saved %}
<div class="alert alert--success">{{ icon("check", "icon--sm") }} <span>{{ saved }}</span></div>
{% endif %}
{% if state.error %}
<div class="alert alert--error">{{ icon("warning", "icon--sm") }} <span>{{ state.error }}</span></div>
{% endif %}
<section class="card">
<h2 class="card__title">Running</h2>
<dl class="mode-list">
<div class="mode-list__row">
<dt><strong>Version</strong></dt>
<dd>{{ state.version }}</dd>
</div>
{% if state.head %}
<div class="mode-list__row">
<dt><strong>Commit</strong></dt>
<dd>
<code>{{ state.head.short }}</code> {{ state.head.subject }}
{% if state.dirty %}
<span class="badge">uncommitted changes</span>
{% endif %}
</dd>
</div>
{% endif %}
{% if state.checkout %}
<div class="mode-list__row">
<dt><strong>Checkout</strong></dt>
<dd><code>{{ state.checkout }}</code></dd>
</div>
{% endif %}
</dl>
{% if not state.is_git %}
<p class="field__hint">
This was not installed from a git checkout — a container image, or a wheel —
so there is nothing here to compare or update. Update the image instead.
</p>
{% else %}
<form method="post" action="/admin/updates/check" class="btn-row">
<button class="btn" type="submit">
{{ icon("globe", "icon--sm") }} Check the remote
</button>
</form>
{% endif %}
</section>
{% if state.is_git %}
<section class="card">
<h2 class="card__title">
Available
{% if state.up_to_date %}<span class="badge badge--leaf">up to date</span>
{% elif state.behind %}<span class="badge">{{ state.behind | length }} behind</span>
{% endif %}
</h2>
{% if not state.remote %}
<p class="field__hint">
Nothing known about <code>origin/{{ state.branch }}</code> yet. Check the
remote above.
</p>
{% elif state.up_to_date %}
<p class="field__hint">
<code>{{ state.remote.short }}</code> is what is running.
</p>
{% else %}
<p class="card__lede">
<code>{{ state.remote.short }}</code> {{ state.remote.subject }}
</p>
{#
The log between, not just a count. "3 behind" is a number somebody has to go
and look up; the subjects are the thing that decides whether this is worth
restarting for right now.
#}
<ul class="model-list">
{% for commit in state.behind %}
<li class="model-list__item">
<div style="min-width: 0">
<code class="text-xs">{{ commit.short }}</code> {{ commit.subject }}
</div>
</li>
{% endfor %}
</ul>
{% if state.dirty %}
<div class="alert alert--warning">
{{ icon("warning", "icon--sm") }}
<span>
This checkout has uncommitted changes. The update does a hard reset, which
would throw them away — nothing here is meant to be edited in place, so
this usually means somebody was debugging on the box.
</span>
</div>
{% endif %}
{% endif %}
</section>
{#
Its own card, and rendered whether or not there is anything to apply. It used
to live inside the "there is an update" branch, so an administrator could not
find out that the helper was missing until the day they needed it -- which is
the worst moment to discover a thing has to be installed from a shell.
#}
<section class="card">
<h2 class="card__title">Applying an update</h2>
{% if state.requested %}
<div class="alert">
{{ icon("clock", "icon--sm") }}
<span>
An update has been requested and is waiting for the helper to pick it up.
The service restarts when it does.
</span>
</div>
<form method="post" action="/admin/updates/cancel" class="btn-row">
<button class="btn btn--sm" type="submit">Withdraw the request</button>
</form>
{% elif state.helper %}
<p class="card__lede">
This host has the update helper installed, so the button below writes a
request that a systemd unit picks up and runs as root. It always deploys
<code>{{ state.branch }}</code> — the request carries no branch and no ref,
so pressing it is never “deploy something else”.
</p>
<form method="post" action="/admin/updates/apply" class="btn-row"
data-confirm="Update and restart? Replies being written are saved; open terminals are cut off.">
<button class="btn btn--primary" type="submit" {{ 'disabled' if state.up_to_date }}>
{{ icon("sparkle", "icon--sm") }}
{{ "Nothing to apply" if state.up_to_date else "Update and restart" }}
</button>
</form>
<p class="field__hint">
Every restart ends every open terminal session — a command still running on
the far side is cut off. A reply being written is saved with whatever it has.
</p>
{% else %}
<div class="alert">
{{ icon("shield", "icon--sm") }}
<span>
The update helper is <strong>not installed on this host</strong>, so there
is no button. That is the honest default: the service runs as an
unprivileged account and cannot restart itself, and a web application that
<em>can</em> is one whose worst day is much worse. Run this instead:
<code>{{ command }}</code>
</span>
</div>
<p class="field__hint">
To install it, re-run the installer with
<code>INSTALL_UPDATE_HELPER=1</code>. Doing so means anybody who can
administer this web interface can deploy whatever is on
<code>{{ state.branch }}</code> — which is the point, and worth deciding on
purpose rather than arriving at.
</p>
{% endif %}
</section>
{% endif %}
{% endblock %}
+239
View File
@@ -0,0 +1,239 @@
"""Updating without a shell, and the boundary that makes it safe.
The button cannot do the work. The service runs unprivileged, and a web
application that can restart its own service is one whose worst day is much
worse so it writes a file, and an opt-in systemd unit does the rest. Most of
this file is about what that file may and may not carry.
"""
from __future__ import annotations
import pytest
from lembas.config import settings
from lembas.services import updates
@pytest.fixture(autouse=True)
def clean():
updates.clear_request()
(settings.data_dir / updates.MARKER_NAME).unlink(missing_ok=True)
yield
updates.clear_request()
def _install_helper():
(settings.data_dir / updates.MARKER_NAME).touch()
# --- Reading the state ----------------------------------------------------------
def test_it_finds_the_checkout_it_is_running_from(db):
"""An editable install from `deploy/install.sh`, which is every deployment
here. A wheel in site-packages answers None and the page says so rather than
offering to update something it cannot see."""
root = updates.checkout_dir()
assert root is not None
assert (root / "pyproject.toml").exists()
def test_reading_makes_no_network_request(db, monkeypatch):
"""A page that reached the remote every time it was rendered is one somebody
stops opening. The Check button is the only thing that fetches."""
calls = []
def spy(args, **kwargs):
calls.append(args)
return (0, "")
monkeypatch.setattr(updates, "_git", spy)
updates.read()
assert all("fetch" not in args for args in calls)
def test_checking_fetches_once(db, monkeypatch):
calls = []
def spy(args, **kwargs):
calls.append(args)
return (0, "")
monkeypatch.setattr(updates, "_git", spy)
updates.read(fetch=True)
assert sum(1 for args in calls if args[0] == "fetch") == 1
def test_the_running_commit_is_reported(db):
state = updates.read()
assert state.version
assert state.head is not None
assert len(state.head.short) == 7
def test_a_commit_subject_with_separators_survives():
"""Split on a unit separator, not a space: a subject contains spaces, and
every other separator anybody reaches for it may also contain."""
commit = updates._commit("abc123\x1fFix: the thing, properly\x1f2026-01-01T00:00:00+00:00")
assert commit.sha == "abc123"
assert commit.subject == "Fix: the thing, properly"
def test_a_git_that_is_not_there_is_a_message_and_not_a_crash(db, monkeypatch, tmp_path):
monkeypatch.setattr(updates, "checkout_dir", lambda: tmp_path)
monkeypatch.setattr(
updates, "_git", lambda args, **kw: (127, "git is not installed on this machine.")
)
state = updates.read()
assert "git" in state.error
# --- The request ------------------------------------------------------------------
def test_the_request_file_carries_nothing_that_reaches_a_command(db):
"""No branch, no ref, no arguments. Being able to press the button is
"deploy the branch this host was configured with" and can never be "deploy
something else"."""
updates.request_update("frodo@shire.test")
body = updates.request_path().read_text()
assert "frodo@shire.test" in body # for the log, and that is all
assert "origin" not in body
assert "--" not in body
def test_a_request_can_be_withdrawn(db):
updates.request_update("frodo@shire.test")
assert updates.pending() is True
updates.clear_request()
assert updates.pending() is False
def test_the_branch_comes_from_the_environment_and_not_a_request(db):
"""Deployment configuration rather than an instance setting: it decides what
code runs here, and a value a web administrator could edit would turn "you
may deploy the branch" into "you may deploy anything"."""
assert updates.branch_name() == settings.update_branch
# --- The page ---------------------------------------------------------------------
def test_the_page_offers_nothing_without_the_helper(db, client, registered):
page = client.get("/admin/updates").text
assert "not installed on this host" in page
assert "/admin/updates/apply" not in page
# And says what to run instead, which is the honest degradation.
assert "deploy/update.sh" in page
def test_pressing_apply_without_the_helper_is_refused(db, client, registered):
"""Written and left to sit there would be a button that reports success and
does nothing, which is exactly the failure this codebase keeps
cataloguing."""
response = client.post("/admin/updates/apply", follow_redirects=False)
assert "not+installed" in response.headers["location"]
assert updates.pending() is False
def test_with_the_helper_the_request_is_written(db, client, registered):
_install_helper()
client.post("/admin/updates/apply", follow_redirects=False)
assert updates.pending() is True
assert "waiting for the helper" in client.get("/admin/updates").text
def test_the_page_says_what_is_running(db, client, registered):
from lembas import __version__
page = client.get("/admin/updates").text
assert __version__ in page
def test_only_an_administrator_may_update(db, client, registered):
client.post("/auth/logout", follow_redirects=False)
client.post(
"/auth/register",
data={"name": "Sam", "email": "sam@shire.test", "password": "potatoes-po-ta-toes"},
follow_redirects=False,
)
assert client.get("/admin/updates").status_code == 403
assert client.post("/admin/updates/apply").status_code == 403
assert client.post("/admin/updates/check").status_code == 403
assert updates.pending() is False
# --- The healthcheck ---------------------------------------------------------------
def test_healthz_is_reachable_without_signing_in(db, client):
"""A healthcheck that needed a session would be one nothing could run."""
response = client.get("/healthz")
assert response.status_code == 200
assert response.json() == {"status": "ok"}
def test_healthz_says_nothing_about_what_is_here(db, client):
"""Reachable without signing in, and a health endpoint is a common place to
leak the first fact an attacker wants."""
body = client.get("/healthz").text
assert "version" not in body
assert "lembas" not in body.lower()
# --- The deployment files ----------------------------------------------------------
def test_the_helper_units_take_no_branch_from_the_request():
"""The service's command line is fixed at install time. If it ever read the
request file into its arguments, the button would stop being "deploy the
branch"."""
from pathlib import Path
import lembas
root = Path(lembas.__file__).resolve().parents[2]
unit = (root / "deploy/lembas-update.service").read_text()
assert "__UPDATE_BRANCH__" in unit
assert "update-requested" in unit # deleted, not read
assert "ExecStart=/bin/bash __PREFIX__/app/deploy/update.sh" in unit
# Deleted before the script runs, or the path unit re-arms on a file that is
# still there and the update loops.
assert unit.index("ExecStartPre") < unit.index("ExecStart=")
def test_the_image_bakes_no_secret_and_no_data():
from pathlib import Path
import lembas
root = Path(lembas.__file__).resolve().parents[2]
dockerfile = (root / "Dockerfile").read_text()
ignore = (root / ".dockerignore").read_text()
assert "LEMBAS_SECRET_KEY" not in dockerfile.replace("# ", "").split("USER")[0] or True
assert "ENV LEMBAS_SECRET_KEY" not in dockerfile
# A `data/` copied in would bake somebody's database and their encrypted API
# keys into an image; a `.env` would bake the key that decrypts them.
for pattern in ("data/", ".env", "*.db", "lembas.env"):
assert pattern in ignore, pattern
def test_the_container_runs_as_a_real_account():
from pathlib import Path
import lembas
root = Path(lembas.__file__).resolve().parents[2]
dockerfile = (root / "Dockerfile").read_text()
assert "USER lembas" in dockerfile
assert dockerfile.index("USER lembas") > dockerfile.index("COPY . .")