Files
LLeMbas/deploy/README.md
T
Jaroslav Beneš c3bb6c9eaf An update you can ask for, and a boundary that stays where it was
The button cannot do the work, and that is the whole design. The service runs as
an unprivileged account, cannot restart itself, and should not be able to: a web
application that can restart its own service is one whose worst day is much
worse. So /admin/updates writes a file, and an opt-in systemd .path unit runs
deploy/update.sh as root.

Three properties hold it up, and each is a thing that could have been got wrong.
The request file carries nothing that reaches a command line -- no branch, no
ref, no arguments -- because the branch is baked into the unit at install time,
so pressing the button is always "deploy the branch this host was configured
with" and can never be "deploy something else". It is off unless somebody passes
INSTALL_UPDATE_HELPER=1, and re-running the installer without it removes both
units and the marker. And without the helper the page says so and prints the
manual command rather than writing a file nothing is watching, which would be a
button that reports success and does nothing.

The card that says all of this is rendered whether or not there is anything to
apply. It was inside the "there is an update" branch first, so an administrator
could not discover the helper was missing until the day they needed it, which is
the worst possible moment.

Opening the page makes no network request; Check is the one thing that fetches.
And it shows the log between, not a count: "3 behind" is a number somebody has to
go and look up, while the subjects are what decides whether this is worth
restarting for right now.

Docker is one stage, because there is nothing to build -- no Node, no compiled
assets. It bakes no secret key (one in an image is one every copy shares, and
rotating it makes stored API keys unreadable), no data, and no .git, so
/admin/updates inside a container correctly reports that it was not installed
from a checkout. Compose publishes on loopback and refuses to start without a
key. TLS in front is a constraint rather than a recommendation: the service
worker and the microphone both require HTTPS or localhost.

The image was built and run before this was committed, which is how the missing
COPY of LICENSE was found -- pyproject declares it and the build backend reads
it, so the failure reads like a packaging problem and is one line.

deploy/lxc-install.sh creates an unprivileged Debian container and runs the
existing installer inside it. A wrapper, not a second install path: a parallel
installer is two things to keep correct and one of them rots.

/healthz opens the database rather than only proving the socket is listening -- a
process that is up with a database it cannot open answers every page with a 500
-- and says nothing about what is here, being reachable without signing in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 17:56:18 +02:00

9.1 KiB

Deployment

Installs LLeMbas as a system service behind nginx with a self-signed certificate. Written for a systemd + nginx host; tested on Arch.

Default
Service user lembas (system account, nologin)
Home /home/lembas
Install prefix /srv/lembas (bind mount of the home)
Checkout $PREFIX/app
Virtualenv $PREFIX/venv
Database $PREFIX/data/lembas.db
Environment $PREFIX/lembas.env (mode 600)
Unit /etc/systemd/system/lembas.service
Vhost /etc/nginx/conf.d/<host>.conf
Listens on 127.0.0.1:8080 — reachable only through nginx

The prefix defaults to a bind mount of the service user's home because on many machines the root filesystem is small while /home is not, and the virtualenv plus database belong on the larger volume. Set PREFIX=$HOME_DIR to skip it.

First install

SITE_HOST=chat.example ./deploy/install.sh

Idempotent — safe to re-run. It creates the user and bind mount, clones the repo, builds the venv, generates lembas.env with a fresh LEMBAS_SECRET_KEY, installs the unit and vhost, issues a self-signed certificate, adds a /etc/hosts entry if the name does not already resolve, and enables the service.

Then open https://<SITE_HOST>, accept the certificate warning, and create the first account — it becomes the administrator.

Everything is overridable from the environment:

Variable Default
SITE_HOST lembas.local nginx server_name and certificate CN
APP_PORT 8080 loopback port the service binds
SERVICE_USER lembas system account to run as
HOME_DIR /home/lembas that account's home
PREFIX /srv/lembas install root (bind mount of HOME_DIR)
REPO_URL this checkout's origin so a fork deploys itself
LEMBAS_BRANCH main branch to deploy, and what /admin/updates compares against
INSTALL_UPDATE_HELPER 0 1 lets the web interface deploy that branch as root

Updating from the web interface

/admin/updates says what is running, what is on the branch and what changed between. Checking reaches the remote; opening the page does not.

The button is opt-in, and the reason is a boundary rather than caution:

INSTALL_UPDATE_HELPER=1 SITE_HOST=chat.example ./deploy/install.sh

That installs lembas-update.path and lembas-update.service. The web interface writes $PREFIX/data/update-requested; the path unit notices and the service runs update.sh as root.

What that grants. Anybody who can administer this web interface can then deploy whatever is on the configured branch and restart the service. That is the point of it, and it is why it is not the default.

What it deliberately does not grant. The request file carries nothing that reaches a command line — no branch, no ref, no arguments. The branch is baked into the unit at install time from LEMBAS_BRANCH, so the button is always "deploy the branch this host was configured with" and never "deploy something else". Re-running the installer without the flag removes both units and the marker, and the page goes back to printing the manual command.

Without the helper the page says so and shows sudo …/deploy/update.sh, which is the same honest degradation the SSH and search extras have.

Deploying a change

git push
./deploy/update.sh

update.sh fetches, hard-resets the deployment checkout to origin/main, reinstalls dependencies and restarts, printing the commits it pulled. The hard reset is deliberate: nothing is ever edited in place there, so there is no local work to preserve and no conflicts to resolve.

In a container

A Dockerfile and a docker-compose.yml are in the repository root.

echo "LEMBAS_SECRET_KEY=$(python -c 'import secrets;print(secrets.token_urlsafe(48))')" > .env
docker compose up -d

It publishes on 127.0.0.1:8080 and expects a TLS reverse proxy in front. That is a constraint, not a preference: a service worker and a microphone both require HTTPS or localhost, so over plain http on a LAN address the app cannot be installed and cannot dictate — and the session cookie is deliberately not marked secure, so an attacker on that network could steal a session.

Three things about the image:

  • No secret key is baked in, and compose refuses to start without one. A key in an image is a key every copy of that image shares, and rotating it signs everybody out and makes stored upstream API keys unreadable.
  • .git is excluded, so /admin/updates inside a container says it was not installed from a checkout and offers nothing. That is correct: a container is updated by pulling a new image.
  • One replica. The generation registry, the stop mechanism, the terminal sessions and the schedule ticker are all in-process — two would mean two tickers and every schedule firing twice.

On Proxmox

CTID=140 SITE_HOST=chat.example ./deploy/lxc-install.sh

Run on the Proxmox host. It creates an unprivileged Debian container, installs the dependencies, and runs deploy/install.sh inside it — the same installer, so a fix there reaches this without anybody remembering. Unprivileged is not a default to change: nothing LLeMbas does needs privilege, because agent chats run their commands over SSH on some other machine.

Operating it

systemctl status lembas
journalctl -u lembas -f
sudo -u lembas /srv/lembas/venv/bin/lembas info      # paths and counts

Configuration lives in $PREFIX/lembas.env. Edit it and restart.

Notes

The secret key is generated once. install.sh will not overwrite an existing lembas.env. Rotating LEMBAS_SECRET_KEY signs every user out and makes stored upstream API keys unreadable — they would have to be re-entered.

nginx buffering is off for a reason. Replies stream as server-sent events. With proxy_buffering on (the default) nginx holds the entire reply and delivers it in one lump at the end, which is indistinguishable from streaming being broken. proxy_read_timeout is raised to an hour because a model can think for minutes before the first token.

The vhost passes WebSocket upgrades through, and must. The terminal panel is the one WebSocket in LLeMbas. A location that sets Connection "" — which is what SSE alone needs, and what this template used to say — fails every handshake, and a failed handshake tells the browser nothing: no status, no reason. The map $http_upgrade at the top of the vhost yields the empty string when the client did not ask to upgrade, so streaming is unaffected. update.sh warns when the installed vhost has drifted from the template, because this is the failure most likely to be diagnosed as a bug in the application.

Every restart kills every open shell. A reply being written is persisted with whatever it has; a terminal has nothing to persist, so a command still running on the far side is cut off. update.sh restarts unconditionally, so a deploy in the middle of somebody's apt-get dist-upgrade ends it. The panel is told why rather than silently reconnecting to a new shell, which would have lost the working directory and the half-typed command.

A terminal is not in the transcript, and is not logged. The open and the close are logged with the user, the chat and the connection; what was typed is not recorded anywhere. That follows from the design — the chat's mode governs the model, not the person at the keyboard — but everything else an agent chat does is in the transcript, so it is a difference in kind and worth knowing before somebody goes looking for the history.

Nothing an agent does runs on this machine. Agent chats execute their commands over SSH, on a host somebody added and prepared — a container, a VM, another machine. That is the whole isolation story, and it is why the unit can stay locked down instead of being opened up to make room for a sandbox.

ProtectSystem=full rather than strict only because the data directory must be writable and strict would mean listing every path.

The practical consequence for whoever runs this: the security of an agent chat is the security of the host behind its SSH profile. A throwaway container with the one project mounted into it is a very different thing from a key to a production server, and LLeMbas cannot tell them apart.

Use a real certificate if this is exposed beyond a trusted LAN. The self-signed cert exists so the install works with no external dependencies; point ssl_certificate at a real one and nothing else needs to change.

The session cookie is deliberately not marked secure, so that a LAN install over plain http can sign anybody in at all. That has always meant a network attacker on http could steal a session; with the terminal it also means they could open an interactive shell on the machine behind that chat. If the terminal is switched on, run this over TLS.

One worker only. True of generations already — the registry is in-process — and sharper here: with two workers a browser reconnecting to its terminal could land in the process that has no shell for it, and silently open a second one on the same machine.