Jaroslav Beneš
c3bb6c9eaf
The button cannot do the work, and that is the whole design. The service runs as an unprivileged account, cannot restart itself, and should not be able to: a web application that can restart its own service is one whose worst day is much worse. So /admin/updates writes a file, and an opt-in systemd .path unit runs deploy/update.sh as root. Three properties hold it up, and each is a thing that could have been got wrong. The request file carries nothing that reaches a command line -- no branch, no ref, no arguments -- because the branch is baked into the unit at install time, so pressing the button is always "deploy the branch this host was configured with" and can never be "deploy something else". It is off unless somebody passes INSTALL_UPDATE_HELPER=1, and re-running the installer without it removes both units and the marker. And without the helper the page says so and prints the manual command rather than writing a file nothing is watching, which would be a button that reports success and does nothing. The card that says all of this is rendered whether or not there is anything to apply. It was inside the "there is an update" branch first, so an administrator could not discover the helper was missing until the day they needed it, which is the worst possible moment. Opening the page makes no network request; Check is the one thing that fetches. And it shows the log between, not a count: "3 behind" is a number somebody has to go and look up, while the subjects are what decides whether this is worth restarting for right now. Docker is one stage, because there is nothing to build -- no Node, no compiled assets. It bakes no secret key (one in an image is one every copy shares, and rotating it makes stored API keys unreadable), no data, and no .git, so /admin/updates inside a container correctly reports that it was not installed from a checkout. Compose publishes on loopback and refuses to start without a key. TLS in front is a constraint rather than a recommendation: the service worker and the microphone both require HTTPS or localhost. The image was built and run before this was committed, which is how the missing COPY of LICENSE was found -- pyproject declares it and the build backend reads it, so the failure reads like a packaging problem and is one line. deploy/lxc-install.sh creates an unprivileged Debian container and runs the existing installer inside it. A wrapper, not a second install path: a parallel installer is two things to keep correct and one of them rots. /healthz opens the database rather than only proving the socket is listening -- a process that is up with a database it cannot open answers every page with a 500 -- and says nothing about what is here, being reachable without signing in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
203 lines
9.1 KiB
Markdown
203 lines
9.1 KiB
Markdown
# Deployment
|
|
|
|
Installs LLeMbas as a **system** service behind nginx with a self-signed
|
|
certificate. Written for a systemd + nginx host; tested on Arch.
|
|
|
|
| | Default |
|
|
|---|---|
|
|
| Service user | `lembas` (system account, `nologin`) |
|
|
| Home | `/home/lembas` |
|
|
| Install prefix | `/srv/lembas` (bind mount of the home) |
|
|
| Checkout | `$PREFIX/app` |
|
|
| Virtualenv | `$PREFIX/venv` |
|
|
| Database | `$PREFIX/data/lembas.db` |
|
|
| Environment | `$PREFIX/lembas.env` (mode 600) |
|
|
| Unit | `/etc/systemd/system/lembas.service` |
|
|
| Vhost | `/etc/nginx/conf.d/<host>.conf` |
|
|
| Listens on | `127.0.0.1:8080` — reachable only through nginx |
|
|
|
|
The prefix defaults to a bind mount of the service user's home because on many
|
|
machines the root filesystem is small while `/home` is not, and the virtualenv
|
|
plus database belong on the larger volume. Set `PREFIX=$HOME_DIR` to skip it.
|
|
|
|
## First install
|
|
|
|
```bash
|
|
SITE_HOST=chat.example ./deploy/install.sh
|
|
```
|
|
|
|
Idempotent — safe to re-run. It creates the user and bind mount, clones the
|
|
repo, builds the venv, generates `lembas.env` with a fresh `LEMBAS_SECRET_KEY`,
|
|
installs the unit and vhost, issues a self-signed certificate, adds a
|
|
`/etc/hosts` entry if the name does not already resolve, and enables the
|
|
service.
|
|
|
|
Then open `https://<SITE_HOST>`, accept the certificate warning, and create the
|
|
first account — it becomes the administrator.
|
|
|
|
Everything is overridable from the environment:
|
|
|
|
| Variable | Default | |
|
|
|---|---|---|
|
|
| `SITE_HOST` | `lembas.local` | nginx `server_name` and certificate CN |
|
|
| `APP_PORT` | `8080` | loopback port the service binds |
|
|
| `SERVICE_USER` | `lembas` | system account to run as |
|
|
| `HOME_DIR` | `/home/lembas` | that account's home |
|
|
| `PREFIX` | `/srv/lembas` | install root (bind mount of `HOME_DIR`) |
|
|
| `REPO_URL` | this checkout's `origin` | so a fork deploys itself |
|
|
| `LEMBAS_BRANCH` | `main` | branch to deploy, and what `/admin/updates` compares against |
|
|
| `INSTALL_UPDATE_HELPER` | `0` | `1` lets the web interface deploy that branch as root |
|
|
|
|
## Updating from the web interface
|
|
|
|
`/admin/updates` says what is running, what is on the branch and what changed
|
|
between. **Checking** reaches the remote; opening the page does not.
|
|
|
|
The button is opt-in, and the reason is a boundary rather than caution:
|
|
|
|
```bash
|
|
INSTALL_UPDATE_HELPER=1 SITE_HOST=chat.example ./deploy/install.sh
|
|
```
|
|
|
|
That installs `lembas-update.path` and `lembas-update.service`. The web
|
|
interface writes `$PREFIX/data/update-requested`; the path unit notices and the
|
|
service runs `update.sh` **as root**.
|
|
|
|
**What that grants.** Anybody who can administer this web interface can then
|
|
deploy whatever is on the configured branch and restart the service. That is the
|
|
point of it, and it is why it is not the default.
|
|
|
|
**What it deliberately does not grant.** The request file carries nothing that
|
|
reaches a command line — no branch, no ref, no arguments. The branch is baked
|
|
into the unit at install time from `LEMBAS_BRANCH`, so the button is always
|
|
"deploy the branch this host was configured with" and never "deploy something
|
|
else". Re-running the installer without the flag removes both units and the
|
|
marker, and the page goes back to printing the manual command.
|
|
|
|
Without the helper the page says so and shows `sudo …/deploy/update.sh`, which is
|
|
the same honest degradation the SSH and search extras have.
|
|
|
|
## Deploying a change
|
|
|
|
```bash
|
|
git push
|
|
./deploy/update.sh
|
|
```
|
|
|
|
`update.sh` fetches, hard-resets the deployment checkout to `origin/main`,
|
|
reinstalls dependencies and restarts, printing the commits it pulled. The hard
|
|
reset is deliberate: nothing is ever edited in place there, so there is no local
|
|
work to preserve and no conflicts to resolve.
|
|
|
|
## In a container
|
|
|
|
A `Dockerfile` and a `docker-compose.yml` are in the repository root.
|
|
|
|
```bash
|
|
echo "LEMBAS_SECRET_KEY=$(python -c 'import secrets;print(secrets.token_urlsafe(48))')" > .env
|
|
docker compose up -d
|
|
```
|
|
|
|
It publishes on `127.0.0.1:8080` and expects **a TLS reverse proxy in front**.
|
|
That is a constraint, not a preference: a service worker and a microphone both
|
|
require HTTPS or localhost, so over plain http on a LAN address the app cannot be
|
|
installed and cannot dictate — and the session cookie is deliberately not marked
|
|
`secure`, so an attacker on that network could steal a session.
|
|
|
|
Three things about the image:
|
|
|
|
- **No secret key is baked in**, and compose refuses to start without one. A key
|
|
in an image is a key every copy of that image shares, and rotating it signs
|
|
everybody out *and* makes stored upstream API keys unreadable.
|
|
- **`.git` is excluded**, so `/admin/updates` inside a container says it was not
|
|
installed from a checkout and offers nothing. That is correct: a container is
|
|
updated by pulling a new image.
|
|
- **One replica.** The generation registry, the stop mechanism, the terminal
|
|
sessions and the schedule ticker are all in-process — two would mean two
|
|
tickers and every schedule firing twice.
|
|
|
|
## On Proxmox
|
|
|
|
```bash
|
|
CTID=140 SITE_HOST=chat.example ./deploy/lxc-install.sh
|
|
```
|
|
|
|
Run on the Proxmox host. It creates an **unprivileged** Debian container,
|
|
installs the dependencies, and runs `deploy/install.sh` inside it — the same
|
|
installer, so a fix there reaches this without anybody remembering. Unprivileged
|
|
is not a default to change: nothing LLeMbas does needs privilege, because agent
|
|
chats run their commands over SSH on some *other* machine.
|
|
|
|
## Operating it
|
|
|
|
```bash
|
|
systemctl status lembas
|
|
journalctl -u lembas -f
|
|
sudo -u lembas /srv/lembas/venv/bin/lembas info # paths and counts
|
|
```
|
|
|
|
Configuration lives in `$PREFIX/lembas.env`. Edit it and restart.
|
|
|
|
## Notes
|
|
|
|
**The secret key is generated once.** `install.sh` will not overwrite an
|
|
existing `lembas.env`. Rotating `LEMBAS_SECRET_KEY` signs every user out *and*
|
|
makes stored upstream API keys unreadable — they would have to be re-entered.
|
|
|
|
**nginx buffering is off for a reason.** Replies stream as server-sent events.
|
|
With `proxy_buffering on` (the default) nginx holds the entire reply and
|
|
delivers it in one lump at the end, which is indistinguishable from streaming
|
|
being broken. `proxy_read_timeout` is raised to an hour because a model can
|
|
think for minutes before the first token.
|
|
|
|
**The vhost passes WebSocket upgrades through, and must.** The terminal panel
|
|
is the one WebSocket in LLeMbas. A `location` that sets `Connection ""` — which
|
|
is what SSE alone needs, and what this template used to say — fails every
|
|
handshake, and a failed handshake tells the browser nothing: no status, no
|
|
reason. The `map $http_upgrade` at the top of the vhost yields the empty string
|
|
when the client did not ask to upgrade, so streaming is unaffected. `update.sh`
|
|
warns when the installed vhost has drifted from the template, because this is
|
|
the failure most likely to be diagnosed as a bug in the application.
|
|
|
|
**Every restart kills every open shell.** A reply being written is persisted
|
|
with whatever it has; a terminal has nothing to persist, so a command still
|
|
running on the far side is cut off. `update.sh` restarts unconditionally, so a
|
|
deploy in the middle of somebody's `apt-get dist-upgrade` ends it. The panel is
|
|
told why rather than silently reconnecting to a new shell, which would have
|
|
lost the working directory and the half-typed command.
|
|
|
|
**A terminal is not in the transcript, and is not logged.** The open and the
|
|
close are logged with the user, the chat and the connection; what was typed is
|
|
not recorded anywhere. That follows from the design — the chat's mode governs
|
|
the model, not the person at the keyboard — but everything else an agent chat
|
|
does *is* in the transcript, so it is a difference in kind and worth knowing
|
|
before somebody goes looking for the history.
|
|
|
|
**Nothing an agent does runs on this machine.** Agent chats execute their
|
|
commands over SSH, on a host somebody added and prepared — a container, a VM,
|
|
another machine. That is the whole isolation story, and it is why the unit can
|
|
stay locked down instead of being opened up to make room for a sandbox.
|
|
|
|
`ProtectSystem=full` rather than `strict` only because the data directory must
|
|
be writable and `strict` would mean listing every path.
|
|
|
|
The practical consequence for whoever runs this: **the security of an agent
|
|
chat is the security of the host behind its SSH profile.** A throwaway
|
|
container with the one project mounted into it is a very different thing from a
|
|
key to a production server, and LLeMbas cannot tell them apart.
|
|
|
|
**Use a real certificate if this is exposed beyond a trusted LAN.** The
|
|
self-signed cert exists so the install works with no external dependencies;
|
|
point `ssl_certificate` at a real one and nothing else needs to change.
|
|
|
|
The session cookie is deliberately not marked `secure`, so that a LAN install
|
|
over plain http can sign anybody in at all. That has always meant a network
|
|
attacker on http could steal a session; with the terminal it also means they
|
|
could open an interactive shell on the machine behind that chat. If the
|
|
terminal is switched on, run this over TLS.
|
|
|
|
**One worker only.** True of generations already — the registry is in-process —
|
|
and sharper here: with two workers a browser reconnecting to its terminal could
|
|
land in the process that has no shell for it, and silently open a second one on
|
|
the same machine.
|