Jaroslav Beneš
5612bf2acd
Updates follow a channel now. `stable` is the newest vX.Y.Z tag; `edge` is the branch tip, which is what this did before. Stable is the default, because a branch tip is not a release -- following one means deploying whatever was pushed five minutes ago, possibly mid-feature, which is right for whoever builds this and wrong for whoever runs it. The page can now say "running 1.0.0, 1.1.0 available" rather than showing two shas and leaving somebody to guess. Read with git plumbing and never a forge API, for three reasons in the order they bite. It would need a token on the deployment host -- a credential that can reach the repository, sitting on a box, to answer a read-only question about version numbers. It would tie this to one forge, so a fork on GitHub gets nothing. And it breaks: checked against the Gitea this is developed on, `tea whoami` works and `tea releases list` returns a 500 from a server-side panic about token scopes, so a page resting on that endpoint would have shipped already broken. Release notes still travel, inside the annotated tag object, which `git for-each-ref` reads with no API anywhere. Two details that are only obvious after getting them wrong. A tag with a suffix is not a release: git's version sort puts v1.1.0-rc1 *above* v1.1.0, so accepting one would step a stable host onto a candidate on the strength of a hyphen. And `--sort=-v:refname` rather than a lexical sort, which puts v1.9.0 above v1.10.0 and does it silently the first time a project reaches ten of anything -- there is a test. What is running is `git describe --tags --always`, so it reads "1.0.0" at a tag, "1.0.0-7-gd4f56d" seven commits past one, and a bare sha before the first release ever exists. That last case is what `--always` is for. When it lands exactly on a tag whose name disagrees with __version__, the page says so: a tag cut before the version bump names a release nobody can identify afterwards, and the check costs no subprocess because both facts are already in hand. update.sh resolves the channel the same way and detaches at the tag rather than resetting -- a `reset --hard <tag>` while on main would move the local branch to it, which is a rewrite of a ref nobody asked to rewrite. A host with no tags falls back to the branch and says so, which is every host until the release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
228 lines
10 KiB
Markdown
228 lines
10 KiB
Markdown
# Deployment
|
|
|
|
Installs LLeMbas as a **system** service behind nginx with a self-signed
|
|
certificate. Written for a systemd + nginx host; tested on Arch.
|
|
|
|
| | Default |
|
|
|---|---|
|
|
| Service user | `lembas` (system account, `nologin`) |
|
|
| Home | `/home/lembas` |
|
|
| Install prefix | `/srv/lembas` (bind mount of the home) |
|
|
| Checkout | `$PREFIX/app` |
|
|
| Virtualenv | `$PREFIX/venv` |
|
|
| Database | `$PREFIX/data/lembas.db` |
|
|
| Environment | `$PREFIX/lembas.env` (mode 600) |
|
|
| Unit | `/etc/systemd/system/lembas.service` |
|
|
| Vhost | `/etc/nginx/conf.d/<host>.conf` |
|
|
| Listens on | `127.0.0.1:8080` — reachable only through nginx |
|
|
|
|
The prefix defaults to a bind mount of the service user's home because on many
|
|
machines the root filesystem is small while `/home` is not, and the virtualenv
|
|
plus database belong on the larger volume. Set `PREFIX=$HOME_DIR` to skip it.
|
|
|
|
## First install
|
|
|
|
```bash
|
|
SITE_HOST=chat.example ./deploy/install.sh
|
|
```
|
|
|
|
Idempotent — safe to re-run. It creates the user and bind mount, clones the
|
|
repo, builds the venv, generates `lembas.env` with a fresh `LEMBAS_SECRET_KEY`,
|
|
installs the unit and vhost, issues a self-signed certificate, adds a
|
|
`/etc/hosts` entry if the name does not already resolve, and enables the
|
|
service.
|
|
|
|
Then open `https://<SITE_HOST>`, accept the certificate warning, and create the
|
|
first account — it becomes the administrator.
|
|
|
|
Everything is overridable from the environment:
|
|
|
|
| Variable | Default | |
|
|
|---|---|---|
|
|
| `SITE_HOST` | `lembas.local` | nginx `server_name` and certificate CN |
|
|
| `APP_PORT` | `8080` | loopback port the service binds |
|
|
| `SERVICE_USER` | `lembas` | system account to run as |
|
|
| `HOME_DIR` | `/home/lembas` | that account's home |
|
|
| `PREFIX` | `/srv/lembas` | install root (bind mount of `HOME_DIR`) |
|
|
| `REPO_URL` | this checkout's `origin` | so a fork deploys itself |
|
|
| `LEMBAS_BRANCH` | `main` | branch to fetch, and what the `edge` channel follows |
|
|
| `LEMBAS_CHANNEL` | `stable` | `stable` follows release tags, `edge` follows the branch tip |
|
|
| `INSTALL_UPDATE_HELPER` | `0` | `1` lets the web interface deploy that branch as root |
|
|
|
|
## Channels
|
|
|
|
| | follows | for |
|
|
|---|---|---|
|
|
| `stable` (default) | the newest `vX.Y.Z` tag | anybody running this |
|
|
| `edge` | the tip of `LEMBAS_BRANCH` | whoever is building it |
|
|
|
|
**A branch tip is not a release.** Following `main` means deploying whatever was
|
|
pushed five minutes ago, possibly mid-feature — right for development and wrong
|
|
for a machine somebody depends on. Stable is the default for that reason.
|
|
|
|
A tag with a suffix (`v1.1.0-rc1`) is deliberately **not** a release: git's
|
|
version sort puts it *above* `v1.1.0`, so accepting one would step a stable host
|
|
onto a release candidate on the strength of a hyphen. A prerelease is something
|
|
you check out by name.
|
|
|
|
Release notes travel inside **annotated** tags, so `git tag -a v1.1.0 -m "…"` is
|
|
what puts them on the update page. No forge API is involved anywhere — which
|
|
matters more than it sounds: a token on the deployment host to answer a
|
|
read-only question about version numbers is a bad trade, it would tie this to
|
|
one forge, and the Gitea API this was checked against returns a 500 from a
|
|
server-side panic on exactly that endpoint.
|
|
|
|
## Updating from the web interface
|
|
|
|
`/admin/updates` says what is running (`git describe`, so `1.0.0` at a tag and
|
|
`1.0.0-7-gd4f56d` seven commits past one), what the channel offers, the release
|
|
notes, and the commits between. **Checking** reaches the remote; opening the page
|
|
does not.
|
|
|
|
The button is opt-in, and the reason is a boundary rather than caution:
|
|
|
|
```bash
|
|
INSTALL_UPDATE_HELPER=1 SITE_HOST=chat.example ./deploy/install.sh
|
|
```
|
|
|
|
That installs `lembas-update.path` and `lembas-update.service`. The web
|
|
interface writes `$PREFIX/data/update-requested`; the path unit notices and the
|
|
service runs `update.sh` **as root**, on the configured channel.
|
|
|
|
**What that grants.** Anybody who can administer this web interface can then
|
|
deploy whatever is on the configured branch and restart the service. That is the
|
|
point of it, and it is why it is not the default.
|
|
|
|
**What it deliberately does not grant.** The request file carries nothing that
|
|
reaches a command line — no ref, no branch, no channel, no arguments. Both are
|
|
baked into the unit at install time, so the button is always "deploy the channel
|
|
this host was configured with" and never "deploy something else". Re-running the installer without the flag removes both units and the
|
|
marker, and the page goes back to printing the manual command.
|
|
|
|
Without the helper the page says so and shows `sudo …/deploy/update.sh`, which is
|
|
the same honest degradation the SSH and search extras have.
|
|
|
|
## Deploying a change
|
|
|
|
```bash
|
|
git push
|
|
./deploy/update.sh
|
|
```
|
|
|
|
`update.sh` fetches, hard-resets the deployment checkout to `origin/main`,
|
|
reinstalls dependencies and restarts, printing the commits it pulled. The hard
|
|
reset is deliberate: nothing is ever edited in place there, so there is no local
|
|
work to preserve and no conflicts to resolve.
|
|
|
|
## In a container
|
|
|
|
A `Dockerfile` and a `docker-compose.yml` are in the repository root.
|
|
|
|
```bash
|
|
echo "LEMBAS_SECRET_KEY=$(python -c 'import secrets;print(secrets.token_urlsafe(48))')" > .env
|
|
docker compose up -d
|
|
```
|
|
|
|
It publishes on `127.0.0.1:8080` and expects **a TLS reverse proxy in front**.
|
|
That is a constraint, not a preference: a service worker and a microphone both
|
|
require HTTPS or localhost, so over plain http on a LAN address the app cannot be
|
|
installed and cannot dictate — and the session cookie is deliberately not marked
|
|
`secure`, so an attacker on that network could steal a session.
|
|
|
|
Three things about the image:
|
|
|
|
- **No secret key is baked in**, and compose refuses to start without one. A key
|
|
in an image is a key every copy of that image shares, and rotating it signs
|
|
everybody out *and* makes stored upstream API keys unreadable.
|
|
- **`.git` is excluded**, so `/admin/updates` inside a container says it was not
|
|
installed from a checkout and offers nothing. That is correct: a container is
|
|
updated by pulling a new image.
|
|
- **One replica.** The generation registry, the stop mechanism, the terminal
|
|
sessions and the schedule ticker are all in-process — two would mean two
|
|
tickers and every schedule firing twice.
|
|
|
|
## On Proxmox
|
|
|
|
```bash
|
|
CTID=140 SITE_HOST=chat.example ./deploy/lxc-install.sh
|
|
```
|
|
|
|
Run on the Proxmox host. It creates an **unprivileged** Debian container,
|
|
installs the dependencies, and runs `deploy/install.sh` inside it — the same
|
|
installer, so a fix there reaches this without anybody remembering. Unprivileged
|
|
is not a default to change: nothing LLeMbas does needs privilege, because agent
|
|
chats run their commands over SSH on some *other* machine.
|
|
|
|
## Operating it
|
|
|
|
```bash
|
|
systemctl status lembas
|
|
journalctl -u lembas -f
|
|
sudo -u lembas /srv/lembas/venv/bin/lembas info # paths and counts
|
|
```
|
|
|
|
Configuration lives in `$PREFIX/lembas.env`. Edit it and restart.
|
|
|
|
## Notes
|
|
|
|
**The secret key is generated once.** `install.sh` will not overwrite an
|
|
existing `lembas.env`. Rotating `LEMBAS_SECRET_KEY` signs every user out *and*
|
|
makes stored upstream API keys unreadable — they would have to be re-entered.
|
|
|
|
**nginx buffering is off for a reason.** Replies stream as server-sent events.
|
|
With `proxy_buffering on` (the default) nginx holds the entire reply and
|
|
delivers it in one lump at the end, which is indistinguishable from streaming
|
|
being broken. `proxy_read_timeout` is raised to an hour because a model can
|
|
think for minutes before the first token.
|
|
|
|
**The vhost passes WebSocket upgrades through, and must.** The terminal panel
|
|
is the one WebSocket in LLeMbas. A `location` that sets `Connection ""` — which
|
|
is what SSE alone needs, and what this template used to say — fails every
|
|
handshake, and a failed handshake tells the browser nothing: no status, no
|
|
reason. The `map $http_upgrade` at the top of the vhost yields the empty string
|
|
when the client did not ask to upgrade, so streaming is unaffected. `update.sh`
|
|
warns when the installed vhost has drifted from the template, because this is
|
|
the failure most likely to be diagnosed as a bug in the application.
|
|
|
|
**Every restart kills every open shell.** A reply being written is persisted
|
|
with whatever it has; a terminal has nothing to persist, so a command still
|
|
running on the far side is cut off. `update.sh` restarts unconditionally, so a
|
|
deploy in the middle of somebody's `apt-get dist-upgrade` ends it. The panel is
|
|
told why rather than silently reconnecting to a new shell, which would have
|
|
lost the working directory and the half-typed command.
|
|
|
|
**A terminal is not in the transcript, and is not logged.** The open and the
|
|
close are logged with the user, the chat and the connection; what was typed is
|
|
not recorded anywhere. That follows from the design — the chat's mode governs
|
|
the model, not the person at the keyboard — but everything else an agent chat
|
|
does *is* in the transcript, so it is a difference in kind and worth knowing
|
|
before somebody goes looking for the history.
|
|
|
|
**Nothing an agent does runs on this machine.** Agent chats execute their
|
|
commands over SSH, on a host somebody added and prepared — a container, a VM,
|
|
another machine. That is the whole isolation story, and it is why the unit can
|
|
stay locked down instead of being opened up to make room for a sandbox.
|
|
|
|
`ProtectSystem=full` rather than `strict` only because the data directory must
|
|
be writable and `strict` would mean listing every path.
|
|
|
|
The practical consequence for whoever runs this: **the security of an agent
|
|
chat is the security of the host behind its SSH profile.** A throwaway
|
|
container with the one project mounted into it is a very different thing from a
|
|
key to a production server, and LLeMbas cannot tell them apart.
|
|
|
|
**Use a real certificate if this is exposed beyond a trusted LAN.** The
|
|
self-signed cert exists so the install works with no external dependencies;
|
|
point `ssl_certificate` at a real one and nothing else needs to change.
|
|
|
|
The session cookie is deliberately not marked `secure`, so that a LAN install
|
|
over plain http can sign anybody in at all. That has always meant a network
|
|
attacker on http could steal a session; with the terminal it also means they
|
|
could open an interactive shell on the machine behind that chat. If the
|
|
terminal is switched on, run this over TLS.
|
|
|
|
**One worker only.** True of generations already — the registry is in-process —
|
|
and sharper here: with two workers a browser reconnecting to its terminal could
|
|
land in the process that has no shell for it, and silently open a second one on
|
|
the same machine.
|