Wake the model when a background job finishes
The other half of background execution: a job that finishes while nobody is looking prompts the model back with its result, rather than sitting unread until the model happens to run again. The vehicle is the queue, because it is the only wiring that already delivers a turn into or after a reply. A per-job poller notices completion and calls jobs.wake. If a reply is being written the completion is left queued for that reply's _inject/_drain; if the chat is idle a fresh reply is started to answer it -- the send_queued_now move. All of it under a per-chat lock with no await between the running-check and ensure, so two jobs finishing at once cannot each spin up a generation: the second sees the first's reply already live and leaves its completion for it. That is the invariant the queue exists to hold, reached from outside a request for the first time. The completion is a user-role turn whose content names itself a machine event -- "A background job you started has finished" -- not a bare person turn. _inject sends a queued turn verbatim, so the framing cannot live there; it lives in the words, the way execute_plan quotes the plan, and a tool.background fragment tells the model these arrive and are a machine event rather than the person speaking. The poller reconnects a fresh connection each tick rather than holding one open -- holding one is the exact live-connection state the whole ssh.py/base.py design forbids, and poll is self-healing besides. Bounded by background_max_jobs and a six-hour ceiling, after which the remote job may keep running but we stop watching it. A Job table, and here the terminal/generation "lost on restart" precedent does NOT transfer: those are seconds long with a human watching, a background job is hours long with nobody watching -- the one case a restart forgetting it would silently break the feature's whole promise. So the row lets a lifespan startup hook rehydrate the watcher and wake as if nothing happened. Cancelling a watcher never stops the detached remote job; it runs on and is picked back up. Tested end to end against a real local shell: launch a detached command, poll it to completion through a watcher, and assert the model was woken with the exit code and output -- plus the lock proving two simultaneous completions start one reply, not two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,7 +20,7 @@ lembas info # paths + counts, useful when confused
|
||||
lembas secret-key # generate LEMBAS_SECRET_KEY
|
||||
lembas create-admin # create or promote an admin
|
||||
|
||||
pytest # 1206 tests, ~70s
|
||||
pytest # 1231 tests, ~75s
|
||||
# PLAN.md tracks what is and is not built
|
||||
ruff check . # lint (line length 100)
|
||||
python scripts/build_artwork.py # regenerate artwork (SVG + PWA icons;
|
||||
@@ -354,6 +354,39 @@ cause of "the agent seems stupid", and the harness says it out loud. So does the
|
||||
other one: on a Debian-derived host `apt-get install` reports the package missing
|
||||
until `apt-get update` has run.
|
||||
|
||||
**A command can outlive the reply, and that is the one place the fresh-shell
|
||||
model is fought rather than obeyed.** `services/agent/jobs.py`: a background job
|
||||
is a `setsid`-detached process on the far side, redirected to a remote logfile
|
||||
and an exit-file, so it survives the connection closing; LLeMbas reconnects (a
|
||||
fresh connection, as always) to read it. Opt-in, off by default. When on, the
|
||||
same wrapper runs *every* command: it launches detached and waits, and a command
|
||||
that outlasts its timeout is kept running as a job rather than killed. Three
|
||||
things in the wrappers are load-bearing and were each got wrong first: the
|
||||
command is **base64'd into a script file**, never put in a quoted `sh -c '…'`
|
||||
(which shatters on `git commit -m 'fix'` and is an injection hole); the child
|
||||
records its **own pid via `$$`** under `setsid` as the group leader, so
|
||||
`job_stop` kills the whole group; and the exit status is read from the
|
||||
**exit-file, not the wrapper's own status**, which is ~0 from its trailing `rm`.
|
||||
A job's files are namespaced by the *calling* chat's id and the wrappers are
|
||||
always built from it, so a model in one chat cannot even name another's job.
|
||||
|
||||
**"Prompt the model back when a job finishes" reuses the queue.** A per-job
|
||||
poller (`jobs._watch`, a fresh connection per tick — never a held one, that
|
||||
being the thing the whole subsystem forbids) notices completion and calls
|
||||
`jobs.wake`. Wake writes the completion as a **user-role turn whose content names
|
||||
itself a machine event** — `_inject` sends a queued turn verbatim, so the framing
|
||||
lives in the words, the way `execute_plan` quotes the plan, and `tool.background`
|
||||
tells the model these arrive. If a reply is running the completion is left
|
||||
`queued` for its `_inject`/`_drain`; if the chat is idle a fresh reply is started
|
||||
(the `send_queued_now` move). All of it is under a **per-chat `asyncio.Lock` with
|
||||
no `await` between the running-check and `ensure`**, so two jobs finishing at
|
||||
once cannot each spin up a generation — the second sees the first's reply live
|
||||
and leaves its completion for it. The `Job` table exists for one reason the
|
||||
terminal/generation "lost on restart" precedent does *not* cover: a job runs for
|
||||
hours with nobody watching, so a restart rehydrates its watcher from the row
|
||||
(`jobs.rehydrate`, in the lifespan) rather than forgetting the one thing the
|
||||
feature promises. Cancelling a watcher never stops the detached remote job.
|
||||
|
||||
**Files never go through a shell.** The SSH exec protocol carries one command
|
||||
*string* that the far side parses, with no argv form at all, so a model-supplied
|
||||
path in a command line is unavoidably a quoting problem. `file_read`/`file_write`
|
||||
|
||||
Reference in New Issue
Block a user