Wake the model when a background job finishes

The other half of background execution: a job that finishes while nobody is
looking prompts the model back with its result, rather than sitting unread until
the model happens to run again.

The vehicle is the queue, because it is the only wiring that already delivers a
turn into or after a reply. A per-job poller notices completion and calls
jobs.wake. If a reply is being written the completion is left queued for that
reply's _inject/_drain; if the chat is idle a fresh reply is started to answer
it -- the send_queued_now move. All of it under a per-chat lock with no await
between the running-check and ensure, so two jobs finishing at once cannot each
spin up a generation: the second sees the first's reply already live and leaves
its completion for it. That is the invariant the queue exists to hold, reached
from outside a request for the first time.

The completion is a user-role turn whose content names itself a machine event --
"A background job you started has finished" -- not a bare person turn. _inject
sends a queued turn verbatim, so the framing cannot live there; it lives in the
words, the way execute_plan quotes the plan, and a tool.background fragment tells
the model these arrive and are a machine event rather than the person speaking.

The poller reconnects a fresh connection each tick rather than holding one open
-- holding one is the exact live-connection state the whole ssh.py/base.py design
forbids, and poll is self-healing besides. Bounded by background_max_jobs and a
six-hour ceiling, after which the remote job may keep running but we stop
watching it.

A Job table, and here the terminal/generation "lost on restart" precedent does
NOT transfer: those are seconds long with a human watching, a background job is
hours long with nobody watching -- the one case a restart forgetting it would
silently break the feature's whole promise. So the row lets a lifespan startup
hook rehydrate the watcher and wake as if nothing happened. Cancelling a watcher
never stops the detached remote job; it runs on and is picked back up.

Tested end to end against a real local shell: launch a detached command, poll it
to completion through a watcher, and assert the model was woken with the exit
code and output -- plus the lock proving two simultaneous completions start one
reply, not two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-03 14:30:44 +02:00
parent 3fc3449726
commit 6cffcb357d
11 changed files with 549 additions and 9 deletions
+31 -1
View File
@@ -100,4 +100,34 @@ class SshProfile(UUIDPrimaryKey, Timestamps, Base):
return f"<SshProfile {self.name} {self.address}>"
__all__ = ["AUTH_KEY", "AUTH_METHODS", "AUTH_PASSWORD", "SshProfile"]
class Job(Timestamps, Base):
"""A command left running on the far side after the reply that started it.
The durable record behind `services/agent/jobs.py`, which otherwise keeps
only an in-process registry lost on restart. A background job runs for
minutes to hours with nobody watching -- exactly the case a restart must not
forget -- so the row lets a startup hook re-poll the job's deterministic
exit-file and wake the model as if nothing had happened.
The id is `jobs`'s own short hex, not a UUIDPrimaryKey, because the same id
names the files on the machine and is quoted back by the model.
"""
__tablename__ = "agent_jobs"
id: Mapped[str] = mapped_column(String(32), primary_key=True)
chat_id: Mapped[str] = mapped_column(
String(32), ForeignKey("chats.id", ondelete="CASCADE"), index=True, nullable=False
)
command: Mapped[str] = mapped_column(Text, default="")
# running | done | killed | lost. `lost` means it stopped without an exit
# code being recorded -- killed out of band, or the host rebooted under it.
status: Mapped[str] = mapped_column(String(16), default="running", nullable=False)
exit_status: Mapped[int | None] = mapped_column(Integer)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
def __repr__(self) -> str:
return f"<Job {self.id} {self.status}>"
__all__ = ["AUTH_KEY", "AUTH_METHODS", "AUTH_PASSWORD", "Job", "SshProfile"]