Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Let a command run in the background instead of being killed

Browse Source 
An agent command is one blocking conn.run over a per-call connection, killed the
moment it hits its timeout -- so a ten-minute apt install is impossible, which is
exactly what a user hit. This is the substrate for running it detached instead:
the model can ask for background=true, or a command that outlasts its timeout is
kept running rather than killed, and either way the model gets tools to read and
stop it. Opt-in, off by default, under Admin -> Agents; off is byte-for-byte the
old behaviour.

The mechanism has to survive the connection closing (that is the whole premise
of the per-call model), so a job is a setsid-detached process on the far side,
redirected to a remote logfile and an exit-file; LLeMbas reconnects, as always,
to read it later. services/agent/jobs.py holds the wrappers.

Three things in those wrappers are load-bearing and each was got wrong in the
first sketch:

- The command never touches a quoted shell context. sh -c '<cmd>' shatters the
  instant the command contains a quote -- git commit -m 'fix', awk '{…}', sed
  's/…/…/' are the common case, and it is an injection hole besides. So the
  command is base64-encoded in Python and decoded on the far side into a script
  file; it is bytes, never shell syntax.
- The child records its own pid via $$ as its first act, under setsid where it
  is the session leader, so job_stop can kill the whole process group. echo $!
  from the launcher captures the wrong pid.
- The command's exit status comes from the exit-file, never the wrapper's own
  status -- which is ~0 from its trailing rm. Reading the wrapper's status would
  mark every job a success.

A command that finishes in time is indistinguishable from a foreground one --
same output, same wording; the difference shows only when it does not, where
instead of "stopped after Ns" it becomes a job id. Auto-convert is its own
sub-switch: with it off, a timeout stays a hard stop and nothing is left
running, because routing the plain case through the detached wrapper would leave
an orphan running past a stop an administrator asked for.

New agent tools job_output/job_list/job_stop, offered only when the feature is
on (the plan_submit gating pattern); job_stop is RISK_EXECUTE since it kills a
process. A job's files are namespaced by the calling chat's id and the wrappers
are always built from it, so a model in one chat cannot even name another's job.

Tested against a real local /bin/sh rather than the fake echo-the-command sshd
fixture, because the shell logic -- setsid, base64, the wait loop, the child
surviving the wait being cut off -- is the whole of the risk. The auto-wake that
prompts the model back when a job finishes is the next commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-03 14:15:49 +02:00
parent bf9287493b
commit 89d2d6ebfd
8 changed files with 1033 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/agent/tools.py
+289 -31
View File
@@ -23,8 +23,8 @@ import posixpath
from typing import Any
from lembas.services import plans
from lembas.services.agent import index, instructions, patch, policy
from lembas.services.agent.base import ExecError, ExecRequest
from lembas.services.agent import index, instructions, jobs, patch, policy
from lembas.services.agent.base import ExecError, ExecRequest, clean_output
from lembas.services.agent.session import AgentContext
from lembas.services.tools import (
RISK_EXECUTE,
@@ -126,29 +126,135 @@ async def _run_shell(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
if reason := _permitted(agent, "shell_run", RISK_EXECUTE, command):
return _refused("shell_run", agent, command, reason)
cwd = str(args.get("cwd") or "").strip()
timeout = _timeout(args.get("timeout"), agent)
background = bool(args.get("background"))
# The explicit choice: launch detached and return at once.
if background and agent.background:
return await _run_background(agent, command, cwd)
# A timed-out command becomes a job only when the feature AND the auto-convert
# are both on. Otherwise -- feature off, or auto-convert off -- the plain path
# runs, which is byte-for-byte what shell_run always did: killed on timeout,
# nothing left running. Routing the plain case through the detached wrapper
# would leave an orphan running past a timeout an administrator said to kill.
if agent.background and agent.background_on_timeout:
return await _run_convertible(agent, command, cwd, timeout)
return await _run_foreground(agent, command, cwd, timeout)
async def _run_foreground(
agent: AgentContext, command: str, cwd: str, timeout: float
) -> ToolOutcome:
"""One command, run to completion or killed at the timeout. The original."""
try:
result = await agent.executor().run(
ExecRequest(
command=command,
cwd=str(args.get("cwd") or "").strip(),
timeout=timeout,
max_bytes=agent.max_output,
)
ExecRequest(command=command, cwd=cwd, timeout=timeout, max_bytes=agent.max_output)
)
except ExecError as exc:
return ToolOutcome(
exc.message, _event("shell_run", agent, command, status="error", error=exc.message)
)
return _shell_outcome(
agent,
command,
result.output.strip(),
exit_status=result.exit_status,
timed_out=result.timed_out,
timeout=timeout,
)
async def _run_convertible(
agent: AgentContext, command: str, cwd: str, timeout: float
) -> ToolOutcome:
"""Launch detached and wait; if it outlasts the timeout, keep it as a job.
While it finishes in time this is indistinguishable from `_run_foreground` --
same output, same wording. The difference is only visible when it does not:
instead of being killed, it is left running and handed back as a job id.
"""
job_id = jobs.new_id()
wrapper = jobs.launch_and_wait_command(agent.chat_id, job_id, command, agent.max_output)
try:
result = await agent.executor().run(
ExecRequest(command=wrapper, cwd=cwd, timeout=timeout, max_bytes=agent.max_output)
)
except ExecError as exc:
return ToolOutcome(
exc.message, _event("shell_run", agent, command, status="error", error=exc.message)
)
body = result.output.strip()
if result.timed_out:
head = f"The command was stopped after {timeout:g}s."
elif result.exit_status == 0:
head = "" if body else "It ran, and printed nothing."
else:
head = f"It exited {result.exit_status}."
job = jobs.JobState(id=job_id, chat_id=agent.chat_id, command=command)
jobs.register(job)
return _backgrounded(agent, command, job, converted=True, timeout=timeout)
# It finished. The wrapper already tailed the log remotely; strip ANSI here.
output, _ = clean_output(result.output or "", limit=agent.max_output)
done = jobs.parse_completed(output, job_id)
return _shell_outcome(
agent, command, done.body, exit_status=done.exit_status, timed_out=False, timeout=timeout
)
async def _run_background(agent: AgentContext, command: str, cwd: str) -> ToolOutcome:
"""The model asked to background it up front."""
try:
job = await jobs.launch(agent, command, cwd)
except ExecError as exc:
return ToolOutcome(
exc.message, _event("shell_run", agent, command, status="error", error=exc.message)
)
return _backgrounded(agent, command, job, converted=False, timeout=0)
def _backgrounded(
agent: AgentContext, command: str, job: jobs.JobState, *, converted: bool, timeout: float
) -> ToolOutcome:
lead = (
f"Still running after {timeout:g}s, so it was kept running in the background"
if converted
else "Started in the background"
)
tail = (
" You will be told when it finishes."
if agent.background_notify
else f' Check on it with job_output("{job.id}").'
)
return ToolOutcome(
f'{lead} as job {job.id}. It keeps running after this reply.{tail}',
_event(
"shell_run",
agent,
command,
status="ok",
text=f"job {job.id} — running in the background",
),
)
def _shell_outcome(
agent: AgentContext,
command: str,
body: str,
*,
exit_status: int | None,
timed_out: bool,
timeout: float,
) -> ToolOutcome:
if timed_out:
head = f"The command was stopped after {timeout:g}s."
elif exit_status == 0:
head = "" if body else "It ran, and printed nothing."
elif exit_status is None:
head = "It stopped before its exit status could be read."
else:
head = f"It exited {exit_status}."
ok = exit_status == 0 and not timed_out
content = f"{head}\n\n{body}".strip() if head else body
return ToolOutcome(
content or "It ran, and printed nothing.",
@@ -156,8 +262,8 @@ async def _run_shell(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
"shell_run",
agent,
command,
status="ok" if result.ok else "error",
error="" if result.ok else head,
status="ok" if ok else "error",
error="" if ok else head,
text=body[:MAX_EVENT_CHARS],
),
)
@@ -501,7 +607,131 @@ async def _run_plan_update(context: ToolContext, args: dict[str, Any]) -> ToolOu
)
# --- Background jobs -----------------------------------------------------------
# A job's files are namespaced by the *calling* chat's id (see jobs.py), and the
# read/stop wrappers are always built from `agent.chat_id`, so a model in one
# chat cannot name another chat's job -- the path simply would not exist. The id
# is still validated as our own hex first, so a crafted id cannot walk out of the
# job directory.
async def _run_job_output(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
agent = _agent(context)
if agent is None:
return _no_machine("job_output")
job_id = str(args.get("id") or "").strip()
if not jobs.valid_id(job_id):
return ToolOutcome(
"There is no job with that id.",
_event("job_output", agent, job_id, status="error", error="Unknown job."),
)
if reason := _permitted(agent, "job_output", RISK_READ):
return _refused("job_output", agent, job_id, reason)
try:
reading = await jobs.read(agent, job_id)
except ExecError as exc:
return ToolOutcome(
exc.message, _event("job_output", agent, job_id, status="error", error=exc.message)
)
if reading.status == "running":
head = "Still running."
elif reading.status == "done":
head = "Finished, and printed nothing." if not reading.body else (
"Finished." if reading.exit_status == 0 else f"Finished, exit {reading.exit_status}."
)
else:
head = "No longer running — no exit status was recorded (it may have been killed)."
content = f"{head}\n\n{reading.body}".strip() if reading.body else head
return ToolOutcome(
content,
_event(
"job_output",
agent,
job_id,
status="ok" if reading.status != "lost" else "error",
text=reading.body[:MAX_EVENT_CHARS],
),
)
async def _run_job_list(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
agent = _agent(context)
if agent is None:
return _no_machine("job_list")
running = jobs.for_chat(agent.chat_id)
if not running:
return ToolOutcome(
"No background jobs in this conversation.",
_event("job_list", agent, "", status="ok", text="none"),
)
lines = [
f"{job.id} [{job.status}] {job.command}"
+ (f" (exit {job.exit_status})" if job.exit_status is not None else "")
for job in running
]
body = "\n".join(lines)
return ToolOutcome(
f"Background jobs:\n{body}",
_event("job_list", agent, f"{len(running)} job(s)", status="ok", text=body),
)
async def _run_job_stop(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
agent = _agent(context)
if agent is None:
return _no_machine("job_stop")
job_id = str(args.get("id") or "").strip()
if not jobs.valid_id(job_id):
return ToolOutcome(
"There is no job with that id.",
_event("job_stop", agent, job_id, status="error", error="Unknown job."),
)
if reason := _permitted(agent, "job_stop", RISK_EXECUTE):
return _refused("job_stop", agent, job_id, reason)
try:
await jobs.stop(agent, job_id)
except ExecError as exc:
return ToolOutcome(
exc.message, _event("job_stop", agent, job_id, status="error", error=exc.message)
)
return ToolOutcome(
f"Stopped job {job_id}.",
_event("job_stop", agent, job_id, status="ok", text="stopped"),
)
def _no_machine(name: str) -> ToolOutcome:
return ToolOutcome(
"This conversation is not connected to a machine.",
{"name": name, "status": "error", "error": "No connection.", "results": []},
)
# --- The definitions -----------------------------------------------------------
def _shell_parameters(background_on: bool) -> dict[str, Any]:
properties: dict[str, Any] = {
"command": {**_STRING, "description": "The command line to run."},
"cwd": {**_STRING, "description": "Where to run it. Defaults to the project directory."},
"timeout": {
"type": "number",
"description": "Seconds to allow. Bounded by the instance settings.",
},
}
if background_on:
properties["background"] = {
"type": "boolean",
"description": (
"Run it detached instead of waiting. It keeps running after this "
"reply; check on it with job_output. Use it for a long install, "
"build or download. A command left waiting is also kept running "
"as a job rather than killed when it hits its timeout."
),
}
return {"type": "object", "properties": properties, "required": ["command"]}
def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
"""The agent tools, bound to one chat's machine.
@@ -525,21 +755,7 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
"answer a prompt, so pass the flags that make a command "
"non-interactive rather than waiting for it to ask."
),
parameters={
"type": "object",
"properties": {
"command": {**_STRING, "description": "The command line to run."},
"cwd": {
**_STRING,
"description": "Where to run it. Defaults to the project directory.",
},
"timeout": {
"type": "number",
"description": "Seconds to allow. Bounded by the instance settings.",
},
},
"required": ["command"],
},
parameters=_shell_parameters(bool(context and context.background)),
run=_run_shell,
risk=RISK_EXECUTE,
),
@@ -779,6 +995,44 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
# about it would mean an approval card per ticked-off task.
risk=RISK_READ,
),
ToolDef(
name="job_output",
family=FAMILY_AGENT,
description=(
"Read what a background job has printed so far, and whether it is "
"still running. Give the id shell_run returned when it was "
"backgrounded."
),
parameters={
"type": "object",
"properties": {"id": {**_STRING, "description": "The job id."}},
"required": ["id"],
},
run=_run_job_output,
risk=RISK_READ,
),
ToolDef(
name="job_list",
family=FAMILY_AGENT,
description="List the background jobs in this conversation and their state.",
parameters={"type": "object", "properties": {}, "required": []},
run=_run_job_list,
risk=RISK_READ,
),
ToolDef(
name="job_stop",
family=FAMILY_AGENT,
description="Stop a background job, killing it and everything it started.",
parameters={
"type": "object",
"properties": {"id": {**_STRING, "description": "The job id."}},
"required": ["id"],
},
run=_run_job_stop,
# It terminates a process on the machine, so it goes through the mode
# table exactly as shell_run does.
risk=RISK_EXECUTE,
),
]
if context is None:
return defs
@@ -790,6 +1044,10 @@ def tool_defs(context: AgentContext | None = None) -> list[ToolDef]:
drop = {"plan_submit"} if context.mode != policy.MODE_PLAN else {"plan_update"}
if not context.plan:
drop.add("plan_update")
# The job tools exist only when commands may run in the background. Offering
# them otherwise is a tool for checking on something that can never exist.
if not context.background:
drop.update({"job_output", "job_list", "job_stop"})
return [tool for tool in defs if tool.name not in drop]