Houmeres/LLeMbas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Files, open beside the conversation

Browse Source 
A third side panel, built the way the terminal is and filled the way the
inspector is: tabs holding open files. Project files over SFTP in an agent
chat; notes, skills, knowledge documents, this chat's text attachments and its
own scratch document everywhere. Read with pygments, edited in a plain
textarea, saved with a conflict check.

A bug found on the way in, and the reason this needed its own read path.
`ssh.read_file` ends in `clean_output`, which strips ANSI escapes and decodes
with errors="replace" -- right for the output of a command, and fatal for an
editor: open a file containing an escape byte, press Save, and you have
silently rewritten it with the escapes gone and every undecodable byte replaced
by U+FFFD. `read_text`/`write_text` decode strictly, report binary rather than
mangling it, carry an mtime:size token for a file that moved underneath, and
refuse an oversize write rather than truncating -- `write_file` truncates
because a model is told how many bytes it wrote, and somebody pressing Save is
not. The model-facing pair is untouched: what it returns is a contract a model
has been shown. A truncated read opens read-only for the mirror-image reason.

Six sources go through one dispatch table, for the reason tool_labels.py is a
table: six independently written permission checks is how one ends up written
slightly differently, and that failure looks like editing somebody else's note.

A save on a project file bypasses agent/policy.py, which makes it the fourth
documented exception to "the modes do not govern the keyboard" and the first
that writes. Same argument as the terminal panel -- whoever owns the credential
could write the file with scp -- but the consequence is larger and is now said
out loud rather than left to be inferred.

The model opens tabs from the file tools it was already calling, so no new
schema and no tokens. It never brings one to the front: an agent reads forty
files in a long reply, and taking the screen each time would drag somebody
through all of them and lose any edit in progress. Only the strip is streamed,
guarded on truthiness so the frame can never blank itself -- an empty one would
close every open tab, the approval card you could press twice with the sign
reversed. Both halves are settled on the server, which is why canvas.js needs
no guard against a swap at all.

No vendored editor. CodeMirror 6 needs a bundler, which is hard rule 1;
CodeMirror 5 would be a larger payload than xterm on every page, and xterm is
the one heavy dependency precisely because it loads only where it can be used.
So: server-rendered highlighting for reading, a textarea for writing, and the
panel says there is no colour while you type rather than pretending.

Also here: a scratch document per chat, with `scratch_write` at RISK_READ on
plan_update's argument, and a test pinning the three numbers that decide a
panel's width -- LAYOUT_BOUNDS drops an unknown variable silently, so a panel
missing from it has a drag handle that works and forgets.

Driven under a DOM stub and against the running application.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
2026-08-04 09:21:03 +02:00
parent 2c914993aa
commit 5766446b84
36 changed files with 2974 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lembas/services/agent/base.py
+53
View File
@@ -107,6 +107,55 @@ class RemoteEntry:
return self.name.startswith(".")
@dataclass(frozen=True)
class RemoteFile:
"""A file as somebody is about to edit it, rather than as a model reads it.
Separate from what `read_file` returns for the same reason `RemoteEntry` is
separate from `list_dir`: the model-facing contract is right for a model and
wrong here. `read_file` runs its result through `clean_output`, which strips
escape sequences and decodes with errors="replace" -- so a file opened
through it and saved back would come out rewritten.
`binary` means there is nothing safe to put in a textarea, and the tab opens
read-only. `truncated` means the same for a different reason: saving back
the first 256KB of a larger file is how the rest of it is deleted.
"""
text: str
size: int = 0
mtime: int = 0
truncated: bool = False
binary: bool = False
@property
def revision(self) -> str:
return revision_of(self.mtime, self.size)
def revision_of(mtime: int, size: int) -> str:
"""An opaque token saying which version of a file was read.
Round-tripped through a hidden field and compared on the way back in. Not a
hash: hashing means reading the whole file again on every save, and this
catches the case it exists for -- somebody else's editor, a build, a
checkout -- without it.
"""
return f"{mtime}:{size}"
class Conflict(Exception):
"""The file moved between being opened and being saved.
Carries the revision found instead, so the card offering Overwrite has
something to compare against.
"""
def __init__(self, found: str = "") -> None:
super().__init__("That file changed after it was opened.")
self.found = found
class Executor(Protocol):
"""How a target is acted on. See `ssh.py`; there is no local variant."""
@@ -116,6 +165,10 @@ class Executor(Protocol):
async def write_file(self, path: str, text: str) -> int: ...
async def read_text(self, path: str, *, max_bytes: int) -> RemoteFile: ...
async def write_text(self, path: str, text: str, *, if_unchanged: str) -> RemoteFile: ...
async def list_dir(self, path: str) -> list[str]: ...
async def scan_dir(self, path: str) -> list[RemoteEntry]: ...
src/lembas/services/agent/ssh.py
+109
View File
@@ -26,17 +26,21 @@ forgot to install it gets a sentence rather than an ImportError at startup.
from __future__ import annotations
import contextlib
import logging
import time
from typing import Any
from lembas.db.models import AUTH_PASSWORD, SshProfile
from lembas.services.agent.base import (
Conflict,
ExecError,
ExecRequest,
ExecResult,
RemoteEntry,
RemoteFile,
clean_output,
revision_of,
)
from lembas.services.crypto import decrypt
@@ -284,6 +288,111 @@ class SshExecutor:
raise self._wrap(exc) from exc
return len(payload)
# --- The same files, for somebody about to edit them ---------------------
# Deliberately not `read_file`/`write_file`, and those two are deliberately
# left exactly as they are: what they return is a contract a model has been
# shown, and it is the right contract for a model.
#
# It is the wrong one for an editor. `read_file` ends in `clean_output`,
# which strips ANSI escape sequences and decodes with errors="replace" --
# correct for the output of a command, and for a file it means that opening
# one containing an escape byte and pressing Save rewrites it with the
# escapes gone and every undecodable byte replaced by U+FFFD. `write_file`
# truncates at MAX_WRITE_BYTES, which a model is told about and a person
# pressing Save is not.
async def read_text(self, path: str, *, max_bytes: int = MAX_READ_BYTES) -> RemoteFile:
"""A file as somebody is about to edit it.
Strict decoding, so a file this cannot represent faithfully is reported
as binary rather than silently mangled into something that would be
saved back. The stat and the read share one connection: connections are
per call, so doing it in two is two handshakes and two authentications
to open one file.
"""
import asyncssh
try:
async with (
self._connect() as conn,
conn.start_sftp_client() as sftp,
sftp.open(self._resolve(path), "rb") as handle,
):
attrs = await handle.stat()
data = await handle.read(max_bytes + 1)
except asyncssh.SFTPNoSuchFile as exc:
raise ExecError(f"There is no file at {path}.") from exc
except asyncssh.SFTPPermissionDenied as exc:
raise ExecError(f"Not allowed to read {path}.") from exc
except (OSError, asyncssh.Error) as exc:
raise self._wrap(exc) from exc
truncated = len(data) > max_bytes
data = data[:max_bytes]
size = int(getattr(attrs, "size", None) or len(data))
mtime = int(getattr(attrs, "mtime", None) or 0)
# A NUL in the first few kilobytes, or anything that will not decode.
# Either way there is nothing safe to put in a textarea.
if b"\0" in data[:8192]:
return RemoteFile("", size, mtime, truncated, binary=True)
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
return RemoteFile("", size, mtime, truncated, binary=True)
return RemoteFile(text, size, mtime, truncated, binary=False)
async def write_text(self, path: str, text: str, *, if_unchanged: str = "") -> RemoteFile:
"""Write a file, refusing if it moved under the editor.
`if_unchanged` is the token `read_text` handed out. The re-stat and the
write happen on one connection, which is the narrowest window SFTP
allows; there is no compare-and-swap here and this does not pretend to
be atomic. It catches what it exists for -- another editor, a build, a
checkout between opening a tab and pressing Save -- and not a race
measured in milliseconds.
Oversize is refused rather than truncated. `write_file` truncates
because a model is told how many bytes it wrote; somebody pressing Save
would lose the tail of their file with nothing said.
"""
import asyncssh
payload = text.encode("utf-8")
if len(payload) > MAX_WRITE_BYTES:
raise ExecError(
f"That is {len(payload) // 1024}KB and the limit is "
f"{MAX_WRITE_BYTES // 1024}KB. Nothing was written."
)
target = self._resolve(path)
try:
async with self._connect() as conn, conn.start_sftp_client() as sftp:
if if_unchanged:
current = ""
with contextlib.suppress(asyncssh.SFTPNoSuchFile):
attrs = await sftp.stat(target)
current = revision_of(
int(getattr(attrs, "mtime", None) or 0),
int(getattr(attrs, "size", None) or 0),
)
if current and current != if_unchanged:
raise Conflict(current)
async with sftp.open(target, "wb") as handle:
await handle.write(payload)
attrs = await sftp.stat(target)
except asyncssh.SFTPPermissionDenied as exc:
raise ExecError(f"Not allowed to write {path}.") from exc
except (OSError, asyncssh.Error) as exc:
raise self._wrap(exc) from exc
return RemoteFile(
text,
len(payload),
int(getattr(attrs, "mtime", None) or 0),
truncated=False,
binary=False,
)
async def list_dir(self, path: str = "") -> list[str]:
import asyncssh
src/lembas/services/agent/tools.py
+45 -3
View File
@@ -343,6 +343,27 @@ def _path_key(agent: AgentContext, path: str) -> str:
return posixpath.normpath(path)
def _canvas(agent: AgentContext, path: str) -> dict[str, str]:
""""This file should be on screen."
Written onto the event because a runner cannot write the message row --
`_persist` is the single writer -- so the generation loop carries it, in
exactly the way it carries a merged plan.
The key comes from `_path_key`, the same normaliser the read-path set uses,
so a tab a model opened and a tab a person opened are one tab rather than
two spellings of the same file.
It never brings the tab to the front; see `canvas.open_tab`. This rides on
calls the model was already making, so it costs no schema and no tokens.
"""
return {
"key": f"agent:{_path_key(agent, path)}",
"title": posixpath.basename(path) or path,
"source": "agent",
}
def _forget_instructions(agent: AgentContext, path: str) -> None:
"""Drop the cached AGENTS.md when the thing just written *is* it.
@@ -395,7 +416,14 @@ async def _run_read(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
return ToolOutcome(
text or "(the file is empty)",
_event("file_read", agent, path, status="ok", text=text[:MAX_EVENT_CHARS]),
_event(
"file_read",
agent,
path,
status="ok",
text=text[:MAX_EVENT_CHARS],
canvas=_canvas(agent, path),
),
)
@@ -436,7 +464,14 @@ async def _run_write(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
index.forget_dir(agent.profile_id, agent.project_dir)
_forget_instructions(agent, path)
event = _event("file_write", agent, path, status="ok", text=f"{written} bytes")
event = _event(
"file_write",
agent,
path,
status="ok",
text=f"{written} bytes",
canvas=_canvas(agent, path),
)
if diffable and before != content:
event["diff"] = patch.render(before, content, path, max_lines=MAX_DIFF_LINES)
@@ -508,7 +543,14 @@ async def _run_edit(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
# cares that it exists, that cache is a copy of what is in it.
_forget_instructions(agent, path)
event = _event("file_edit", agent, path, status="ok", text=f"{written} bytes")
event = _event(
"file_edit",
agent,
path,
status="ok",
text=f"{written} bytes",
canvas=_canvas(agent, path),
)
if diffable:
event["diff"] = patch.render(before, after, path, max_lines=MAX_DIFF_LINES)
src/lembas/services/canvas.py
+480
View File
@@ -0,0 +1,480 @@
"""What is open in the canvas panel, and where its contents come from.
Six sources behind one shape. A tab key is `"<source>:<ref>"` and every source
answers the same two questions -- load this, and save that -- through one table.
A table rather than six branches for the reason `tool_labels.py` and
`sharing.RESOURCE_TYPES` are tables: six independently written permission checks
is how one of them ends up written slightly differently, and the way *that*
failure shows up is somebody editing somebody else's note.
The panel is a person's own hands. A save on an `agent:` tab therefore does not
go through `agent/policy.py`, exactly as the terminal panel and the directory
browser do not: whoever owns the credential could write the file with `scp`.
This is the first of those exceptions that *writes*, which is worth saying out
loud -- Manual mode's "everything is shown to you before it happens" is a promise
about the model, not about the interface.
"""
from __future__ import annotations
import posixpath
from dataclasses import dataclass
from datetime import UTC
from sqlalchemy.orm import Session as DBSession
from lembas.db.models import KIND_AGENT, Attachment, Chat, SshProfile, User
from lembas.security import permissions
from lembas.services import scratch as scratch_service
from lembas.services import settings_store, sharing
from lembas.services.agent import index as index_service
from lembas.services.agent import instructions as instructions_service
from lembas.services.agent import ssh as ssh_service
from lembas.services.agent.base import Conflict, ExecError, revision_of
from lembas.services.library import documents as documents_service
from lembas.services.library import notes as notes_service
from lembas.services.library import skills as skills_service
# How many tabs a chat keeps. A model in a long reply reads forty files, and an
# unbounded strip is a strip nobody can read -- and it would live on the chat
# row forever. Past this the oldest tab that is not in front is dropped.
MAX_TABS = 12
SOURCE_AGENT = "agent"
SOURCE_NOTE = "note"
SOURCE_SKILL = "skill"
SOURCE_DOC = "doc"
SOURCE_FILE = "file"
SOURCE_SCRATCH = "scratch"
class Refused(Exception):
"""This person may not have this, or it is not there any more.
One exception for every source, because the panel answers all of them the
same way: a fragment saying so, in the tab, rather than an error page
swapped into the middle of a chat.
"""
@dataclass(frozen=True)
class Doc:
"""One open file, whatever it actually is underneath."""
key: str
title: str
subtitle: str = ""
text: str = ""
# An opaque token saying which version this was read at, round-tripped
# through a hidden field so a save can refuse a file that moved underneath.
revision: str = ""
writable: bool = False
# A filename or close enough, for choosing a lexer.
language: str = ""
markdown: bool = False
truncated: bool = False
binary: bool = False
@property
def editable(self) -> bool:
"""Whether the box is offered at all.
Not the same as `writable`. Saving back the first 256KB of a larger file
is how the rest of it is deleted, and a binary file has nothing safe to
put in a textarea -- both open read-only however the permissions read.
"""
return self.writable and not self.truncated and not self.binary
def path_key(project_dir: str, path: str) -> str:
"""One name for one file, so `./a.py` and `a.py` open the same tab.
The same normalisation `agent/tools.py:_path_key` applies to the read-path
set, and lifted here so the two cannot disagree: a tab a model opened and a
tab a person opened have to be one tab, or the panel shows the same file
twice and only one of them is the one being saved.
"""
if not posixpath.isabs(path) and project_dir:
path = posixpath.join(project_dir, path)
return posixpath.normpath(path)
def split(key: str) -> tuple[str, str]:
"""`"agent:/srv/a:b.py"` -> `("agent", "/srv/a:b.py")`.
`partition`, not `split`: a path may contain a colon, and a key that lost
half its path would silently open the wrong file.
"""
source, _, ref = (key or "").partition(":")
return source, ref
# --- The tab strip ---------------------------------------------------------------
def tabs_of(chat: Chat) -> list[dict]:
return list((chat.canvas_json or {}).get("tabs") or [])
def active_of(chat: Chat) -> str:
return str((chat.canvas_json or {}).get("active") or "")
def open_tab(state: dict, tab: dict, *, activate: bool = True) -> dict:
"""Add a tab, and optionally bring it to the front. Mutates `state`.
Mutating rather than returning a copy because the generation loop folds
several of these into one snapshot within a round: two `file_read` calls
that each read the state and wrote it back would leave only the second.
That is the lost update `plan_update` documents, in a different place.
`activate=False` is what a *model* opening a tab does, and it is the whole
of how this feature avoids being infuriating. An agent reads forty files in
a long reply; if each one took the panel, somebody reading the third would
be dragged through the other thirty-seven, and anybody halfway through an
edit would lose it. So the model fills the strip and the person decides
what is in front. A tab they open themselves activates, because opening
something and not being shown it is the opposite failure.
"""
key = str(tab.get("key") or "")
if not key:
return state
tabs = [t for t in (state.get("tabs") or []) if t.get("key") != key]
tabs.append({
"key": key,
"title": str(tab.get("title") or key)[:120],
"source": str(tab.get("source") or split(key)[0]),
})
# Evict from the front, and never the tab in front or the one just opened.
# A model reading its way through a project must not close the file
# somebody is looking at.
keep = {key, str(state.get("active") or "")}
while len(tabs) > MAX_TABS:
victim = next((t for t in tabs if t["key"] not in keep), None)
if victim is None:
break
tabs.remove(victim)
state["tabs"] = tabs
if activate or not state.get("active"):
# Not activating an empty panel would leave tabs with nothing in front,
# which reads as a panel that failed to load.
state["active"] = key
return state
def close_tab(state: dict, key: str) -> dict:
tabs = [t for t in (state.get("tabs") or []) if t.get("key") != key]
state["tabs"] = tabs
if state.get("active") == key:
state["active"] = tabs[-1]["key"] if tabs else ""
return state
def merge(stored: dict | None, live: dict | None) -> dict:
"""Fold a reply's tabs into whatever the row says now.
A union rather than an overwrite. `_persist` is the single writer, and the
snapshot it holds was taken when the reply began -- so overwriting would
drop a tab the person opened by hand while the reply was running.
"""
state = {
"tabs": list((stored or {}).get("tabs") or []),
"active": (stored or {}).get("active") or "",
}
for tab in (live or {}).get("tabs") or []:
# Never activating: what the row says is in front is what the person
# last chose, and a reply that finishes ten minutes later must not move
# it. The reply's own `active` is deliberately not consulted.
open_tab(state, tab, activate=False)
return state
# --- Which sources this chat may reach ---------------------------------------------
def agent_ready(db: DBSession, user: User, chat: Chat | None) -> SshProfile | None:
"""The profile an `agent:` tab would use, or None.
Everything `_terminal_enabled` checks except `agent.terminal`. Reading and
writing project files is what `tools.agent` is named after, and somebody who
may have a model write a file may certainly write one themselves.
Re-derived on every request. The template flag of the same name is
decoration; this is the control.
"""
if chat is None or chat.kind != KIND_AGENT or not chat.ssh_profile_id:
return None
if not permissions.has(db, user, "tools.agent"):
return None
if not settings_store.agents(db).get("enabled"):
return None
if ssh_service.available() != "":
return None
profile = db.get(SshProfile, chat.ssh_profile_id)
if profile is None or profile.owner_id != user.id or not profile.enabled:
return None
if not profile.host_key:
return None
return profile
def _executor(db: DBSession, user: User, chat: Chat) -> ssh_service.SshExecutor:
profile = agent_ready(db, user, chat)
if profile is None:
raise Refused(
"This chat has no connection you can reach. Check the connection's "
"host key on the Connections page if it has not been accepted yet."
)
return ssh_service.SshExecutor(ssh_service.spec_from(profile), chat.project_dir)
# --- Loading ------------------------------------------------------------------------
async def _load_agent(db: DBSession, user: User, chat: Chat, ref: str) -> Doc:
executor = _executor(db, user, chat)
path = path_key(chat.project_dir, ref)
try:
found = await executor.read_text(path)
except ExecError as exc:
raise Refused(str(exc)) from exc
return Doc(
key=f"{SOURCE_AGENT}:{path}",
title=posixpath.basename(path) or path,
subtitle=path,
text=found.text,
revision=found.revision,
writable=True,
language=posixpath.basename(path),
markdown=path.lower().endswith((".md", ".markdown")),
truncated=found.truncated,
binary=found.binary,
)
async def _load_note(db: DBSession, user: User, chat: Chat, ref: str) -> Doc:
_needs_library(db, user)
note = notes_service.get(db, ref, user)
if note is None:
raise Refused("That note is not there any more.")
return Doc(
key=f"{SOURCE_NOTE}:{note.id}",
title=note.title or "Note",
subtitle="Note",
text=note.body or "",
revision=_stamp(note, note.body or ""),
writable=sharing.can_write(note, user),
language="note.md",
markdown=True,
)
async def _load_skill(db: DBSession, user: User, chat: Chat, ref: str) -> Doc:
_needs_library(db, user)
skill = skills_service.get(db, ref, user)
if skill is None:
raise Refused("That skill is not there any more.")
return Doc(
key=f"{SOURCE_SKILL}:{skill.id}",
title=skill.name or "Skill",
subtitle="Skill",
text=skill.body or "",
revision=_stamp(skill, skill.body or ""),
writable=sharing.can_write(skill, user),
language="skill.md",
markdown=True,
)
async def _load_doc(db: DBSession, user: User, chat: Chat, ref: str) -> Doc:
_needs_library(db, user)
document = documents_service.get(db, ref, user)
if document is None:
raise Refused("That document is not there any more.")
return Doc(
key=f"{SOURCE_DOC}:{document.id}",
title=document.title or document.filename or "Document",
subtitle="Knowledge document",
text=document.extracted_text or document.extraction_error or "",
revision=_stamp(document, document.extracted_text or ""),
writable=documents_service.can_write(document, user),
language=document.filename or "",
markdown=(document.filename or "").lower().endswith((".md", ".markdown")),
truncated=bool(document.truncated),
)
async def _load_file(db: DBSession, user: User, chat: Chat, ref: str) -> Doc:
attachment = db.get(Attachment, ref)
if attachment is None or attachment.user_id != user.id:
raise Refused("That attachment is not there any more.")
# Belonging to this conversation, so a canvas cannot browse another one's
# files by id. `chat_id` covers one still in the composer; the message check
# covers one that has been sent.
if attachment.chat_id != chat.id:
raise Refused("That attachment belongs to another chat.")
return Doc(
key=f"{SOURCE_FILE}:{attachment.id}",
title=attachment.filename or "Attachment",
subtitle=attachment.source_path or "Attachment",
text=attachment.extracted_text or attachment.extraction_error or "",
# Read-only, and not for want of a write path: `DELETE /api/files/{id}`
# already refuses once the attachment has been sent, because it would
# rewrite a message somebody already read. Editing is the same act with
# a quieter failure.
writable=False,
language=attachment.filename or "",
markdown=(attachment.filename or "").lower().endswith((".md", ".markdown")),
truncated=bool(attachment.truncated),
)
async def _load_scratch(db: DBSession, user: User, chat: Chat, ref: str) -> Doc:
if ref != chat.id:
raise Refused("That scratch document belongs to another chat.")
doc = scratch_service.for_chat(db, chat)
return Doc(
key=f"{SOURCE_SCRATCH}:{chat.id}",
title=doc.title or "Scratch",
subtitle="This chat's scratch document",
text=doc.body or "",
revision=_stamp(doc, doc.body or ""),
writable=True,
language="scratch.md",
markdown=True,
)
# --- Saving --------------------------------------------------------------------------
async def _save_agent(db: DBSession, user: User, chat: Chat, ref: str, text: str, revision: str):
executor = _executor(db, user, chat)
path = path_key(chat.project_dir, ref)
try:
await executor.write_text(path, text, if_unchanged=revision)
except ExecError as exc:
raise Refused(str(exc)) from exc
profile = agent_ready(db, user, chat)
if profile is not None:
# Unconditionally, unlike `file_edit` -- whose skip is an optimisation
# for the model's hot path on the grounds that the file was already
# there. The canvas can create one, and a listing known to be wrong is
# what the cache note warns about.
index_service.forget_dir(profile.id, chat.project_dir)
if instructions_service.is_instruction_file(path, chat.project_dir):
instructions_service.forget(profile.id, chat.project_dir)
return await _load_agent(db, user, chat, path)
async def _save_note(db: DBSession, user: User, chat: Chat, ref: str, text: str, revision: str):
note = notes_service.get(db, ref, user)
if note is None:
raise Refused("That note is not there any more.")
if not sharing.can_write(note, user):
raise Refused("That note is not yours to change.")
_check_stamp(note, note.body or "", revision)
notes_service.update(db, note, body=text)
return await _load_note(db, user, chat, ref)
async def _save_skill(db: DBSession, user: User, chat: Chat, ref: str, text: str, revision: str):
skill = skills_service.get(db, ref, user)
if skill is None:
raise Refused("That skill is not there any more.")
if not sharing.can_write(skill, user):
raise Refused("That skill is not yours to change.")
_check_stamp(skill, skill.body or "", revision)
# Snapshots into a SkillRevision first, which is why a skill needs no
# conflict story beyond the token: a clobber is recoverable.
skills_service.update(db, skill, body=text, note="Edited in the canvas")
return await _load_skill(db, user, chat, ref)
async def _save_doc(db: DBSession, user: User, chat: Chat, ref: str, text: str, revision: str):
document = documents_service.get(db, ref, user)
if document is None:
raise Refused("That document is not there any more.")
if not documents_service.can_write(document, user):
raise Refused("That document is not yours to change.")
_check_stamp(document, document.extracted_text or "", revision)
documents_service.set_text(db, document, text)
return await _load_doc(db, user, chat, ref)
async def _save_scratch(db: DBSession, user: User, chat: Chat, ref: str, text: str, revision: str):
if ref != chat.id:
raise Refused("That scratch document belongs to another chat.")
doc = scratch_service.for_chat(db, chat)
_check_stamp(doc, doc.body or "", revision)
scratch_service.update(db, doc, body=text)
return await _load_scratch(db, user, chat, ref)
# --- One table -------------------------------------------------------------------------
_SOURCES: dict[str, tuple] = {
SOURCE_AGENT: (_load_agent, _save_agent),
SOURCE_NOTE: (_load_note, _save_note),
SOURCE_SKILL: (_load_skill, _save_skill),
SOURCE_DOC: (_load_doc, _save_doc),
SOURCE_FILE: (_load_file, None),
SOURCE_SCRATCH: (_load_scratch, _save_scratch),
}
async def load(db: DBSession, user: User, chat: Chat, key: str) -> Doc:
source, ref = split(key)
entry = _SOURCES.get(source)
if entry is None or not ref:
raise Refused("There is nothing to open here.")
return await entry[0](db, user, chat, ref)
async def save(
db: DBSession, user: User, chat: Chat, key: str, text: str, revision: str = ""
) -> Doc:
source, ref = split(key)
entry = _SOURCES.get(source)
if entry is None or not ref:
raise Refused("There is nothing to save here.")
saver = entry[1]
if saver is None:
raise Refused("This one can only be read.")
return await saver(db, user, chat, ref, text, revision)
# --- Small shared pieces ------------------------------------------------------------------
def _needs_library(db: DBSession, user: User) -> None:
if not permissions.has(db, user, "library.use"):
raise Refused("You do not have access to the library.")
def _stamp(row, text: str) -> str:
"""A revision token for a database row.
`updated_at` alone would not move for two saves inside one clock tick, so
the length rides along -- the same pairing the file token uses, and for the
same reason. The text is passed in rather than guessed at: a note keeps it
in `body` and a document in `extracted_text`, and a getattr chain that
silently found neither would hand every row the same token.
"""
when = getattr(row, "updated_at", None)
if when is not None and when.tzinfo is None:
# SQLite does not store the offset, so a row loaded from disk comes back
# naive while one still in the session's identity map keeps the tzinfo
# it was created with -- and `.timestamp()` reads a naive value as local
# time. Without this the same row yields two different tokens depending
# on where it was loaded, and every save outside UTC would report a
# conflict that is not there. The same normalisation
# `compaction.moment` makes, for the same reason.
when = when.replace(tzinfo=UTC)
return revision_of(int(when.timestamp()) if when else 0, len(text or ""))
def _check_stamp(row, text: str, revision: str) -> None:
"""Refuse a save whose token no longer matches. An empty token overwrites.
Empty is what Overwrite on the conflict card sends: somebody has been shown
both versions and chosen. Never save silently over a change; never discard
silently either.
"""
if revision and _stamp(row, text) != revision:
raise Conflict(_stamp(row, text))
src/lembas/services/generation.py
+25
View File
@@ -28,6 +28,7 @@ from sqlalchemy import select
from lembas.db.models import KIND_AGENT, ROLE_ASSISTANT, ROLE_USER, Chat, Message, User
from lembas.db.session import session_scope
from lembas.services import canvas as canvas_service
from lembas.services import chat as chat_service
from lembas.services import compaction as compaction_service
from lembas.services import interaction, settings_store, tokens, tool_labels
@@ -167,6 +168,12 @@ class Generation:
# `_persist` stays one writer with one rule; only this decides whether the
# tools are withdrawn for a final round.
plan_final: bool = False
# Which files this reply has put in the canvas panel. Seeded once from
# `chat.canvas_json` where `_run` already has the chat loaded, then mutated
# in place -- two `file_read` calls in one round that each re-read the row
# would leave only the second, which is the lost update `plan` above
# documents. Folded back by `_persist`, the single writer.
canvas: dict = field(default_factory=dict)
# The queue, seen from the reply's side. `drained` says this reply's ending
# handed the next waiting prompt to a fresh one; `injected_ids` names the
# prompts taken into *this* reply between two rounds of tool calls. Both are
@@ -413,6 +420,12 @@ async def _run(generation: Generation) -> None:
# worth asking for. Read here with the rest, because titling happens
# after this session has closed.
title_from_prompt = chat.kind == KIND_AGENT
# Seeded once, here, where the chat is already loaded. Mutated from
# then on; see the field's own note.
generation.canvas = {
"tabs": list((chat.canvas_json or {}).get("tabs") or []),
"active": (chat.canvas_json or {}).get("active") or "",
}
# Read here, with the rest, because titling happens after this
# session has closed and must not open another one.
title_prompt = prompts_service.resolve(db, "task.title")
@@ -643,6 +656,13 @@ async def _run(generation: Generation) -> None:
generation.tool_events.append(outcome.event)
generation.output_bytes += len(outcome.content)
messages.append(tools_service.tool_turn(call, outcome.content))
if opened := outcome.event.get("canvas"):
# A runner cannot write the message row, so the loop carries
# this exactly as it carries a merged plan. Never activated:
# an agent reads forty files in a long reply, and dragging
# somebody through all of them -- or away from a file they
# are editing -- is what makes a panel like this unusable.
canvas_service.open_tab(generation.canvas, opened, activate=False)
if outcome.event.get("plan"):
generation.plan = outcome.event["plan"]
# Only `plan_submit` sets this. `plan_update` writes the
@@ -1591,6 +1611,11 @@ def _persist(generation: Generation, title: str, elapsed: float) -> None:
message.reasoning_ms = generation.reasoning_ms
message.tool_calls_json = generation.tool_events
message.plan_json = generation.plan or {}
if generation.canvas.get("tabs"):
# A union with whatever the row says *now*, not an overwrite:
# the snapshot above was seeded when the reply began, and
# somebody may have opened a tab by hand since.
chat.canvas_json = canvas_service.merge(chat.canvas_json, generation.canvas)
if generation.plan:
# This bubble now carries the plan in force, and the chat points
# at it so the harness can find it with one primary-key lookup
src/lembas/services/library/documents.py
+40
View File
@@ -254,6 +254,46 @@ def get(db: DBSession, document_id: str, user: User | None) -> Document | None:
return document
def can_write(document: Document, user: User | None) -> bool:
"""Whether this person may change a document's text.
Ownership, through the same helper every other library store uses. Sharing
grants **reading only**, so being able to see a document through somebody
else's base is never enough to rewrite it -- and reading is already settled
by `get`, which resolves visibility through the base.
Its own function rather than `sharing.can_write` at the call site because
`Document` is the one store whose visibility does not come from itself, and
a reader arriving at a bare `sharing.can_write(document, …)` would have to
go and check whether that is the right question.
"""
return sharing.can_write(document, user)
def set_text(db: DBSession, document: Document, text: str) -> Document:
"""Replace the extracted text a person reads and a model searches.
The stored file is untouched: the bytes are the record, and this is what was
made of them. That is the same line PDF extraction draws -- extracted once
at upload, so a reply cannot change because a parser was upgraded -- and it
is why editing this is safe for transcripts: `files.copy_document` copies
the text when a document is attached, so an edit only changes what future
searches find.
`extraction_error` is cleared, because replacing a failed extraction by hand
is the main reason to want this at all; leaving the old apology beside the
new text would be the page contradicting itself.
The commit fires the `documents_fts` UPDATE trigger, so search stays correct
with nothing else to do. See `db/migrations.py:ensure_fts`.
"""
document.extracted_text = text[:files_service.MAX_EXTRACTED_CHARS]
document.truncated = len(text) > files_service.MAX_EXTRACTED_CHARS
document.extraction_error = ""
db.commit()
return document
def search(
db: DBSession,
user: User | None,
src/lembas/services/markdown.py
+38 -1
View File
@@ -19,7 +19,7 @@ import nh3
from markdown_it import MarkdownIt
from pygments import highlight
from pygments.formatters import HtmlFormatter
from pygments.lexers import get_lexer_by_name, guess_lexer
from pygments.lexers import get_lexer_by_name, get_lexer_for_filename, guess_lexer
from pygments.util import ClassNotFound
# Class-based highlighting; the colours come from theme tokens in chat.css, so
@@ -95,6 +95,43 @@ def _render_fence(tokens, idx, _options, _env) -> str:
)
def highlight_code(text: str, filename: str = "") -> str:
"""A whole file, class-highlighted, for the canvas panel to read.
Here rather than in a module of its own because `markdown.py` is where
pygments lives and `_FORMATTER` is already configured: a second formatter
would mean a second set of class names and a second thing to theme, and the
`.pg-*` rules would then be right about code fences and wrong about files.
Pygments' `HtmlFormatter` escapes what it is given, which is what makes this
the one call the canvas templates mark `|safe`. The content came off
somebody else's disk, so that property is the whole of the argument -- if
the lexer cannot be found the text is escaped by hand instead, never passed
through.
Chooses by filename, because that is what the canvas has: a lexer guessed
from contents is confidently wrong on short files, and there is no fence
info string here to read a language out of.
"""
if not text:
return ""
lexer = None
if filename:
try:
lexer = get_lexer_for_filename(filename, stripall=False)
except (ClassNotFound, ValueError):
lexer = None
if lexer is None and len(text) > 200:
try:
lexer = guess_lexer(text)
except (ClassNotFound, ValueError):
lexer = None
body = nh3.clean_text(text) if lexer is None else highlight(text, lexer, _FORMATTER)
return f'<pre class="canvas__code"><code>{body}</code></pre>'
@functools.lru_cache(maxsize=1)
def _parser() -> MarkdownIt:
md = MarkdownIt("commonmark", {"linkify": True, "typographer": False})
src/lembas/services/prompts.py
+21
View File
@@ -924,6 +924,27 @@ BUILTIN: tuple[Fragment, ...] = (
"since that is all you will see next time."
),
),
Fragment(
key="tool.scratch",
label="The scratch document",
group=GROUP_TOOLS,
order=245,
families=("scratch",),
hint="Appears when scratch_write is offered. The point worth making to "
"a model is the one it cannot infer from the schema: this is *watched* "
"while it is written, so building something up here is visible work "
"rather than a result announced at the end — and it is not searchable "
"afterwards, which is what keeps it from being used as a note.",
default=(
"- This chat has a scratch document, open beside the conversation and visible "
"to the person as you write it. Use scratch_write for something you build up "
"as you work — a draft, a table of findings, a list you keep adding to — "
"rather than repeating the whole thing in each reply. Append unless you mean "
"to start again. They can edit it themselves and attach it to a later message. "
"It belongs to this chat and cannot be searched afterwards, so anything worth "
"keeping beyond it is a note."
),
),
# --- Context -------------------------------------------------------------
Fragment(
key="context.knowledge_scope",
src/lembas/services/scratch.py
+82
View File
@@ -0,0 +1,82 @@
"""A chat's own working surface.
One text artefact per chat, written by the model through `scratch_write` and by
the person through the canvas panel, and handed to a message as an ordinary
attachment when it is ready.
Not a note. A note is a durable artefact of the reader's that outlives the chat
and is searchable; this is the chat's own record of what it is working on, which
is the line `plan_update` sits on rather than `notes_edit`. It is deliberately
not injected as context every turn either -- that is what a memory is for, and a
working document injected whole on every request is how a window fills up.
"""
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.orm import Session as DBSession
from lembas.db.models import AUTHOR_USER, Chat, ScratchDoc
# Between a note's 40k and an attachment's 120k. Large enough to hold a draft
# somebody is actually working on, small enough that attaching one does not
# quietly cost most of a context window.
MAX_BODY_CHARS = 100_000
MAX_TITLE_CHARS = 300
def get(db: DBSession, chat: Chat) -> ScratchDoc | None:
"""The chat's pad, or None. No side effect.
Separate from `for_chat` because the harness and the mention picker ask
whether there is one, and a question must not create the thing it asks
about -- otherwise every chat ever opened acquires an empty row.
"""
return db.scalar(select(ScratchDoc).where(ScratchDoc.chat_id == chat.id))
def for_chat(db: DBSession, chat: Chat) -> ScratchDoc:
"""The chat's pad, made if it is not there yet."""
existing = get(db, chat)
if existing is not None:
return existing
doc = ScratchDoc(chat_id=chat.id, user_id=chat.user_id)
db.add(doc)
db.commit()
return doc
def update(
db: DBSession,
doc: ScratchDoc,
*,
body: str | None = None,
title: str | None = None,
author: str = AUTHOR_USER,
) -> ScratchDoc:
"""Replace what is in the pad. Absent arguments are left alone.
The body is **not** stripped, unlike a note's. This is a document somebody
is editing, and trailing whitespace they typed is theirs -- a save that
silently trims the line you are standing on is the kind of thing that makes
an editor feel broken.
"""
if title is not None and title.strip():
doc.title = title.strip()[:MAX_TITLE_CHARS]
if body is not None:
doc.body = body[:MAX_BODY_CHARS]
doc.author = author
db.commit()
return doc
def append(db: DBSession, doc: ScratchDoc, text: str, *, author: str) -> ScratchDoc:
"""Add to the end, with a blank line between what was there and what is new.
Its own function rather than the caller reading and concatenating, because
two calls in one round would otherwise each read the same body and the
second would drop the first -- the same lost update `plan_update` documents.
"""
existing = doc.body or ""
joined = f"{existing.rstrip()}\n\n{text}" if existing.strip() else text
return update(db, doc, body=joined, author=author)
src/lembas/services/tool_labels.py
+3
View File
@@ -61,6 +61,7 @@ LABELS: dict[str, str] = {
"notes_create": "Note written",
"notes_edit": "Note updated",
"notes_delete": "Note deleted",
"scratch_write": "Canvas written",
"memory_add": "Memory saved",
"memory_forget": "Memory removed",
"skill_get": "Skill read",
@@ -92,6 +93,7 @@ ICONS: dict[str, str] = {
"notes_create": "pencil",
"notes_edit": "pencil",
"notes_delete": "trash",
"scratch_write": "file-text",
"memory_add": "star",
"memory_forget": "trash",
"skill_get": "sparkle",
@@ -127,6 +129,7 @@ ACTIONS: dict[str, str] = {
"notes_create": "Write a note",
"notes_edit": "Change a note",
"notes_delete": "Delete a note",
"scratch_write": "Write in the canvas",
"memory_add": "Remember something",
"memory_forget": "Forget something",
"skill_get": "Read a skill",
src/lembas/services/tools.py
+89 -2
View File
@@ -35,6 +35,7 @@ from sqlalchemy.orm import Session as DBSession
from lembas.db.models import AUTHOR_MODEL, Chat, User
from lembas.db.session import session_scope
from lembas.services import prompts as prompts_service
from lembas.services import scratch as scratch_service
from lembas.services import search as search_service
from lembas.services import settings_store
from lembas.services.library import documents as documents_service
@@ -94,6 +95,13 @@ FAMILY_MCP = "mcp"
# is the only tool the model cannot resolve by itself.
FAMILY_ASK = "ask"
# The chat's own working surface -- the canvas panel's scratch document.
# Deliberately not part of `notes`: a note is a durable artefact of the reader's
# that outlives the chat and is searchable, while this is the chat's own record
# of what it is doing, which is the line `plan_update` sits on. It is also its
# own switch, because narrowing notes off must not silently take the pad too.
FAMILY_SCRATCH = "scratch"
# Acting on the machine an agent chat is pointed at. Offered only when the chat
# is one, has a usable connection, and the feature is switched on -- see
# services/agent/session.py:resolve, which answers all three at once.
@@ -107,6 +115,7 @@ FAMILIES = (
FAMILY_NOTES,
FAMILY_MEMORY,
FAMILY_SKILLS,
FAMILY_SCRATCH,
FAMILY_ASK,
FAMILY_AGENT,
)
@@ -507,6 +516,51 @@ async def _run_notes_delete(context: ToolContext, args: dict[str, Any]) -> ToolO
)
# --- The chat's scratch document ---------------------------------------------
async def _run_scratch_write(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
"""Write into the pad the person can see beside the conversation.
Opens its own session, like every other runner: a generation outlives the
session that resolved it.
`append` is a service function rather than a read-and-concatenate here,
because two calls in one round would otherwise each read the same body and
the second would drop the first.
"""
with session_scope() as db:
chat = db.get(Chat, context.chat_id) if context.chat_id else None
if chat is None:
return ToolOutcome(
"There is no chat to write into.",
{"name": "scratch_write", "status": "error", "error": "No chat."},
)
doc = scratch_service.for_chat(db, chat)
text = str(args.get("text") or "")
if str(args.get("mode") or "append").strip().lower() == "replace":
scratch_service.update(db, doc, body=text, author=AUTHOR_MODEL)
what = "Replaced"
else:
scratch_service.append(db, doc, text, author=AUTHOR_MODEL)
what = "Added to"
return ToolOutcome(
f"{what} the scratch document ({len(doc.body)} characters). "
"It is on screen beside the conversation.",
{
"name": "scratch_write",
"query": doc.title,
"status": "ok",
"results": [],
# Opens the tab, the same way a file tool does. Never brings it
# to the front -- see `canvas.open_tab`.
"canvas": {
"key": f"scratch:{chat.id}",
"title": doc.title,
"source": "scratch",
},
},
)
# --- Memory ------------------------------------------------------------------
async def _run_memory_add(context: ToolContext, args: dict[str, Any]) -> ToolOutcome:
content = str(args.get("content") or "").strip()
@@ -819,6 +873,38 @@ REGISTRY: dict[str, ToolDef] = {
run=_run_notes_delete,
risk=RISK_WRITE,
),
ToolDef(
name="scratch_write",
family=FAMILY_SCRATCH,
description=(
"Write into this chat's scratch document, which the person can "
"see and edit beside the conversation. Use it for something you "
"are building up as you work — a draft, a table of findings, a "
"list you keep adding to — rather than putting it in the reply "
"and rewriting the whole thing each turn. It is not searchable "
"later and belongs to this chat alone; use a note for anything "
"worth keeping beyond it."
),
parameters=_object(
{
"mode": {
**_STRING,
"enum": ["append", "replace"],
"description": "append is the default.",
},
"text": {**_STRING, "description": "Markdown."},
},
["text"],
),
run=_run_scratch_write,
# What a tool does to the *world the four modes govern*, which is the
# machine -- and this cannot touch it. RISK_WRITE would put an
# approval card on screen every time the model jotted a paragraph,
# which is exactly the interruption batching exists to prevent. The
# same argument `plan_update` carries. An administrator who
# disagrees puts it in `deny_default`.
risk=RISK_READ,
),
ToolDef(
name="memory_add",
family=FAMILY_MEMORY,
@@ -990,11 +1076,12 @@ def _family_allowed(
# attach path keeps working, because that one is a person's instruction
# rather than a model's choice.
return bool(allowed.get("tools.fetch") and config.get("fetch_enabled"))
if gate in (FAMILY_CUSTOM, FAMILY_MCP, FAMILY_ASK, FAMILY_AGENT):
if gate in (FAMILY_CUSTOM, FAMILY_MCP, FAMILY_ASK, FAMILY_AGENT, FAMILY_SCRATCH):
# Deliberately without `library.use`: an HTTP endpoint an administrator
# wrote has nothing to do with this person's own documents and notes,
# and requiring the library permission for it would be a coincidence of
# naming rather than a rule. The same goes for being asked a question.
# naming rather than a rule. The same goes for being asked a question,
# and for a pad that belongs to this chat and goes nowhere else.
return bool(allowed.get(f"tools.{gate}"))
return bool(allowed.get(f"tools.{gate}") and allowed.get("library.use"))