A directory the model knows about, and @ to name a file in it

An agent chat used to open with the model knowing the name of a machine and
nothing about what was on it, so the first two rounds of every reply went on
finding out. It now gets a listing: one read-only command, `git ls-files` where
that works and `find` otherwise, falling back to an SFTP walk that always does.
git first because a repository already carries somebody's considered list of
what is not part of the project, and reproducing it by hand is how an index
ends up mostly build output.

The listing is budgeted rather than dumped. A tree of a thousand files is worse
than no tree -- it costs the window on every request forever and buries the four
names that mattered -- so directories that will not fit are shown as a count and
the model is told to open one itself. Collapsing picks the deepest and largest
first: by saving alone it would take `src/` before `src/web/static/vendor/`,
because it contains it, and lose every name worth having.

Read from a cache and never fetched. `harness.context_variables` is synchronous
and sits on the request path; the walk happens in the generation setup, which is
async and already doing network work, with a short wait. A chat whose first
reply outruns its first walk simply has no listing that turn and the fragment
disappears rather than appearing as an empty heading.

Then `@`, over the same index and over the library, and `/` for commands with an
Alt-based keyboard for the same jobs. A mentioned file arrives as contents, not
a reference -- a small model asked to call file_read often does not bother -- and
it arrives with its absolute path and the machine it came from, because a model
handed `main.py` cannot tell which of four it is and cannot name it back when
asked to change something.

The rule that matters for `/`: a message that merely starts with a slash still
sends. `//` escapes and an unrecognised command is posted as written. Swallowing
somebody's message is a much worse failure than an unknown command.

Two exceptions to Manual mode now, not one. Browsing and indexing are a person
acting, not a model, so neither passes through policy.py -- the same argument
the terminal panel rests on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/lembas/services/agent/index.py

@@ -0,0 +1,507 @@
+"""What is in a project directory, for the picker and for the model.
+
+Two things want this list. The `@` picker needs something to filter, and a
+model working in a directory should know roughly what is in it rather than
+spending its first two rounds finding out. Both want the same walk, so it
+happens once and is cached.
+
+**Three ways of getting it, in order.** `git ls-files` first, because most
+project directories are repositories and it applies `.gitignore` for free --
+without which the answer for a Node project is forty thousand paths under
+`node_modules`. Then `find`, with the usual noise pruned by hand. Then a
+recursive SFTP walk, which always works and costs a round trip per directory.
+
+**Two commands run here, and neither goes through `agent/policy.py`.** That is
+deliberate and it is the same argument the terminal panel and the directory
+browser rest on: this is LLeMbas listing a directory on somebody's behalf, not
+a model choosing to run something. Both are read-only, both are built here
+rather than assembled from anything a model said, and the project directory is
+configuration rather than input. It is still an exception to Manual mode's
+"everything is shown to you before it happens", and it is written down in
+CLAUDE.md next to the others.
+
+**Nothing here is trusted.** Filenames come off somebody else's machine and end
+up inside a system prompt, so they are stripped of control characters, capped
+in length, capped in number, and never interpreted.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import re
+import time
+from dataclasses import dataclass, field
+
+from lembas.services.agent.base import ExecError, ExecRequest, Executor
+
+log = logging.getLogger(__name__)
+
+# How many paths are kept. Past this the index says it was truncated, which the
+# rendering repeats to the model -- "there is nothing else here" and "I stopped
+# looking" are different answers and it must not give the first for the second.
+MAX_ENTRIES = 20_000
+# One path. Longer than any real one and shorter than an attack.
+MAX_PATH = 400
+# How long a walk may take before it is abandoned. The index is a convenience;
+# a chat must never sit waiting for one.
+BUILD_TIMEOUT = 20.0
+# Output budget for the listing commands. Twenty thousand paths at forty
+# characters is 800KB, so this has room and still refuses a runaway.
+MAX_OUTPUT = 2 * 1024 * 1024
+
+# How long a built index is reused, and how many are kept at once. A project
+# directory changes under you -- the model writes files into it -- so this is
+# short. `refresh` exists for when short is not short enough.
+TTL = 300.0
+MAX_CACHED = 64
+
+# How deep the SFTP fallback goes, and how many directories it will open. It is
+# a round trip per directory, so an unbounded walk of somebody's home directory
+# would take minutes and achieve nothing.
+SFTP_MAX_DEPTH = 6
+SFTP_MAX_DIRS = 400
+
+# Pruned from the `find` and SFTP paths. Not applied to `git ls-files`, which
+# has already applied the repository's own rules and where a checked-in
+# `vendor/` is checked in on purpose -- this project's own hash-pinned browser
+# libraries live in one.
+IGNORED = (
+    ".git",
+    ".hg",
+    ".svn",
+    "node_modules",
+    "__pycache__",
+    ".venv",
+    "venv",
+    ".mypy_cache",
+    ".pytest_cache",
+    ".ruff_cache",
+    ".tox",
+    ".next",
+    ".nuxt",
+    ".gradle",
+    ".terraform",
+    "target",
+    "dist",
+    "build",
+    ".DS_Store",
+)
+
+# Control characters, including the escape that would let a filename repaint
+# the transcript it is quoted in.
+_CONTROL = re.compile(r"[\x00-\x1f\x7f-\x9f]")
+
+
+@dataclass(frozen=True)
+class ProjectIndex:
+    """A snapshot of what was in a directory, and how it was found out."""
+
+    paths: tuple[str, ...] = ()
+    total: int = 0
+    truncated: bool = False
+    source: str = ""
+    built_at: float = field(default=0.0)
+
+    @property
+    def ok(self) -> bool:
+        return bool(self.paths)
+
+
+# --- Building ----------------------------------------------------------------
+def _clean(raw: str) -> str:
+    """One path, made safe to put in a prompt and in an attribute."""
+    path = _CONTROL.sub("", raw.strip()).lstrip("./")
+    return path[:MAX_PATH]
+
+
+def _collect(output: str) -> tuple[tuple[str, ...], int, bool]:
+    seen: set[str] = set()
+    paths: list[str] = []
+    total = 0
+    for line in output.splitlines():
+        path = _clean(line)
+        if not path or path in seen:
+            continue
+        total += 1
+        if len(paths) < MAX_ENTRIES:
+            seen.add(path)
+            paths.append(path)
+    paths.sort()
+    return tuple(paths), total, total > len(paths)
+
+
+async def _from_git(executor: Executor, project_dir: str) -> ProjectIndex | None:
+    """Tracked and untracked files, minus whatever `.gitignore` excludes.
+
+    `--exclude-standard` is what makes this worth trying first: the repository
+    already carries somebody's considered list of what is not part of the
+    project, and reproducing it by hand is how an index ends up ninety percent
+    build output.
+    """
+    result = await executor.run(
+        ExecRequest(
+            command="git ls-files -c -o --exclude-standard 2>/dev/null",
+            cwd=project_dir,
+            timeout=BUILD_TIMEOUT,
+            max_bytes=MAX_OUTPUT,
+        )
+    )
+    if not result.ok or not result.output.strip():
+        return None
+    paths, total, truncated = _collect(result.output)
+    if not paths:
+        return None
+    return ProjectIndex(
+        paths=paths, total=total, truncated=truncated or result.truncated, source="git"
+    )
+
+
+def _find_command() -> str:
+    prunes = " -o ".join(f"-name {name!r}" for name in IGNORED)
+    # -print rather than -print0: the output is read as text either way, and a
+    # filename containing a newline splits into two entries that resolve to
+    # nothing rather than into anything dangerous.
+    return f"find . \\( {prunes} \\) -prune -o -print 2>/dev/null"
+
+
+async def _from_find(executor: Executor, project_dir: str) -> ProjectIndex | None:
+    result = await executor.run(
+        ExecRequest(
+            command=_find_command(),
+            cwd=project_dir,
+            timeout=BUILD_TIMEOUT,
+            max_bytes=MAX_OUTPUT,
+        )
+    )
+    if not result.output.strip():
+        return None
+    paths, total, truncated = _collect(result.output)
+    if not paths:
+        return None
+    return ProjectIndex(
+        paths=paths, total=total, truncated=truncated or result.truncated, source="find"
+    )
+
+
+async def _from_sftp(executor: Executor, project_dir: str) -> ProjectIndex:
+    """The one that always works, and the one that is slow.
+
+    Bounded twice over -- by depth and by how many directories it will open --
+    because this is a network round trip per directory and an unbounded walk of
+    a home directory would take minutes to produce something unusable.
+    """
+    found: list[str] = []
+    opened = 0
+    queue: list[tuple[str, int]] = [("", 0)]
+
+    while queue and opened < SFTP_MAX_DIRS and len(found) < MAX_ENTRIES:
+        where, depth = queue.pop(0)
+        opened += 1
+        try:
+            entries = await executor.scan_dir(where or project_dir)
+        except ExecError:
+            continue
+        for entry in entries:
+            if entry.name in IGNORED:
+                continue
+            path = f"{where}/{entry.name}" if where else entry.name
+            found.append(path + "/" if entry.is_dir else path)
+            if entry.is_dir and depth + 1 < SFTP_MAX_DEPTH:
+                queue.append((path, depth + 1))
+
+    paths, total, truncated = _collect("\n".join(found))
+    return ProjectIndex(
+        paths=paths,
+        total=total,
+        truncated=truncated or bool(queue),
+        source="sftp",
+    )
+
+
+async def build(executor: Executor, project_dir: str) -> ProjectIndex:
+    """Walk the directory, by whichever means works first."""
+    started = time.monotonic()
+    try:
+        for attempt in (_from_git, _from_find):
+            found = await attempt(executor, project_dir)
+            if found is not None:
+                break
+        else:
+            found = None
+        if found is None:
+            found = await _from_sftp(executor, project_dir)
+    except ExecError as exc:
+        log.info("could not index %s: %s", project_dir, exc.message)
+        return ProjectIndex(built_at=time.monotonic())
+
+    log.debug(
+        "indexed %s: %d paths by %s in %dms",
+        project_dir,
+        len(found.paths),
+        found.source,
+        int((time.monotonic() - started) * 1000),
+    )
+    return ProjectIndex(
+        paths=found.paths,
+        total=found.total,
+        truncated=found.truncated,
+        source=found.source,
+        built_at=time.monotonic(),
+    )
+
+
+# --- The cache ---------------------------------------------------------------
+# Keyed on the connection and the directory, not the chat: two chats on the same
+# box in the same tree are looking at the same files, and indexing it twice
+# would double the cost to prove it.
+_CACHE: dict[tuple[str, str], ProjectIndex] = {}
+_BUILDING: dict[tuple[str, str], asyncio.Task] = {}
+
+
+def cached(profile_id: str, project_dir: str) -> ProjectIndex | None:
+    """What is already known, or None. Never does any work.
+
+    `harness.context_variables` is synchronous and sits on the request path, so
+    it may only ever call this -- an SFTP round trip from there would block a
+    request while somebody's box thought about it.
+    """
+    found = _CACHE.get((profile_id, project_dir))
+    if found is None:
+        return None
+    if time.monotonic() - found.built_at > TTL:
+        _CACHE.pop((profile_id, project_dir), None)
+        return None
+    return found
+
+
+async def ensure(
+    executor: Executor, profile_id: str, project_dir: str, *, refresh: bool = False
+) -> ProjectIndex:
+    """The index, building it if there is not a fresh one already.
+
+    Concurrent callers share one build. A reply and the `@` picker asking at
+    the same moment is the ordinary case, not a rare one, and two walks of the
+    same tree would be two of everything for one answer.
+    """
+    key = (profile_id, project_dir)
+    if refresh:
+        _CACHE.pop(key, None)
+    elif (found := cached(profile_id, project_dir)) is not None:
+        return found
+
+    if (running := _BUILDING.get(key)) is not None:
+        return await asyncio.shield(running)
+
+    task = asyncio.create_task(build(executor, project_dir))
+    _BUILDING[key] = task
+    try:
+        found = await task
+    finally:
+        _BUILDING.pop(key, None)
+
+    _CACHE[key] = found
+    while len(_CACHE) > MAX_CACHED:
+        _CACHE.pop(next(iter(_CACHE)))
+    return found
+
+
+# --- Rendering ---------------------------------------------------------------
+# A tree that lists a thousand files is worse than no tree: it costs the window
+# on every request forever and buries the four names that mattered. So the
+# rendering has a character budget and elides what will not fit, saying how much
+# it elided -- a directory shown as `src/vendor/ (412 files)` is a model being
+# told where to look, which is the useful half of listing it.
+INDENT = "  "
+# Below this a directory is never collapsed. Elision costs a line either way, so
+# collapsing three files into "(3 files)" saves nothing and loses everything.
+ALWAYS_SHOW = 4
+
+
+def _tree(paths: tuple[str, ...]) -> dict:
+    root: dict = {}
+    for path in paths:
+        node = root
+        parts = [part for part in path.rstrip("/").split("/") if part]
+        for part in parts[:-1]:
+            node = node.setdefault(part, {})
+            if not isinstance(node, dict):  # a file and a directory share a name
+                break
+        else:
+            if parts:
+                leaf = parts[-1]
+                if path.endswith("/"):
+                    node.setdefault(leaf, {})
+                else:
+                    node.setdefault(leaf, None)
+    return root
+
+
+def _files_under(node: dict) -> int:
+    total = 0
+    for child in node.values():
+        total += _files_under(child) if isinstance(child, dict) else 1
+    return total
+
+
+def _candidates(node: dict, prefix: str, depth: int, out: list) -> None:
+    """Every directory, with what collapsing it would save."""
+    for name, child in node.items():
+        if not isinstance(child, dict):
+            continue
+        path = f"{prefix}{name}/"
+        count = _files_under(child)
+        full = _cost(child, depth + 1)
+        collapsed = len(f" ({count} files)")
+        if count > ALWAYS_SHOW and full > collapsed:
+            out.append((depth, count, path, full - collapsed))
+        _candidates(child, path, depth + 1, out)
+
+
+def _cost(node: dict, depth: int) -> int:
+    """Roughly how many characters rendering this subtree in full would take."""
+    total = 0
+    for name, child in node.items():
+        total += len(INDENT) * (depth + 1) + len(name) + 2
+        if isinstance(child, dict):
+            total += _cost(child, depth + 1)
+    return total
+
+
+def _plan(root: dict, budget: int) -> set[str]:
+    """Which directories to show as a count, so the rest fits.
+
+    Deepest and largest first. Collapsing by saving alone would take `src/`
+    before `src/web/static/vendor/` -- it is bigger, because it *contains* it --
+    and lose every name worth having to save one directory of hash-pinned
+    third-party files. Depth is the proxy for "further from what somebody was
+    looking for", and it is a good one.
+    """
+    if _cost(root, 0) <= budget:
+        return set()
+
+    candidates: list[tuple[int, int, str, int]] = []
+    _candidates(root, "", 0, candidates)
+    candidates.sort(key=lambda item: (-item[0], -item[1]))
+
+    chosen: dict[str, int] = {}
+    saved = 0
+    total = _cost(root, 0)
+    for _depth, _count, path, saving in candidates:
+        if total - saved <= budget:
+            break
+        # A directory inside one already collapsed is not rendered at all, so
+        # collapsing it saves nothing.
+        if any(path.startswith(done) for done in chosen):
+            continue
+        # And a directory *containing* one already collapsed subsumes it. Its
+        # own saving is measured against the full subtree, so the descendant's
+        # has to come back off or the two are counted twice -- which stopped
+        # the loop early believing it had made room it had not.
+        for inside in [done for done in chosen if done.startswith(path)]:
+            saved -= chosen.pop(inside)
+        chosen[path] = saving
+        saved += saving
+    return set(chosen)
+
+
+def _lines(
+    node: dict, prefix: str, depth: int, collapsed: set[str], budget: list[int]
+) -> list[str]:
+    out: list[str] = []
+    # Files before directories at each level: the shallow names are the ones
+    # somebody would recognise, and if the budget runs out mid-tree they are
+    # the ones worth having spent it on.
+    files = sorted(name for name, child in node.items() if not isinstance(child, dict))
+    folders = sorted(name for name, child in node.items() if isinstance(child, dict))
+
+    for position, name in enumerate(files):
+        line = f"{INDENT * depth}{name}"
+        if budget[0] < len(line) + 1:
+            out.append(f"{INDENT * depth}… {len(files) - position} more files")
+            budget[0] = 0
+            return out
+        budget[0] -= len(line) + 1
+        out.append(line)
+
+    for name in folders:
+        child = node[name]
+        path = f"{prefix}{name}/"
+        header = f"{INDENT * depth}{name}/"
+        if path in collapsed:
+            line = f"{header} ({_files_under(child)} files)"
+            budget[0] -= len(line) + 1
+            out.append(line)
+            continue
+        if budget[0] < len(header) + 1:
+            return out
+        budget[0] -= len(header) + 1
+        out.append(header)
+        out.extend(_lines(child, path, depth + 1, collapsed, budget))
+    return out
+
+
+def render(index: ProjectIndex, budget: int) -> str:
+    """The listing as the model sees it, inside `budget` characters.
+
+    Returns "" when there is nothing to say, so the fragment carrying it can
+    vanish entirely rather than appear as an empty heading -- which is what
+    `Fragment.requires` is for.
+    """
+    if not index.ok or budget <= 0:
+        return ""
+
+    root = _tree(index.paths)
+    collapsed = _plan(root, budget)
+    # The plan has already made it fit, so this is a backstop rather than the
+    # mechanism -- with enough slack that an estimate a little off does not
+    # truncate a listing that was fine. What it is really for is the one shape
+    # collapsing cannot help with: five thousand files directly in the root,
+    # where there is no directory to fold them into.
+    remaining = [int(budget * 1.5) + 200]
+    lines = _lines(root, "", 0, collapsed, remaining)
+    if not lines:
+        return ""
+
+    note = ""
+    if index.truncated:
+        note = (
+            f"\n\nThere are more than {len(index.paths)} entries here; this is the "
+            "first of them, so treat it as a sample rather than the whole tree."
+        )
+    elif collapsed:
+        note = (
+            "\n\nDirectories shown with a count were left unopened to save room. "
+            "Use `file_list` to look inside one."
+        )
+    return "\n".join(lines) + note
+
+
+def forget(profile_id: str) -> int:
+    """Drop everything indexed through one connection.
+
+    Called when a profile is deleted, disabled or has its host key forgotten --
+    the same moments that close its terminals. Keeping a listing of a machine
+    somebody has just revoked would be a small leak of exactly the kind the
+    rest of this module is careful about.
+    """
+    doomed = [key for key in _CACHE if key[0] == profile_id]
+    for key in doomed:
+        _CACHE.pop(key, None)
+    return len(doomed)
+
+
+def clear() -> None:
+    _CACHE.clear()
+
+
+__all__ = [
+    "MAX_ENTRIES",
+    "ProjectIndex",
+    "build",
+    "cached",
+    "clear",
+    "ensure",
+    "forget",
+    "render",
+]

src/lembas/services/agent/ssh.py

@@ -385,6 +385,7 @@ async def check(spec: dict[str, Any], project_dir: str = "") -> dict[str, Any]:
 
 __all__ = [
     "INSTALL_HINT",
+    "MAX_ENTRIES",
     "MAX_READ_BYTES",
     "SshExecutor",
     "available",

src/lembas/services/chat.py

@@ -89,14 +89,30 @@ def document_context(message: Message) -> str:
         if not attachment.extracted_text.strip():
             continue
         note = " (truncated)" if attachment.truncated else ""
+        # Where it came from, when there is a where. A model handed `main.py`
+        # cannot tell which of four it is looking at, and cannot name the file
+        # back when asked to change something -- so a file read off a machine
+        # says which machine and which path. Quotes are stripped rather than
+        # escaped: these are attribute values in a tag the model reads, and a
+        # path containing one would otherwise close it early.
+        where = ""
+        if attachment.source_path:
+            where += f' path="{_attr(attachment.source_path)}"'
+        if attachment.source_label:
+            where += f' from="{_attr(attachment.source_label)}"'
         blocks.append(
-            f'<document name="{attachment.filename}"{note}>\n'
+            f'<document name="{_attr(attachment.filename)}"{where}{note}>\n'
             f"{attachment.extracted_text.strip()}\n"
             f"</document>"
         )
     return "\n\n".join(blocks)
 
 
+def _attr(value: str) -> str:
+    """A value safe to sit inside the double quotes of a tag we are writing."""
+    return value.replace('"', "").replace("<", "").replace(">", "").replace("\n", " ")
+
+
 def message_payload(message: Message, *, vision: bool) -> dict[str, Any]:
     """One history entry in the shape the endpoint expects.
 

src/lembas/services/files.py

@@ -334,12 +334,20 @@ def store_text(
     text: str,
     truncated: bool = False,
     source_note: str = "",
+    source_path: str = "",
+    source_label: str = "",
 ) -> Attachment:
     """Attach text that did not arrive as a file -- a fetched web page.
 
     Written to disk like any other attachment so it can be downloaded and so
     there is one cleanup path, rather than a second kind of attachment that
     exists only in the database.
+
+    `source_note` leads the *text*; `source_path` and `source_label` are
+    columns. The two are not the same thing and both are wanted: the note is
+    prose a model reads inside the document, and the columns become attributes
+    on the tag around it, which is what a reader sees on the chip and what
+    survives if the text is later truncated away from its own first line.
     """
     body = text[:MAX_EXTRACTED_CHARS]
     payload = body.encode("utf-8")
@@ -359,6 +367,8 @@ def store_text(
         # can see where an attachment called "Some Page.txt" came from.
         extracted_text=f"Source: {source_note}\n\n{body}" if source_note else body,
         truncated=truncated,
+        source_path=source_path[:1000],
+        source_label=source_label[:200],
     )
     db.add(attachment)
     db.commit()

src/lembas/services/generation.py

@@ -26,7 +26,7 @@ from datetime import UTC, datetime, timedelta
 
 from sqlalchemy import select
 
-from lembas.db.models import ROLE_ASSISTANT, ROLE_USER, Chat, Message, User
+from lembas.db.models import KIND_AGENT, ROLE_ASSISTANT, ROLE_USER, Chat, Message, User
 from lembas.db.session import session_scope
 from lembas.services import chat as chat_service
 from lembas.services import compaction as compaction_service
@@ -294,6 +294,12 @@ async def _run(generation: Generation) -> None:
         # summarisation in front of it would break exactly that.
         await _maybe_compact(generation)
 
+        # Before the session opens, for the same reason compaction is: the
+        # listing is an SSH round trip, and holding a database session across
+        # one to save opening a second is the wrong trade. `build_request`
+        # below reads whatever this left in the cache and never fetches.
+        await _warm_index(generation)
+
         with session_scope() as db:
             chat = db.get(Chat, generation.chat_id)
             message = db.get(Message, generation.message_id)
@@ -524,6 +530,55 @@ async def _run(generation: Generation) -> None:
         generation.touch()
 
 
+# How long a reply will wait for a directory listing before starting without
+# one. Short on purpose: the listing is a convenience and the reply is the
+# thing somebody is waiting for. A walk that outruns this keeps going in the
+# background and the next turn has it.
+INDEX_WAIT = 6.0
+
+
+async def _warm_index(generation: Generation) -> None:
+    """Build this chat's project listing, or leave whatever is cached.
+
+    Never raises and never blocks for long. `harness` reads the cache
+    synchronously while assembling the system message, so something has to fill
+    it, and this is the one place in a reply's life that is both asynchronous
+    and already doing network work.
+
+    The first reply in a brand-new chat on a big tree may start before the walk
+    finishes. That is deliberate: the fragment carrying the listing vanishes
+    when it is empty rather than appearing as a heading with nothing under it,
+    and by the following turn it is there.
+    """
+    from lembas.services import settings_store
+    from lembas.services.agent import index as index_service
+    from lembas.services.agent import session as agent_session
+
+    try:
+        with session_scope() as db:
+            if not settings_store.agents(db).get("index_enabled"):
+                return
+            chat = db.get(Chat, generation.chat_id)
+            if chat is None or chat.kind != KIND_AGENT:
+                return
+            owner = db.get(User, chat.user_id)
+            context = agent_session.resolve(db, chat, owner)
+            profile_id = chat.ssh_profile_id or ""
+        if context is None or not profile_id:
+            return
+        if index_service.cached(profile_id, context.project_dir) is not None:
+            return
+
+        await asyncio.wait_for(
+            index_service.ensure(context.executor(), profile_id, context.project_dir),
+            timeout=INDEX_WAIT,
+        )
+    except TimeoutError:
+        log.debug("index for chat %s outran its wait; carrying on", generation.chat_id)
+    except Exception as exc:  # noqa: BLE001 - a missing listing is not a failed reply
+        log.info("could not warm the index for chat %s: %s", generation.chat_id, exc)
+
+
 async def _maybe_compact(generation: Generation) -> None:
     """Summarise the earlier turns if the window is about to be full.
 

src/lembas/services/harness.py

@@ -135,6 +135,7 @@ def context_variables(
         "agent_dir": "",
         "agent_mode": "",
         "agent_rewound": "",
+        "project_files": "",
     }
 
     if chat is not None:
@@ -161,6 +162,8 @@ def context_variables(
 
 def _agent_values(db: DBSession, chat, user) -> dict[str, str]:
     """What an agent chat's harness needs to say about where it is."""
+    from lembas.services import settings_store
+    from lembas.services.agent import index as index_service
     from lembas.services.agent import policy
     from lembas.services.agent import session as agent_session
 
@@ -178,9 +181,36 @@ def _agent_values(db: DBSession, chat, user) -> dict[str, str]:
         "agent_mode": policy.MODE_GUIDANCE.get(context.mode, ""),
         "agent_rewound": rewound,
         "max_rounds": str(context.limits.steps),
+        "project_files": _project_files(db, chat, context, settings_store, index_service),
     }
 
 
+def _project_files(db: DBSession, chat, context, settings_store, index_service) -> str:
+    """The directory listing, *read from cache and never fetched*.
+
+    This whole module runs synchronously on the request path, so an SFTP round
+    trip here would hold a request open while somebody's box thought about it.
+    The build happens in the generation setup, which is async and already doing
+    network work; here we take whatever it left behind.
+
+    A chat whose very first reply outruns its first index simply has no listing
+    that turn -- the fragment's `requires` makes it vanish rather than appear as
+    an empty heading, and the next turn has it.
+    """
+    agents = settings_store.agents(db)
+    if not agents.get("index_enabled"):
+        return ""
+    budget = int(agents.get("index_chars") or 0)
+    if budget <= 0:
+        return ""
+
+    profile_id = getattr(chat, "ssh_profile_id", "") or ""
+    found = index_service.cached(profile_id, context.project_dir)
+    if found is None:
+        return ""
+    return index_service.render(found, budget)
+
+
 def limit_for(db: DBSession) -> int:
     """The ceiling on the assembled block."""
     stored = settings_store.get(db, "max_harness_chars", key=settings_store.PROMPTS)

src/lembas/services/prompts.py

@@ -155,6 +155,14 @@ VARIABLES: tuple[Variable, ...] = (
         "When an agent chat was last edited or regenerated. Empty otherwise, "
         "which is what keeps the note about it out of every other reply.",
     ),
+    Variable(
+        "project_files",
+        "Project files",
+        "What is in the project directory, as an indented tree with large "
+        "directories shown as a count. Empty until the first listing has been "
+        "built, when the feature is off, or when the directory could not be "
+        "read -- and the section it lives in disappears with it.",
+    ),
     Variable(
         "memories",
         "Memories",
@@ -805,6 +813,33 @@ BUILTIN: tuple[Fragment, ...] = (
             "not look for another way round it."
         ),
     ),
+    Fragment(
+        key="tool.project_files",
+        label="What is in the project directory",
+        group=GROUP_CONTEXT,
+        order=325,
+        families=("agent",),
+        requires=("project_files",),
+        variables=("project_files", "agent_dir"),
+        hint="A listing of the project directory, so the first two rounds of a "
+        "reply are not spent finding out what is in it. Large directories are "
+        "shown as a count rather than expanded, and the budget for the whole "
+        "thing is set under Admin -> Agents. Clearing this box switches the "
+        "listing off in the prompt while leaving it available to the file "
+        "picker.",
+        default=(
+            "### Files in {{agent_dir}}\n"
+            "\n"
+            "```\n"
+            "{{project_files}}\n"
+            "```\n"
+            "\n"
+            "A snapshot from when this reply started, and not necessarily "
+            "complete. It is a map, not an authority: check a path before "
+            "relying on it, and do not conclude a file is absent because it is "
+            "not listed here."
+        ),
+    ),
     Fragment(
         key="tool.agent_rewound",
         label="After a rewind",

src/lembas/services/settings_store.py

@@ -88,6 +88,14 @@ def _agents_defaults() -> dict[str, Any]:
         # SSH connection held open, so this is a real resource, not a scruple.
         "terminal_max_sessions": 20,
         "terminal_max_per_user": 3,
+        # A listing of the project directory, put in front of the model so the
+        # first rounds of a reply are not spent discovering what is there. It
+        # costs its budget on *every* request in an agent chat, forever, which
+        # is why it is a switch and a number rather than a constant.
+        "index_enabled": True,
+        # Characters. Clamped on read: a huge value here would quietly spend
+        # somebody's whole context window on filenames.
+        "index_chars": 2000,
     }
 
 
@@ -244,4 +252,8 @@ def agents(db: DBSession) -> dict[str, Any]:
         max(int(values.get("terminal_max_sessions") or 0), 1), 500
     )
     values["terminal_max_per_user"] = min(max(int(values.get("terminal_max_per_user") or 0), 1), 50)
+    # Zero is meaningful here and is not clamped away: it means "index the
+    # directory for the file picker, but put none of it in the prompt", which
+    # is a reasonable thing to want and has no other way of being said.
+    values["index_chars"] = min(max(int(values.get("index_chars") or 0), 0), 20_000)
     return values