A directory the model knows about, and @ to name a file in it

An agent chat used to open with the model knowing the name of a machine and
nothing about what was on it, so the first two rounds of every reply went on
finding out. It now gets a listing: one read-only command, `git ls-files` where
that works and `find` otherwise, falling back to an SFTP walk that always does.
git first because a repository already carries somebody's considered list of
what is not part of the project, and reproducing it by hand is how an index
ends up mostly build output.

The listing is budgeted rather than dumped. A tree of a thousand files is worse
than no tree -- it costs the window on every request forever and buries the four
names that mattered -- so directories that will not fit are shown as a count and
the model is told to open one itself. Collapsing picks the deepest and largest
first: by saving alone it would take `src/` before `src/web/static/vendor/`,
because it contains it, and lose every name worth having.

Read from a cache and never fetched. `harness.context_variables` is synchronous
and sits on the request path; the walk happens in the generation setup, which is
async and already doing network work, with a short wait. A chat whose first
reply outruns its first walk simply has no listing that turn and the fragment
disappears rather than appearing as an empty heading.

Then `@`, over the same index and over the library, and `/` for commands with an
Alt-based keyboard for the same jobs. A mentioned file arrives as contents, not
a reference -- a small model asked to call file_read often does not bother -- and
it arrives with its absolute path and the machine it came from, because a model
handed `main.py` cannot tell which of four it is and cannot name it back when
asked to change something.

The rule that matters for `/`: a message that merely starts with a slash still
sends. `//` escapes and an unrecognised command is posted as written. Swallowing
somebody's message is a much worse failure than an unknown command.

Two exceptions to Manual mode now, not one. Browsing and indexing are a person
acting, not a model, so neither passes through policy.py -- the same argument
the terminal panel rests on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/lembas/services/generation.py

@@ -26,7 +26,7 @@ from datetime import UTC, datetime, timedelta
 
 from sqlalchemy import select
 
-from lembas.db.models import ROLE_ASSISTANT, ROLE_USER, Chat, Message, User
+from lembas.db.models import KIND_AGENT, ROLE_ASSISTANT, ROLE_USER, Chat, Message, User
 from lembas.db.session import session_scope
 from lembas.services import chat as chat_service
 from lembas.services import compaction as compaction_service
@@ -294,6 +294,12 @@ async def _run(generation: Generation) -> None:
         # summarisation in front of it would break exactly that.
         await _maybe_compact(generation)
 
+        # Before the session opens, for the same reason compaction is: the
+        # listing is an SSH round trip, and holding a database session across
+        # one to save opening a second is the wrong trade. `build_request`
+        # below reads whatever this left in the cache and never fetches.
+        await _warm_index(generation)
+
         with session_scope() as db:
             chat = db.get(Chat, generation.chat_id)
             message = db.get(Message, generation.message_id)
@@ -524,6 +530,55 @@ async def _run(generation: Generation) -> None:
         generation.touch()
 
 
+# How long a reply will wait for a directory listing before starting without
+# one. Short on purpose: the listing is a convenience and the reply is the
+# thing somebody is waiting for. A walk that outruns this keeps going in the
+# background and the next turn has it.
+INDEX_WAIT = 6.0
+
+
+async def _warm_index(generation: Generation) -> None:
+    """Build this chat's project listing, or leave whatever is cached.
+
+    Never raises and never blocks for long. `harness` reads the cache
+    synchronously while assembling the system message, so something has to fill
+    it, and this is the one place in a reply's life that is both asynchronous
+    and already doing network work.
+
+    The first reply in a brand-new chat on a big tree may start before the walk
+    finishes. That is deliberate: the fragment carrying the listing vanishes
+    when it is empty rather than appearing as a heading with nothing under it,
+    and by the following turn it is there.
+    """
+    from lembas.services import settings_store
+    from lembas.services.agent import index as index_service
+    from lembas.services.agent import session as agent_session
+
+    try:
+        with session_scope() as db:
+            if not settings_store.agents(db).get("index_enabled"):
+                return
+            chat = db.get(Chat, generation.chat_id)
+            if chat is None or chat.kind != KIND_AGENT:
+                return
+            owner = db.get(User, chat.user_id)
+            context = agent_session.resolve(db, chat, owner)
+            profile_id = chat.ssh_profile_id or ""
+        if context is None or not profile_id:
+            return
+        if index_service.cached(profile_id, context.project_dir) is not None:
+            return
+
+        await asyncio.wait_for(
+            index_service.ensure(context.executor(), profile_id, context.project_dir),
+            timeout=INDEX_WAIT,
+        )
+    except TimeoutError:
+        log.debug("index for chat %s outran its wait; carrying on", generation.chat_id)
+    except Exception as exc:  # noqa: BLE001 - a missing listing is not a failed reply
+        log.info("could not warm the index for chat %s: %s", generation.chat_id, exc)
+
+
 async def _maybe_compact(generation: Generation) -> None:
     """Summarise the earlier turns if the window is about to be full.