Finding a thing that does not use your words
Three pieces, and the first one is that they are all optional. Extraction stops being constants. Upload size, image edge, JPEG quality, PDF pages, extracted characters, orphan age and the text-extension list are settings now, read through a process-level snapshot rather than a session -- `prepare` and everything under it are called from routes, tool runners and the startup sweep, and several of those have no session in hand. Two things deliberately stayed constants: the decompression-bomb guard, which is a guard and not a preference, and ORPHAN_AGE, which would have been evaluated at import if it stayed in the signature and pinned the shipped 24 hours whatever anybody set. An embedding model is picked from the models an administrator flagged for it, and one that has since lost its flag is *named* rather than dropped from the picker: a setting that vanishes is one nobody can tell from a setting never made. Nothing here is required. Choosing none means no chunk rows, no requests, and retrieval.search returning exactly what fts.search_ids returns in exactly that order -- asserted, because it is what makes this safe to land on an instance that never asked for it. The two rankings are fused by reciprocal rank fusion: ranks and not scores, because bm25 is a corpus-dependent negative and cosine is 0..1, and normalising them onto one scale means picking a constant nobody can tune without a labelled set they do not have. RRF's one constant is famously insensitive and degrades to whichever list is non-empty -- which is what turns "no embedding model" into a branch that does not exist. A record scores as its best chunk rather than its average, or a long document about something else outranks a short one that says the thing. Width and model are stored beside every vector and a mismatch is skipped, because vectors from two spaces score against each other perfectly happily and mean nothing -- a search that works and is wrong is the worst failure this can have, and a model change now leaves stale rows ignored rather than trusted. Indexing is fired and forgotten, and how a change is noticed is a session event rather than a call in each of the ten library writers. That is a departure from this codebase's taste for explicit seams, for the reason tool_label is a Jinja global: a step every writer has to remember is one that gets forgotten, and here forgetting is silent -- the record saves, keyword search still finds it, and only its recall goes stale. Chunks are embedded before anything is deleted, so a failure leaves the old index rather than half a new one. Also: `embeddings` joins the model capabilities, and the three tool flags that had shipped with no checkbox -- canvas, scheduling and helpers -- have one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jaroslav Beneš
+10
-7
@@ -72,19 +72,22 @@ def fresh_database(tmp_path: Path) -> Iterator[None]:
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_branding() -> Iterator[None]:
|
||||
"""Drop the branding snapshot between tests.
|
||||
def fresh_snapshots() -> Iterator[None]:
|
||||
"""Drop the process-level snapshots between tests.
|
||||
|
||||
It is a process-level cache read by a Jinja global, so without this the
|
||||
first test to render a page pins one instance's name, logo and themes for
|
||||
every test after it -- against a database that has since been thrown away.
|
||||
The same shape as the registries below, and the reason each of them exists.
|
||||
Two of them now, and both are read once per process against a database this
|
||||
fixture throws away between tests -- so without this, the first test to
|
||||
render a page pins one instance's name and themes for every test after it,
|
||||
and the first to save an upload limit pins that too. The same shape as the
|
||||
registries below, and the reason each of them exists.
|
||||
"""
|
||||
from lembas.services import branding
|
||||
from lembas.services import branding, files
|
||||
|
||||
branding.forget()
|
||||
files.forget()
|
||||
yield
|
||||
branding.forget()
|
||||
files.forget()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
"""Turning text into vectors, and the pieces text is cut into first.
|
||||
|
||||
The failure this file is mostly about is the quiet one: a response paired with
|
||||
the wrong input produces a search that works and is wrong, which nothing
|
||||
downstream can notice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from lembas.services.library import chunks
|
||||
from lembas.services.llm import embeddings
|
||||
from lembas.services.llm.openai_client import Endpoint, LLMError
|
||||
|
||||
ENDPOINT = Endpoint(base_url="http://127.0.0.1:9", api_key="", extra_headers={}, name="test")
|
||||
|
||||
|
||||
def _answer(vectors, *, shuffle=False):
|
||||
data = [{"embedding": vector, "index": i} for i, vector in enumerate(vectors)]
|
||||
if shuffle:
|
||||
data = list(reversed(data))
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json={"data": data})
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
# --- The client -----------------------------------------------------------------
|
||||
async def test_vectors_come_back_normalised(mock_http):
|
||||
"""Cosine between unit vectors is their dot product, so normalising once at
|
||||
write time turns every later comparison into a multiply-and-add."""
|
||||
mock_http(_answer([[3.0, 4.0]]))
|
||||
|
||||
(vector,) = await embeddings.embed(ENDPOINT, "m", ["hello"])
|
||||
|
||||
assert math.isclose(vector[0], 0.6)
|
||||
assert math.isclose(vector[1], 0.8)
|
||||
|
||||
|
||||
async def test_the_declared_index_decides_the_order(mock_http):
|
||||
"""Nothing in the specification promises the order of `data`. A provider
|
||||
that sorts differently would pair every chunk with somebody else's vector —
|
||||
a search that works and is wrong, which is the worst failure here."""
|
||||
mock_http(_answer([[1.0, 0.0], [0.0, 1.0]], shuffle=True))
|
||||
|
||||
first, second = await embeddings.embed(ENDPOINT, "m", ["a", "b"])
|
||||
|
||||
assert first == [1.0, 0.0]
|
||||
assert second == [0.0, 1.0]
|
||||
|
||||
|
||||
async def test_a_short_answer_is_refused_rather_than_guessed(mock_http):
|
||||
"""Silently accepting it would pair chunk three's text with chunk four's
|
||||
vector from there on, for the life of the index."""
|
||||
mock_http(_answer([[1.0, 0.0]]))
|
||||
|
||||
with pytest.raises(LLMError) as caught:
|
||||
await embeddings.embed(ENDPOINT, "m", ["a", "b"])
|
||||
|
||||
assert "2" in str(caught.value)
|
||||
|
||||
|
||||
async def test_vectors_of_different_widths_are_refused(mock_http):
|
||||
mock_http(_answer([[1.0, 0.0], [0.0, 1.0, 0.0]]))
|
||||
|
||||
with pytest.raises(LLMError):
|
||||
await embeddings.embed(ENDPOINT, "m", ["a", "b"])
|
||||
|
||||
|
||||
async def test_an_endpoint_that_does_not_do_embeddings_says_so(mock_http):
|
||||
def handler(request):
|
||||
return httpx.Response(404, json={"error": {"message": "no such endpoint"}})
|
||||
|
||||
mock_http(handler)
|
||||
|
||||
with pytest.raises(LLMError):
|
||||
await embeddings.embed(ENDPOINT, "m", ["a"])
|
||||
|
||||
|
||||
async def test_nothing_asked_is_nothing_sent(mock_http):
|
||||
"""A request with no inputs is a round trip for nothing, and some endpoints
|
||||
refuse it outright."""
|
||||
sent = []
|
||||
|
||||
def handler(request):
|
||||
sent.append(request)
|
||||
return httpx.Response(200, json={"data": []})
|
||||
|
||||
mock_http(handler)
|
||||
|
||||
assert await embeddings.embed(ENDPOINT, "m", []) == []
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_a_zero_vector_survives_normalising():
|
||||
"""What an endpoint returns for empty input, and the one arithmetic error
|
||||
this path can make."""
|
||||
assert embeddings.normalise([0.0, 0.0]) == [0.0, 0.0]
|
||||
|
||||
|
||||
# --- Splitting ------------------------------------------------------------------
|
||||
def test_short_text_is_one_piece():
|
||||
assert chunks.split("a short note", size=1200) == ["a short note"]
|
||||
|
||||
|
||||
def test_splitting_prefers_a_paragraph_boundary():
|
||||
body = "\n\n".join(["A" * 400, "B" * 400, "C" * 400])
|
||||
pieces = chunks.split(body, size=900, overlap=0)
|
||||
|
||||
assert len(pieces) >= 2
|
||||
# No piece ends mid-run, which is what a boundary being honoured looks like.
|
||||
assert all(piece.strip() == piece for piece in pieces)
|
||||
assert pieces[0].startswith("A")
|
||||
|
||||
|
||||
def test_the_overlap_carries_the_tail_forward():
|
||||
body = "".join(f"sentence {n}. " for n in range(200))
|
||||
pieces = chunks.split(body, size=400, overlap=100)
|
||||
|
||||
assert len(pieces) > 1
|
||||
# Consecutive pieces share text, which is what stops a sentence across a
|
||||
# boundary being absent from both embeddings.
|
||||
assert any(pieces[0][-40:] in pieces[1] for _ in (0,)) or pieces[1][:40] in body
|
||||
|
||||
|
||||
def test_an_overlap_as_large_as_the_piece_does_not_hang():
|
||||
"""An overlap at or past the size means every piece starts where the last
|
||||
one did. Clamped here as well as in the settings accessor, because the
|
||||
failure is a hang rather than a bad index."""
|
||||
pieces = chunks.split("x" * 5000, size=400, overlap=4000)
|
||||
|
||||
assert len(pieces) < 40
|
||||
|
||||
|
||||
def test_a_scrap_is_not_worth_a_row():
|
||||
"""The embedding of six words is mostly noise, and a search whose best hit
|
||||
is "and the following:" is worse than one that returns nothing."""
|
||||
assert chunks.split("hi") == ["hi"]
|
||||
pieces = chunks.split("A" * 400 + "\n\n" + "B" * 5, size=380, overlap=0)
|
||||
assert all(len(piece) >= chunks.MIN_CHUNK_CHARS for piece in pieces)
|
||||
|
||||
|
||||
def test_nothing_in_is_nothing_out():
|
||||
assert chunks.split("") == []
|
||||
assert chunks.split(" \n\n ") == []
|
||||
|
||||
|
||||
# --- Packing --------------------------------------------------------------------
|
||||
def test_a_vector_survives_a_round_trip():
|
||||
vector = [0.5, -0.25, 0.125]
|
||||
blob = chunks.pack(vector)
|
||||
|
||||
assert len(blob) == 12
|
||||
assert chunks.unpack(blob, 3) == pytest.approx(vector)
|
||||
|
||||
|
||||
def test_a_truncated_blob_unpacks_to_nothing():
|
||||
"""Inferring the width from the length would let a short BLOB unpack into a
|
||||
shorter vector and score against a query happily — a wrong answer rather
|
||||
than a missing one."""
|
||||
assert chunks.unpack(chunks.pack([1.0, 2.0, 3.0])[:8], 3) == []
|
||||
|
||||
|
||||
def test_scoring_across_widths_is_zero_rather_than_an_error():
|
||||
"""It means the two vectors came from different models, and the honest
|
||||
answer to "how similar are these?" across two spaces is nothing."""
|
||||
assert chunks.dot([1.0, 0.0], [1.0, 0.0, 0.0]) == 0.0
|
||||
|
||||
|
||||
def test_the_dot_product_is_the_cosine_for_unit_vectors():
|
||||
a = embeddings.normalise([1.0, 1.0])
|
||||
b = embeddings.normalise([1.0, 0.0])
|
||||
|
||||
assert chunks.dot(a, a) == pytest.approx(1.0)
|
||||
assert chunks.dot(a, b) == pytest.approx(math.sqrt(0.5))
|
||||
|
||||
|
||||
def test_the_hash_changes_with_the_text():
|
||||
assert chunks.digest("a") != chunks.digest("b")
|
||||
assert chunks.digest("a") == chunks.digest("a")
|
||||
+87
-3
@@ -188,12 +188,15 @@ def test_empty_files_are_rejected():
|
||||
|
||||
def test_oversized_files_are_rejected():
|
||||
with pytest.raises(files_service.FileError) as caught:
|
||||
files_service.prepare(b"x" * (files_service.MAX_UPLOAD_BYTES + 1), "huge.txt")
|
||||
files_service.prepare(b"x" * (files_service.limits().max_upload_bytes + 1), "huge.txt")
|
||||
assert "MB" in str(caught.value)
|
||||
|
||||
|
||||
def test_extracted_text_is_capped(monkeypatch):
|
||||
monkeypatch.setattr(files_service, "MAX_EXTRACTED_CHARS", 50)
|
||||
"""Through the setting rather than the constant. The constant is only the
|
||||
default now; what `prepare` reads is the snapshot, which is the thing that
|
||||
would have gone on returning 120,000 if the wiring were wrong."""
|
||||
monkeypatch.setattr(files_service, "_LIMITS", files_service.Limits(max_extracted_chars=50))
|
||||
prepared = files_service.prepare(b"x" * 500, "long.txt")
|
||||
assert len(prepared.extracted_text) == 50
|
||||
assert prepared.truncated is True
|
||||
@@ -446,7 +449,7 @@ def test_documents_reach_a_model_without_vision(client: TestClient, db, chat_wit
|
||||
|
||||
def test_truncation_is_declared_to_the_model(client: TestClient, db, chat_with_model, monkeypatch):
|
||||
"""A model asked about page 400 should be able to say it did not see it."""
|
||||
monkeypatch.setattr(files_service, "MAX_EXTRACTED_CHARS", 20)
|
||||
monkeypatch.setattr(files_service, "_LIMITS", files_service.Limits(max_extracted_chars=20))
|
||||
client.post("/api/files", files={"file": ("big.txt", b"y" * 200, "text/plain")})
|
||||
attachment = db.scalar(select(Attachment))
|
||||
client.post(
|
||||
@@ -600,3 +603,84 @@ def test_a_browser_serialising_the_form_actually_sends_the_attachment(
|
||||
content = turns(chat_service.build_request(db, chat))[0]["content"]
|
||||
assert isinstance(content, list), "the image never reached the model"
|
||||
assert any(p.get("type") == "image_url" for p in content)
|
||||
|
||||
|
||||
# --- The extraction settings ---------------------------------------------------
|
||||
# The constants above are defaults now, and what `prepare` actually reads is a
|
||||
# process-level snapshot. Every one of these failures would be silent: a limit
|
||||
# that looks configured and is not.
|
||||
def test_a_saved_limit_reaches_the_snapshot(db, client, registered):
|
||||
from lembas.services import settings_store
|
||||
|
||||
client.post(
|
||||
"/admin/extraction",
|
||||
data={
|
||||
"max_upload_mb": "5",
|
||||
"max_image_edge": "800",
|
||||
"jpeg_quality": "70",
|
||||
"max_pdf_pages": "10",
|
||||
"max_extracted_chars": "2000",
|
||||
"orphan_hours": "3",
|
||||
"extra_text_extensions": "env\n.conf",
|
||||
},
|
||||
follow_redirects=False,
|
||||
)
|
||||
|
||||
bounds = files_service.limits()
|
||||
assert bounds.max_upload_bytes == 5 * 1024 * 1024
|
||||
assert bounds.max_extracted_chars == 2000
|
||||
assert bounds.max_image_edge == 800
|
||||
assert settings_store.extraction(db)["extra_text_extensions"] == ["env", ".conf"]
|
||||
|
||||
|
||||
def test_an_extra_extension_gets_a_leading_dot(db, client, registered):
|
||||
"""Typed both ways by different people, and a mapping somebody has to get
|
||||
right twice is one they get wrong once."""
|
||||
client.post(
|
||||
"/admin/extraction", data={"extra_text_extensions": "env"}, follow_redirects=False
|
||||
)
|
||||
|
||||
assert files_service.limits().media_type_for(".env") == "text/plain"
|
||||
|
||||
|
||||
def test_an_unknown_extension_is_still_stored_as_text(db):
|
||||
"""Decodability is what decides. The list only picks a media type, which is
|
||||
why an unlisted extension has always worked and must go on working."""
|
||||
prepared = files_service.prepare(b"hello there", "notes.wat")
|
||||
assert prepared.kind == "text"
|
||||
assert prepared.extension == ".txt"
|
||||
|
||||
|
||||
def test_a_number_out_of_range_is_clamped(db, client, registered):
|
||||
client.post(
|
||||
"/admin/extraction",
|
||||
data={"max_upload_mb": "99999", "jpeg_quality": "1"},
|
||||
follow_redirects=False,
|
||||
)
|
||||
|
||||
bounds = files_service.limits()
|
||||
assert bounds.max_upload_bytes == 512 * 1024 * 1024
|
||||
assert bounds.jpeg_quality == 30
|
||||
|
||||
|
||||
def test_the_snapshot_is_dropped_when_the_page_saves(db, client, registered):
|
||||
"""Read once per process. A save that did not drop it would take effect at
|
||||
the next restart, which is the failure this codebase keeps cataloguing."""
|
||||
assert files_service.limits().max_upload_bytes == 20 * 1024 * 1024
|
||||
|
||||
client.post("/admin/extraction", data={"max_upload_mb": "1"}, follow_redirects=False)
|
||||
|
||||
assert files_service.limits().max_upload_bytes == 1024 * 1024
|
||||
|
||||
|
||||
def test_only_an_administrator_may_change_extraction(client, registered):
|
||||
client.post("/auth/logout", follow_redirects=False)
|
||||
client.post(
|
||||
"/auth/register",
|
||||
data={"name": "Sam", "email": "sam@shire.test", "password": "potatoes-po-ta-toes"},
|
||||
follow_redirects=False,
|
||||
)
|
||||
|
||||
assert client.post("/admin/extraction", data={"max_upload_mb": "1"}).status_code == 403
|
||||
assert client.post("/admin/extraction/search", data={}).status_code == 403
|
||||
assert client.post("/admin/extraction/rebuild", data={}).status_code == 403
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
"""Keyword search, semantic search, and the two fused.
|
||||
|
||||
The property this file exists to hold is the first one: **with no embedding
|
||||
model configured, everything here is byte-for-byte the search that has always
|
||||
been.** That is what makes this safe to add to an instance that never asked for
|
||||
it, and it is the one claim a comment cannot make credible.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from lembas.db.models import CHUNK_NOTE, Chunk, Connection, Model, User
|
||||
from lembas.services import settings_store
|
||||
from lembas.services.crypto import encrypt
|
||||
from lembas.services.library import chunks as chunk_service
|
||||
from lembas.services.library import indexing, retrieval
|
||||
from lembas.services.library import notes as notes_service
|
||||
from lembas.services.library.fts import SearchHit
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean():
|
||||
indexing.clear()
|
||||
yield
|
||||
indexing.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def user(db, registered) -> User:
|
||||
return db.scalars(select(User).order_by(User.created_at)).first()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def embedding_model(db):
|
||||
"""A connection and a model marked for embeddings, and the setting pointing
|
||||
at it. Nothing here makes a request; the tests that need one mock it."""
|
||||
connection = Connection(
|
||||
name="Embed", base_url="http://127.0.0.1:9", api_key_encrypted=encrypt("")
|
||||
)
|
||||
db.add(connection)
|
||||
db.commit()
|
||||
db.add(
|
||||
Model(
|
||||
connection_id=connection.id,
|
||||
model_id="embed-1",
|
||||
capabilities_json={"embeddings": True},
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
settings_store.update(db, {"embedding_model_id": "embed-1"}, key=settings_store.EXTRACTION)
|
||||
return "embed-1"
|
||||
|
||||
|
||||
def _vector_answer(vector):
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
count = len(request.read().decode().split('"input"')[1].split("[")[1].split(","))
|
||||
return httpx.Response(
|
||||
200, json={"data": [{"embedding": vector, "index": i} for i in range(count)]}
|
||||
)
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
def _store_chunk(db, note, vector, *, text="stored"):
|
||||
db.add(
|
||||
Chunk(
|
||||
owner_id=note.owner_id,
|
||||
resource_type=CHUNK_NOTE,
|
||||
resource_id=note.id,
|
||||
ordinal=0,
|
||||
text=text,
|
||||
vector=chunk_service.pack(vector),
|
||||
dims=len(vector),
|
||||
model_id="embed-1",
|
||||
source_hash=chunk_service.digest(text),
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
# --- Nothing configured ---------------------------------------------------------
|
||||
def test_with_no_model_nothing_is_indexed(db, user):
|
||||
"""No chunk rows, no requests, no cost. Asserted rather than assumed: it is
|
||||
the whole reason this is safe to add to an existing instance."""
|
||||
assert indexing.enabled(db) is False
|
||||
notes_service.create(db, owner=user, title="Moria", body="the west gate")
|
||||
|
||||
assert db.scalars(select(Chunk)).all() == []
|
||||
|
||||
|
||||
def test_with_no_model_search_is_exactly_the_keyword_search(db, user):
|
||||
from lembas.services.library import fts
|
||||
|
||||
notes_service.create(db, owner=user, title="Moria", body="the west gate opens")
|
||||
notes_service.create(db, owner=user, title="Bree", body="an inn on the road")
|
||||
|
||||
through_retrieval = retrieval.search(db, "notes_fts", "gate", kind=CHUNK_NOTE, limit=10)
|
||||
direct = fts.search_ids(db, "notes_fts", "gate", limit=10)
|
||||
|
||||
assert [hit.id for hit in through_retrieval] == [hit.id for hit in direct]
|
||||
|
||||
|
||||
async def test_with_no_model_a_query_embeds_to_nothing(db, user):
|
||||
assert await retrieval.embed_query(db, "anything") is None
|
||||
|
||||
|
||||
# --- Scoring --------------------------------------------------------------------
|
||||
def test_a_record_scores_as_its_best_piece(db, user, embedding_model):
|
||||
"""One paragraph that answers the question is what makes a document worth
|
||||
returning. Averaging would rank a long document about something else above a
|
||||
short one that says exactly the thing."""
|
||||
near = notes_service.create(db, owner=user, title="Near", body="x")
|
||||
far = notes_service.create(db, owner=user, title="Far", body="y")
|
||||
# Two pieces for `near`, one of them irrelevant. The good one has to win.
|
||||
_store_chunk(db, near, [1.0, 0.0])
|
||||
db.add(
|
||||
Chunk(
|
||||
owner_id=user.id,
|
||||
resource_type=CHUNK_NOTE,
|
||||
resource_id=near.id,
|
||||
ordinal=1,
|
||||
text="unrelated",
|
||||
vector=chunk_service.pack([0.0, 1.0]),
|
||||
dims=2,
|
||||
model_id="embed-1",
|
||||
)
|
||||
)
|
||||
_store_chunk(db, far, [0.7, 0.714])
|
||||
db.commit()
|
||||
|
||||
hits = retrieval.semantic_ids(db, CHUNK_NOTE, [1.0, 0.0], limit=10)
|
||||
|
||||
assert [hit.id for hit in hits][0] == near.id
|
||||
|
||||
|
||||
def test_a_vector_from_another_model_is_skipped(db, user, embedding_model):
|
||||
"""A change of embedding model with a rebuild still pending. Scoring across
|
||||
two spaces produces a confident wrong answer rather than a missing one."""
|
||||
note = notes_service.create(db, owner=user, title="Old", body="x")
|
||||
_store_chunk(db, note, [1.0, 0.0, 0.0]) # three wide
|
||||
|
||||
assert retrieval.semantic_ids(db, CHUNK_NOTE, [1.0, 0.0], limit=10) == []
|
||||
|
||||
|
||||
def test_a_semantic_only_match_is_found(db, user, embedding_model):
|
||||
"""The point of the whole feature: a record whose words do not appear in the
|
||||
query at all."""
|
||||
note = notes_service.create(db, owner=user, title="Doors", body="mellon")
|
||||
_store_chunk(db, note, [1.0, 0.0])
|
||||
|
||||
keyword = retrieval.search(db, "notes_fts", "how do I get in", kind=CHUNK_NOTE, limit=5)
|
||||
hybrid = retrieval.search(
|
||||
db, "notes_fts", "how do I get in", kind=CHUNK_NOTE, vector=[1.0, 0.0], limit=5
|
||||
)
|
||||
|
||||
assert keyword == []
|
||||
assert [hit.id for hit in hybrid] == [note.id]
|
||||
|
||||
|
||||
# --- Fusion ---------------------------------------------------------------------
|
||||
def test_fusion_keeps_what_only_one_side_found():
|
||||
"""Neither ranking's finds are dropped. That is the property that makes
|
||||
turning this on unable to make search worse."""
|
||||
keyword = [SearchHit(id="a", rank=-1.0), SearchHit(id="b", rank=-2.0)]
|
||||
meaning = [SearchHit(id="c", rank=0.9)]
|
||||
|
||||
fused = {hit.id for hit in retrieval.fuse(keyword, meaning, limit=10)}
|
||||
|
||||
assert fused == {"a", "b", "c"}
|
||||
|
||||
|
||||
def test_agreeing_on_a_record_ranks_it_first():
|
||||
"""RRF's whole behaviour: both lists having it beats either list alone."""
|
||||
keyword = [SearchHit(id="a", rank=-1.0), SearchHit(id="both", rank=-2.0)]
|
||||
meaning = [SearchHit(id="both", rank=0.9), SearchHit(id="c", rank=0.8)]
|
||||
|
||||
assert retrieval.fuse(keyword, meaning, limit=10)[0].id == "both"
|
||||
|
||||
|
||||
def test_fusing_with_nothing_is_the_other_list():
|
||||
keyword = [SearchHit(id="a", rank=-1.0), SearchHit(id="b", rank=-2.0)]
|
||||
|
||||
assert [hit.id for hit in retrieval.fuse(keyword, [], limit=10)] == ["a", "b"]
|
||||
|
||||
|
||||
def test_the_fused_score_is_larger_for_better():
|
||||
"""The opposite of bm25's convention, which is worth saying out loud rather
|
||||
than leaving somebody to infer it from a number that stopped being
|
||||
negative."""
|
||||
fused = retrieval.fuse([SearchHit(id="a", rank=-1.0)], [SearchHit(id="a", rank=0.9)])
|
||||
|
||||
assert fused[0].rank > 0
|
||||
|
||||
|
||||
# --- Indexing -------------------------------------------------------------------
|
||||
async def test_a_record_is_chunked_and_stored(db, user, embedding_model, mock_http):
|
||||
mock_http(_vector_answer([1.0, 0.0]))
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="the west gate " * 200)
|
||||
|
||||
written = await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
|
||||
db.expire_all()
|
||||
rows = list(db.scalars(select(Chunk).where(Chunk.resource_id == note.id)))
|
||||
assert written == len(rows) > 1
|
||||
assert all(row.dims == 2 and row.model_id == "embed-1" for row in rows)
|
||||
assert [row.ordinal for row in rows] == list(range(len(rows)))
|
||||
|
||||
|
||||
async def test_indexing_an_unchanged_record_is_free(db, user, embedding_model, mock_http):
|
||||
"""The hash is what makes it free, and what makes "is this current?"
|
||||
answerable without embedding anything."""
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(200, json={"data": [{"embedding": [1.0, 0.0], "index": 0}]})
|
||||
|
||||
mock_http(handler)
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="short")
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
before = len(requests)
|
||||
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
|
||||
assert len(requests) == before
|
||||
|
||||
|
||||
async def test_re_indexing_twice_is_idempotent(db, user, embedding_model, mock_http):
|
||||
mock_http(_vector_answer([1.0, 0.0]))
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="short")
|
||||
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id, force=True)
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id, force=True)
|
||||
|
||||
db.expire_all()
|
||||
assert len(list(db.scalars(select(Chunk).where(Chunk.resource_id == note.id)))) == 1
|
||||
|
||||
|
||||
async def test_a_failed_embedding_leaves_the_old_chunks(db, user, embedding_model, mock_http):
|
||||
"""Deleting first and failing half way through would leave a record indexed
|
||||
by half of itself, which ranks worse than not being indexed and looks like
|
||||
nothing at all."""
|
||||
mock_http(_vector_answer([1.0, 0.0]))
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="short")
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
|
||||
mock_http(lambda request: httpx.Response(500, json={"error": "down"}))
|
||||
notes_service.update(db, note, body="something else entirely")
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
|
||||
db.expire_all()
|
||||
assert len(list(db.scalars(select(Chunk).where(Chunk.resource_id == note.id)))) == 1
|
||||
|
||||
|
||||
async def test_a_record_with_no_text_left_drops_its_chunks(
|
||||
db, user, embedding_model, mock_http, monkeypatch
|
||||
):
|
||||
"""A guard rather than a state anything reaches today: every one of the four
|
||||
stores requires a title, so `text_of` is never empty for a record that
|
||||
exists. It is driven anyway, because the alternative to a guard here is a
|
||||
record whose chunks outlive its content -- and a store whose title becomes
|
||||
optional is a change nobody would think to test this against.
|
||||
"""
|
||||
mock_http(_vector_answer([1.0, 0.0]))
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="short")
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
|
||||
monkeypatch.setattr(indexing, "text_of", lambda row: "")
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
|
||||
db.expire_all()
|
||||
assert list(db.scalars(select(Chunk).where(Chunk.resource_id == note.id))) == []
|
||||
|
||||
|
||||
async def test_a_deleted_record_takes_its_chunks_with_it(db, user, embedding_model, mock_http):
|
||||
mock_http(_vector_answer([1.0, 0.0]))
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="short")
|
||||
await indexing.index_resource(CHUNK_NOTE, note.id)
|
||||
note_id = note.id
|
||||
|
||||
notes_service.delete(db, note)
|
||||
await indexing.index_resource(CHUNK_NOTE, note_id)
|
||||
|
||||
db.expire_all()
|
||||
assert list(db.scalars(select(Chunk).where(Chunk.resource_id == note_id))) == []
|
||||
|
||||
|
||||
def test_the_sweep_catches_what_had_no_loop_to_clean_up(db, user, embedding_model):
|
||||
"""The backstop for a delete with no event loop running -- a CLI command, or
|
||||
a cascade from removing an account."""
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="short")
|
||||
_store_chunk(db, note, [1.0, 0.0])
|
||||
note_id = note.id
|
||||
notes_service.delete(db, note)
|
||||
|
||||
assert indexing.sweep_orphans(db) == 1
|
||||
assert list(db.scalars(select(Chunk).where(Chunk.resource_id == note_id))) == []
|
||||
|
||||
|
||||
def test_the_text_of_a_record_is_the_indexed_columns(db, user):
|
||||
note = notes_service.create(db, owner=user, title="Moria", body="the west gate")
|
||||
|
||||
body = indexing.text_of(note)
|
||||
|
||||
assert "Moria" in body and "west gate" in body
|
||||
assert indexing.kind_of(note) == CHUNK_NOTE
|
||||
|
||||
|
||||
# --- The picker -----------------------------------------------------------------
|
||||
def test_a_chat_model_is_not_offered_as_an_embedder(db, user, embedding_model, client):
|
||||
connection = db.scalars(select(Connection)).first()
|
||||
db.add(
|
||||
Model(
|
||||
connection_id=connection.id,
|
||||
model_id="chat-1",
|
||||
capabilities_json={"tools": True},
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
page = client.get("/admin/extraction").text
|
||||
|
||||
assert 'value="embed-1"' in page
|
||||
assert 'value="chat-1"' not in page
|
||||
|
||||
|
||||
def test_a_model_that_lost_its_flag_is_named(db, user, embedding_model, client):
|
||||
"""A setting that vanishes from a picker is one nobody can tell from a
|
||||
setting that was never made."""
|
||||
model = db.scalar(select(Model).where(Model.model_id == "embed-1"))
|
||||
model.capabilities_json = {}
|
||||
db.commit()
|
||||
|
||||
assert "embed-1" in client.get("/admin/extraction").text
|
||||
|
||||
|
||||
def test_a_deleted_model_means_no_embedder_rather_than_an_error(db, user, embedding_model):
|
||||
db.delete(db.scalar(select(Model).where(Model.model_id == "embed-1")))
|
||||
db.commit()
|
||||
|
||||
assert indexing.embedder(db) is None
|
||||
assert indexing.enabled(db) is False
|
||||
Reference in New Issue
Block a user