A list column backfilled with a dictionary

Reported as a 500 on a live instance, immediately after it updated, and read
off its journal rather than guessed at:

  ValueError: Attribute 'reasoning_efforts' does not accept objects of
              type <class 'dict'>

`Mapped[list[str]]` is not Optional, so the column is NOT NULL, so SQLite
demands a default for the rows that already exist. `_literal_default` chose one
by asking `column.type.python_type` -- and `MutableList.as_mutable(JSON)`
returns the *same* JSON type object with a listener attached rather than
subclassing it, so `python_type` is `dict` for both flavours. Every existing row
got '{}' in a list column, and MutableList refuses a dict while *loading*: not a
wrong value sitting quietly, an exception on every read of the table.

Model.reasoning_efforts was the first list-shaped JSON column this project had
ever added to a table that already had rows, so the flaw had been harmless since
the runner was written. 1.2.0 stepped on it.

The shape now comes from the column's Python-side default -- `default=list`
against `default=dict` -- which is the only thing that can tell the two apart.
And `repair_json_shapes` puts right what was already written, on start,
converging like ensure_fts beside it, narrow enough that a legitimate {} in a
dict column survives.

Why 1981 tests missed it: conftest builds a fresh database, where the column is
created from the model with its real default. The backfill only runs on a
database that already exists, so the suite had never once exercised the path
that broke. The new tests corrupt a row exactly as the migration did and assert
it loads again.

Verified against a backup of the reporting instance's own database: the load
raises before, eleven rows are repaired, all eleven models load after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/lembas/db/migrations.py

@@ -39,6 +39,29 @@ log = logging.getLogger(__name__)
 MANUAL_STEPS: list[str] = []
 
 
+def _default_shape(column: Column) -> type | None:
+    """`list` or `dict`, from the column's own Python-side default.
+
+    `default=list` and `default=dict` are how the two JSON flavours are
+    declared, and SQLAlchemy keeps the callable. Calling it is cheap and is the
+    only way to tell a MutableList column from a MutableDict one -- see the note
+    in `_literal_default`.
+    """
+    default = column.default
+    if default is None or not getattr(default, "is_callable", False):
+        return None
+    try:
+        # SQLAlchemy wraps a zero-argument callable to take a context.
+        produced = default.arg(None)
+    except Exception:  # noqa: BLE001 - a default we cannot call tells us nothing
+        return None
+    if isinstance(produced, list):
+        return list
+    if isinstance(produced, dict):
+        return dict
+    return None
+
+
 def _literal_default(column: Column) -> str | None:
     """A SQL literal to backfill an existing row's new column with.
 
@@ -63,8 +86,22 @@ def _literal_default(column: Column) -> str | None:
     if "JSON" in affinity:
         # MutableList columns must start as [] and MutableDict as {}; guessing
         # wrong makes the first read blow up rather than return empty.
-        python_type = getattr(column.type, "python_type", None)
-        return "'[]'" if python_type is list else "'{}'"
+        #
+        # 🚨 NOT `column.type.python_type`. `MutableList.as_mutable(JSON)`
+        # returns the *same* JSON type object with an event listener attached --
+        # it does not subclass or wrap it -- so the type cannot tell you which
+        # of the two it is, and `JSON.python_type` is `dict` for both. That read
+        # as "this is a dict column" for every list column, and the first one
+        # ever added by a migration (`Model.reasoning_efforts`, 1.2.0) arrived
+        # as `'{}'` on every existing row. `MutableList` refuses a dict, so the
+        # failure was not an empty list but a ValueError on *load* -- every page
+        # that lists models, 500, on an instance that had simply been updated.
+        #
+        # The Python-side default is the only honest signal: a JSONList column
+        # is declared `default=list` and a JSONDict one `default=dict`, and
+        # calling it says which. Anything that cannot be called or produces
+        # neither falls back to `{}`, which is what this always assumed.
+        return "'[]'" if _default_shape(column) is list else "'{}'"
     if "BOOL" in affinity:
         return "0"
     if any(token in affinity for token in ("INT", "FLOAT", "NUMERIC", "DECIMAL")):
@@ -190,6 +227,50 @@ def ensure_fts(engine: Engine) -> list[str]:
     return created
 
 
+def repair_json_shapes(engine: Engine) -> list[str]:
+    """Put right any JSON column backfilled with the wrong empty value.
+
+    `_literal_default` used to read the shape off `column.type.python_type`,
+    which is `dict` for a MutableList column as well as a MutableDict one -- so
+    the first list-shaped JSON column ever added by a migration arrived as
+    `'{}'` on every row that already existed. `MutableList` refuses a dict, and
+    refuses it while *loading*, so the symptom was not an empty list but a
+    `ValueError` and a 500 on every page that touched the table.
+
+    Converges, like `ensure_fts` beside it: it runs on every start, it is
+    idempotent, and on a database that was never damaged it does nothing. Only
+    the exact wrong value is rewritten -- `'{}'` in a column whose default
+    produces a list -- because `{}` cannot be a legitimate value there, while
+    anything else in that column might be somebody's data.
+    """
+    fixed: list[str] = []
+    inspector = inspect(engine)
+    known = set(inspector.get_table_names())
+
+    with engine.begin() as connection:
+        for table in Base.metadata.sorted_tables:
+            if table.name not in known:
+                continue
+            for column in table.columns:
+                if "JSON" not in column.type.__class__.__name__.upper():
+                    continue
+                if _default_shape(column) is not list:
+                    continue
+                result = connection.execute(
+                    text(
+                        f'UPDATE "{table.name}" SET "{column.name}" = \'[]\' '
+                        f'WHERE "{column.name}" = \'{{}}\''
+                    )
+                )
+                if result.rowcount:
+                    fixed.append(f"{table.name}.{column.name} ({result.rowcount} row(s))")
+                    log.warning(
+                        "repaired %s.%s on %d row(s): was '{}' in a list column",
+                        table.name, column.name, result.rowcount,
+                    )
+    return fixed
+
+
 def sync_schema(engine: Engine) -> list[str]:
     """Bring the database up to the declared schema. Returns what it changed."""
     import lembas.db.models  # noqa: F401  (registers every table on the metadata)
@@ -219,6 +300,14 @@ def sync_schema(engine: Engine) -> list[str]:
                 changes.append(f"add column {table.name}.{column.name}")
                 log.info("schema: %s", statement)
 
+    # Before the search indexes, and before anything can try to load a row:
+    # a column left holding the wrong empty value makes the ORM raise on read.
+    try:
+        for repair in repair_json_shapes(engine):
+            changes.append(f"repair {repair}")
+    except Exception:  # noqa: BLE001 - a repair that fails must not stop a start
+        log.exception("could not repair JSON column shapes")
+
     try:
         for index in ensure_fts(engine):
             changes.append(f"create search index {index}")