lacing.store.migrations

Store-level schema migrations — the on-disk counterpart of the body ladder.

lacing.schema migrates annotation bodies (dict -> dict, keyed (schema_name, from_version)). This module is the same mental model one level down: it migrates the store — table layout, column rewrites, the meta.schema_version stamp — keyed (store_kind, from_version).

The split matters because the two ladders move different things:

  • a body migration rewrites one annotation’s payload and can run anywhere;

  • a store migration receives an open connection and performs DDL, row rewrites, and the version stamp for a whole database, atomically.

Contract (mirrors lacing.schema.register_migration()):

  • steps are single-step only (to_version == from_version + 1); chains compose by repeated lookup;

  • re-registering a (store_kind, from_version) pair replaces the previous entry — convenient in tests, intentional for hot-reload;

  • an upgrade function receives the open connection and must leave the store readable at to_version, including writing the new version into the ``meta`` table.

What the runner guarantees around each step (all verified inside the step’s transaction, so any breach rolls the whole step back):

  • the version is re-read under the write lock before the step runs — a concurrent migrator that already applied the step is detected and the step skipped, never double-applied (multi-process servers open the same .annot file);

  • the step stamped the version it claims to reach;

  • PRAGMA foreign_key_check is clean, and the annotations_rtree index agrees with the annotations table (see rebuild_annotations_rtree()).

Rules for step authors (sqlite):

  • Never call ``conn.executescript``, ``COMMIT`` or ``ROLLBACK`` inside a step — executescript implicitly commits the wrapper’s transaction, destroying atomicity. The runner detects a step that ended its transaction and fails loudly.

  • Foreign-key enforcement is pinned OFF during migration (sqlite cannot rebuild tables under FK enforcement); PRAGMA foreign_key_check before commit is the compensating guarantee.

  • A table rebuild (CREATE new → copy → DROP old → RENAME) must preserve rowids — INSERT INTO new (rowid, ...) SELECT rowid, ... FROM old — because annotations_rtree keys on them; rebuild the index with rebuild_annotations_rtree() afterwards.

Backends own their runners (transaction idiom differs per driver): migrate_annot_file() here for SQLite .annot files; a Postgres runner joins it with the first registered "postgres" step. Migration is opt-in — SqliteStore(path, migrate=True) or lacing migrate <path> — because silently rewriting a file on open is worse than refusing (lacing#15).

lacing.store.migrations.POSTGRES_KIND = 'postgres'

store_kind of the Postgres backend.

lacing.store.migrations.SQLITE_KIND = 'sqlite'

store_kind of the SQLite / .annot backend.

lacing.store.migrations.SQLITE_MIGRATION_BUSY_TIMEOUT_MS = 30000

How long a migrating connection waits on another writer’s lock.

Generous on purpose: when several workers race to open the same .annot with migrate=True, the losers should wait for the winner and then skip the already-applied steps, not fail at sqlite’s 5s default.

exception lacing.store.migrations.StoreMigrationError[source]

Raised when a store migration step is missing or fails.

lacing.store.migrations.migrate_annot_file(path: str | PathLike, *, to_version: int | None = None) → tuple[int, int][source]

Migrate a .annot file in place, returning (from, to) versions.

to_version defaults to the current build’s lacing.store.sqlite.SCHEMA_VERSION. Already-current files are a no-op (from == to). Each step runs in its own BEGIN IMMEDIATE transaction with the version re-checked under the lock, so concurrent migrators converge and an interrupted chain resumes from the last version that completed (idempotent).

Raises StoreMigrationError when the file does not exist, is not a .annot file, a step is missing, fails, or breaks one of the runner’s in-transaction guarantees.

lacing.store.migrations.migrate_sqlite_connection(conn: Connection, *, to_version: int) → int[source]

Run the SQLite ladder on an already-open connection.

The hook SqliteStore uses for its migrate=True opt-in; external callers with a file path want migrate_annot_file().

lacing.store.migrations.reachable_versions(store_kind: str, from_version: int) → tuple[int, ...][source]

Versions reachable from from_version by chaining registered steps.

Ascending, excluding from_version itself. Empty when no step leaves from_version — which is what a refusal message should say out loud instead of the bare “run a migration” it used to say.

lacing.store.migrations.rebuild_annotations_rtree(conn: Connection) → int[source]

Rebuild the interval index from the annotations table, in place.

For use inside a migration step after a table rebuild. Reproduces the store’s ULP-widening contract (bounds widened by one float ULP so float→exact-bound comparisons never drop hits — see lacing.store.sqlite). Returns the number of rows indexed.

lacing.store.migrations.register_store_migration(*, store_kind: str, from_version: int, to_version: int)[source]

Register a forward store migration from from_version to to_version.

The decorated function takes the backend’s open connection and must perform every change of the step — DDL, row rewrites, and the meta.schema_version write. Steps must be one version at a time (to_version == from_version + 1); the runner chains them.

Step authors: read the module docstring’s rules — no executescript / COMMIT / ROLLBACK inside a step, preserve rowids on table rebuilds, and rebuild the interval index with rebuild_annotations_rtree() if the annotations table was rebuilt.

Re-registering the same (store_kind, from_version) pair replaces the previous entry.