Migrating Permit Records Across Taxonomy Versions

11 min read

Migrating permit records across taxonomy versions means remapping existing files from an outgoing code edition’s classification to the incoming one without corrupting history or losing the original codes. This guide extends Versioning Permit Code Taxonomies for Annual Updates, part of the Core Architecture & Code Taxonomy for Municipal Permits track, to the one operation the versioning layer deliberately avoids for in-flight files but eventually must perform for the record store.

The Focused Problem: Remapping Records When Codes Change, Reversibly

The versioning layer pins each in-flight permit to the edition in force on its filing date, so an annual code update never retroactively re-judges an active application. That is correct at request time. But the record store still holds thousands of historical and reference files coded against editions that are drifting toward deprecation, and some are legitimately migrated forward — a stalled project resumes under a new edition, a reporting system needs every record expressed in the current classification, a superseded local code splits one category into three. Migration is the controlled, audited process of moving those records from an old taxonomy’s codes to a new one’s.

The distinct failure here is a lossy one-way overwrite. A naive migration reads each record, looks up the new code, writes it back over the old value, and moves on. The first time the crosswalk is wrong — a code that split, a category the new edition dropped, an ambiguous many-to-one merge — the record is silently mis-coded, and because the original code was overwritten, there is no way to detect the error or roll it back. Compliance officers then cannot certify what code a permit was originally filed under, which most permitting statutes require preserving. A migration that cannot be reversed and cannot prove the original classification is not a migration; it is data loss with extra steps.

The input is a batch of permit records plus a crosswalk mapping old codes to new ones. The output is each record re-expressed in the target edition, validated against the new schema, with its original code preserved verbatim for audit, and a reverse mapping recorded so the whole batch can be rolled back to the byte.

Crosswalk-driven, reversible migration of permit records between taxonomy editions A source permit record coded against the 2024 edition flows left to right into a crosswalk resolver that maps its old code onto the 2025 edition and classifies the result as one-to-one, a split requiring a choice, or a dropped code with no target. Cleanly mapped records are dual-written: the new code is set while the original code is preserved verbatim in an immutable original_code field. Each migrated record is validated against the new edition's schema; records that pass are committed to the store, while ambiguous splits and dropped codes are routed to a quarantine queue for manual review rather than guessed. A reverse mapping is recorded alongside each committed record so the full batch can be rolled back to its exact prior state. Source record code @ 2024-IRC R310.1 Crosswalk resolver old → new code 1:1 · split · dropped @ 2025-IRC Dual-write set new code preserve original_code immutable Quarantine split / dropped manual review clean Validate vs new schema pass / fail Commit to record store record reverse map Rollback reverse map → prior state original_code kept for audit & reversal

Step 1 — Build the Crosswalk and Classify Every Mapping

Migration is only as trustworthy as its crosswalk — the table mapping each old code to its target. Rather than a flat dict[str, str], classify every entry, because the hard cases are not the one-to-one renames; they are the splits (one old code becomes several), the merges (several collapse into one), and the drops (a category the new edition removed). The crosswalk itself is generated and maintained upstream, so build this migration on top of the mappings produced when automating cross-walk tables between IBC and local amendments.

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum


class MapKind(str, Enum):
    ONE_TO_ONE = "one_to_one"   # unambiguous rename or renumber
    SPLIT = "split"             # old code maps to several — needs a choice
    DROPPED = "dropped"         # no target in the new edition — quarantine


@dataclass(frozen=True, slots=True)
class CrosswalkEntry:
    old_code: str
    new_codes: tuple[str, ...]  # one target for 1:1, several for a split, () if dropped
    kind: MapKind


def classify(old_code: str, targets: tuple[str, ...]) -> CrosswalkEntry:
    if not targets:
        return CrosswalkEntry(old_code, (), MapKind.DROPPED)
    kind = MapKind.ONE_TO_ONE if len(targets) == 1 else MapKind.SPLIT
    return CrosswalkEntry(old_code, targets, kind)

Only ONE_TO_ONE entries migrate automatically; splits and drops are routed to review rather than guessed, because a wrong automatic choice on a split is exactly the silent mis-coding that a migration must never introduce.

Step 2 — Remap Reversibly, Preserving the Original Code

The core write is dual: set the new code and preserve the original verbatim in an immutable field. The original code is what proves, during a records request, which classification a permit was filed under — and it is what a rollback reads to restore prior state. Never overwrite it.

from dataclasses import dataclass, field


@dataclass(slots=True)
class MigrationResult:
    application_no: str
    original_code: str          # preserved verbatim, never overwritten
    new_code: str | None        # None when quarantined
    from_edition: str
    to_edition: str
    quarantined: bool = False
    reason: str = ""


def migrate_record(record: dict, entry: CrosswalkEntry,
                   to_edition: str) -> MigrationResult:
    """Produce the migrated record without ever losing the original code."""
    original = record["code"]
    if entry.kind is not MapKind.ONE_TO_ONE:
        # Split or dropped: do not guess. Quarantine with the reason attached.
        return MigrationResult(record["application_no"], original, None,
                               record["edition"], to_edition,
                               quarantined=True, reason=entry.kind.value)
    record["code"] = entry.new_codes[0]          # the new classification
    record.setdefault("original_code", original)  # set once, immutable thereafter
    record["edition"] = to_edition
    return MigrationResult(record["application_no"], original,
                           entry.new_codes[0], record["edition"], to_edition)

Using setdefault for original_code means a record migrated twice across three editions still carries its first code, so the provenance chain never collapses to the most recent hop.

Step 3 — Validate Each Migrated Record Against the New Schema

A remapped code is not a migrated record until it validates against the edition it now claims. The new edition may add required fields, tighten a pattern, or drop one the old record carried, so validate every record against the target schema before it is eligible to commit — the same canonical JSON schema for building permits the rest of the platform uses.

from pydantic import BaseModel, ConfigDict, Field, ValidationError


class MigratedPermit(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")
    application_no: str
    code: str = Field(min_length=1)
    original_code: str = Field(min_length=1)   # provenance must survive migration
    edition: str


def validate_against_new_schema(record: dict) -> tuple[bool, str]:
    try:
        MigratedPermit.model_validate(record)
        return True, ""
    except ValidationError as exc:
        # A schema failure means the crosswalk or the record is wrong — never
        # force-commit past it; route to quarantine with the error captured.
        return False, exc.errors()[0]["msg"]

Run this in a dry-run pass over the whole batch first and commit nothing until the failure count is zero or every failure is an accepted quarantine — a validation-clean rehearsal is your evidence that the migration will not corrupt the store.

Step 4 — Dual-Write, Backfill, and Keep a Reverse Map for Rollback

For a live store, cut over with a dual-write window rather than a stop-the-world swap: new writes populate both the old and new code fields while a backfill job migrates the historical tail, so reads never see a half-migrated record. Record a reverse mapping for every committed change so the entire batch can be rolled back deterministically.

def rollback(results: list[MigrationResult], store) -> int:
    """Restore every committed record to its pre-migration code. The reverse
    map is just original_code — preserved in Step 2 — so rollback is exact."""
    restored = 0
    for r in results:
        if r.quarantined:
            continue                     # never committed; nothing to undo
        store.update(r.application_no, code=r.original_code,
                     edition=r.from_edition)
        restored += 1
    return restored

Because original_code and from_edition were preserved on every record, rollback needs no separate backup — the reverse map lives in the data itself, and restoring is a bounded, testable operation rather than a database restore from tape.

Parameter and Flag Reference

Parameter Type Default Municipal-context notes
dry_run bool True Rehearse the full batch and report failures before any write; commit only when clean.
preserve_original_code bool True Always keep the filing-time code; required to satisfy records law and to roll back.
on_split str "quarantine" Splits need a human choice; quarantine never guesses a target automatically.
on_dropped str "quarantine" A code with no target is flagged for review, never silently discarded.
batch_size int 500 Commit in bounded transactions so a failure rolls back a batch, not the whole run.
dual_write_window_days int 14 How long new writes populate both editions while the backfill drains the tail.
validate_before_commit bool True Every migrated record must pass the new schema before it is eligible to commit.

Common Failure Patterns and Fixes

The original code is overwritten

Writing the new code over the old field destroys the only proof of the filing-time classification and makes rollback impossible. Preserve original_code with setdefault so it is written once and never mutated, even across multiple migrations.

A split code is auto-resolved to the wrong target

When one old code maps to several, picking the first silently mis-codes every ambiguous record. Classify splits explicitly and quarantine them for a reviewer; only one_to_one entries migrate without a human decision.

Migration commits without schema validation

A record can carry a valid new code yet violate the new edition’s schema — a now-required field left blank. Validate every migrated record against the target schema in a dry run and commit nothing until failures are zero or accepted quarantines.

Reads see half-migrated records mid-cutover

A stop-the-world overwrite leaves readers observing a mix of old and new codes. Use a dual-write window so both fields stay populated while the backfill drains the historical tail, and cut reads over only after the backfill completes.

The batch cannot be rolled back

Discovering a bad crosswalk after committing thousands of records, with no reverse path, forces a restore from backup and loses everything since. Record the reverse map (original_code + from_edition) on every committed record so rollback restores exact prior state in one pass.

Audit and Logging Guidance

Emit one structured, append-only entry per migrated record — application number, original_code, new code, from- and to-edition, and the outcome (committed or quarantined with its reason) — so a compliance officer can prove exactly how any file was reclassified and reconstruct the filing-time code on demand. Log the batch as a whole too: who ran it, against which crosswalk version, the dry-run failure count, and the commit timestamp, gating the run behind the same role-based access for clerk portals that guards any privileged data change. Retain these records for the full state-mandated period, since the original-code provenance is itself a public-records artifact. Where a migration batch ingests from an external release feed, route transient failures through error handling and retry logic for ingestion pipelines so a partial batch retries and quarantines rather than committing half a run.

Frequently Asked Questions

If the versioning layer pins in-flight files to their filing edition, why migrate at all?

In-flight resolution is deliberately non-destructive, but the record store still holds historical and reference files that reporting, analytics, and resumed projects need expressed in the current classification. Migration is the controlled path for those records — always preserving the original code so the filing-time classification is never lost.

How do I preserve the original code without bloating every record?

One immutable original_code field set once with setdefault is enough; it survives multiple migrations because it is only ever written the first time. That single field is both your records-law provenance and your reverse map for rollback, so it earns its place many times over.

What happens to a code the new edition dropped entirely?

Quarantine it — never silently discard or force it into the nearest category. A dropped code means the new edition made a deliberate classification change that a human must adjudicate, and guessing would introduce exactly the silent mis-coding a reversible migration exists to prevent.