Building Fallback Routing for Legacy System Downtime

16 min read 1 sub-topic

Fallback routing is the resilience tier of the Core Architecture & Code Taxonomy for Municipal Permits: the layer that keeps clerks issuing permits and inspectors logging results when the authoritative system-of-record goes dark. Municipal permitting runs on aging mainframes, vendor-hosted schedulers, and legacy relational databases that experience frequent unplanned downtime. When the primary routing layer loses its connection to that record system, statutory processing clocks keep ticking even though no work can be committed. This guide shows Python automation builders how to design a degraded-mode path that absorbs an outage, preserves a defensible audit trail, and reconciles cleanly once the primary recovers.

Problem Statement and Scope

Without a deliberate fallback path, a downed backend stalls the entire permitting front office. A clerk clicks “issue” and the request blocks on a database that will not answer; a field inspector tries to record a failed footing inspection and the call times out. Because municipal deadlines are set by ordinance — a 10-day plan-review window, a 48-hour re-inspection guarantee — an outage that lasts hours can translate directly into a missed statutory obligation and an audit finding.

Three groups feel the failure. Municipal clerks lose the ability to advance applications and have no way to know whether their last action was saved. Python automation builders get paged because synchronous calls pile up against an unresponsive endpoint and exhaust the connection pool. Compliance officers cannot certify chain-of-custody when work performed during the outage has no durable record of when it was accepted versus when it was finally committed.

The scope of this component is the routing decision itself. Its input is an inbound work item — a new permit application, a status transition, an inspection result — plus the live health signal of the primary system-of-record. Its output is a route: send the item straight through to the primary, or divert it into a durable degraded-mode store that accepts, validates locally, queues, and later reconciles. Everything below builds toward making that route automatic, deterministic, and invisible to the person at the counter.

Fallback routing state machine for legacy system-of-record downtime Inbound work items pass through a routing layer governed by a circuit breaker. When the breaker is closed the item is written straight to the primary system-of-record (normal mode). When consecutive failures open the breaker, the router diverts the item into a durable SQLite queue after local validation (degraded mode). A reconciler then drains that queue oldest-first into the primary using idempotent upserts once the half-open canary probe confirms recovery. Inbound work item Routing layer circuit-breaker gate reads primary health System-of-record (primary backend) Validate + enqueue durable queue · SQLite WAL Reconciler drains queue NORMAL · breaker CLOSED DEGRADED · breaker OPEN drain oldest-first replay → idempotent upsert CLOSED — route to primary HALF-OPEN — single canary probe OPEN — divert to degraded queue
The circuit breaker is the routing signal: closed routes to the primary, open diverts to the durable queue, and a reconciler replays queued items once a half-open probe confirms recovery.

Prerequisites

This component targets Python 3.10+ (the examples use match statements, X | Y unions, and slots=True dataclasses). Install the runtime dependencies:

pip install "httpx>=0.27" "pybreaker>=1.2" "pydantic>=2.6" "structlog>=24.1" "sqlite-utils>=3.36"

You will also need:

  • A durable local store for the degraded-mode queue. It must survive a process restart, so an in-memory list is not enough. A single-file SQLite database (WAL mode) is the pragmatic default for a constrained municipal server; a Redis stream or a small Postgres table works equally well. The examples assume a thin FallbackQueue you can back with any of these.
  • Idempotency keys on every work item. Each application or transition must carry a stable identifier — a county-issued application number, or a SHA-256 of the canonical payload — so the reconciler can replay the queue without creating duplicate permits. This is the same key discipline used in error handling and retry logic for ingestion pipelines.
  • A local copy of the validation contract. Degraded mode must validate without calling the backend, which means the JSON schema for building permits and any controlled vocabularies have to be cached on the routing host.
  • Write access to a structured log sink so every route decision, queued item, and reconciliation event is queryable during an audit.

Decouple Intake From the System-of-Record

Resilience starts before any outage, in the topology. Treat the legacy backend as an asynchronous dependency, never a synchronous gatekeeper that the clerk’s request blocks on. The router owns a small local state machine that tracks each application’s lifecycle stage independently, so preliminary review, fee estimation, and conditional approval can advance whether or not the backend answers.

from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any


class Stage(str, Enum):
    INTAKE = "intake"
    PRELIM_REVIEW = "prelim_review"
    FEE_ESTIMATED = "fee_estimated"
    CONDITIONAL_APPROVAL = "conditional_approval"
    COMMITTED = "committed"          # durably written to the system-of-record


@dataclass(slots=True)
class WorkItem:
    idem_key: str
    application_no: str
    stage: Stage
    payload: dict[str, Any]
    accepted_at: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )
    committed_at: datetime | None = None   # set only after reconciliation


# The local machine advances state without ever blocking on the backend.
_ALLOWED: dict[Stage, set[Stage]] = {
    Stage.INTAKE: {Stage.PRELIM_REVIEW},
    Stage.PRELIM_REVIEW: {Stage.FEE_ESTIMATED},
    Stage.FEE_ESTIMATED: {Stage.CONDITIONAL_APPROVAL},
    Stage.CONDITIONAL_APPROVAL: {Stage.COMMITTED},
}


def advance(item: WorkItem, to: Stage) -> WorkItem:
    """Move an item forward only along a permitted transition. Keeping the
    machine local means a downed backend never freezes the counter workflow."""
    if to not in _ALLOWED.get(item.stage, set()):
        raise ValueError(f"illegal transition {item.stage} -> {to}")
    item.stage = to
    return item

Because the lifecycle lives in the router, a localized backend failure cannot cascade into agency-wide paralysis. The clerk keeps working; only the final COMMITTED transition needs the primary to be reachable, and that transition is exactly what fallback routing defers.

Detect Failure and Switch Modes Automatically

The switch into degraded mode must be automatic and deterministic — no clerk should have to know an outage is happening. A circuit breaker at the routing boundary is the cleanest mechanism: it watches the recent failure rate against the backend and, once a threshold is crossed, “opens” and fails fast for a cool-down window instead of letting requests pile up against a dependency that is already on the floor. The full tuning of thresholds, timeouts, and probe behaviour is covered in configuring circuit breakers for permit database timeouts; here the breaker is simply the signal that drives the route.

import httpx
import pybreaker
import structlog

log = structlog.get_logger()

# Open after 4 consecutive failures; stay open 30s before a single trial probe.
sor_breaker = pybreaker.CircuitBreaker(fail_max=4, reset_timeout=30)


class SystemOfRecordDown(Exception):
    """Raised when the primary is unreachable so the router can divert."""


@sor_breaker
async def commit_to_sor(client: httpx.AsyncClient, item: WorkItem) -> None:
    """Attempt the authoritative write. While the breaker is open this raises
    CircuitBreakerError immediately, which the router treats as 'go degraded'."""
    resp = await client.post(
        "https://sor.internal.gov/permits",
        json=item.payload,
        headers={"Idempotency-Key": item.idem_key},
        timeout=5.0,
    )
    resp.raise_for_status()


def primary_is_available() -> bool:
    """The breaker's own state is the routing signal: closed/half-open == try
    the primary; open == fail fast into degraded mode."""
    return sor_breaker.current_state != pybreaker.STATE_OPEN

Recovery is equally automatic. After the cool-down, the breaker enters a half-open state and lets a single canary write through; if it succeeds the breaker closes and normal routing resumes, and if it fails the cool-down restarts. This removes manual intervention from the most stressful moment of an outage and eliminates human error in the routing decision.

Accept and Validate Submissions in Degraded Mode

When the primary is unavailable the router diverts the item to a durable local queue, but it must still validate before accepting — degraded does not mean unguarded. Validation runs against the cached schema so a malformed application is rejected at the counter rather than discovered hours later during reconciliation. Spatial checks fall back to cached parcel geometry instead of a live GIS query, mirroring the approach in mapping municipal zoning overlays to GIS data.

import json
import sqlite3
from pydantic import BaseModel, ValidationError, field_validator


class PermitSubmission(BaseModel):
    """Local mirror of the authoritative contract, loaded from cached schema.
    Validating here keeps degraded-mode intake as strict as normal intake."""
    application_no: str
    parcel_id: str
    work_class: str
    declared_valuation: float

    @field_validator("declared_valuation")
    @classmethod
    def non_negative(cls, v: float) -> float:
        if v < 0:
            raise ValueError("declared_valuation must be >= 0")
        return v


def accept_degraded(db: sqlite3.Connection, raw: dict[str, Any]) -> WorkItem:
    """Validate against the cached contract, then enqueue durably. Any
    validation failure is surfaced to the clerk immediately, not deferred."""
    model = PermitSubmission(**raw)             # raises ValidationError on bad data
    key = idempotency_key(raw)
    item = WorkItem(
        idem_key=key,
        application_no=model.application_no,
        stage=Stage.CONDITIONAL_APPROVAL,
        payload=raw,
    )
    db.execute(
        "INSERT OR IGNORE INTO fallback_queue (idem_key, application_no, payload, accepted_at) "
        "VALUES (?, ?, ?, ?)",
        (key, model.application_no, json.dumps(raw), item.accepted_at.isoformat()),
    )
    db.commit()
    log.info("permit.accepted_degraded", idem_key=key, application_no=model.application_no)
    return item

The INSERT OR IGNORE against a unique idem_key makes the enqueue itself idempotent: if a clerk double-submits during a flaky moment, the second write is a harmless no-op rather than a duplicate queued record. The SQLite database (run in WAL mode) gives the queue durability across a process restart without standing up additional infrastructure on a constrained municipal host.

Reconcile the Queue When the Primary Recovers

The real test of the design is recovery. Once the breaker closes, a reconciler drains the local queue into the system-of-record, applies any deferred business logic, and resolves conflicts without creating duplicate records or violating deadlines. Idempotent upserts keyed on the stable identifier are what make a replay safe even if the reconciler itself is interrupted and restarted.

import hashlib


def idempotency_key(record: dict[str, Any]) -> str:
    """Stable SHA-256 over the canonical payload so a replay converges to the
    same backend row regardless of dict ordering or retry timing."""
    canonical = json.dumps(record, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


async def reconcile(db: sqlite3.Connection, client: httpx.AsyncClient) -> int:
    """Drain queued items oldest-first into the primary. Each successful commit
    stamps committed_at and clears the row; failures are left for the next pass."""
    committed = 0
    rows = db.execute(
        "SELECT idem_key, payload FROM fallback_queue ORDER BY accepted_at ASC"
    ).fetchall()
    for idem_key, payload_json in rows:
        if not primary_is_available():
            break                               # breaker reopened — stop, retry later
        item = WorkItem(
            idem_key=idem_key,
            application_no=json.loads(payload_json)["application_no"],
            stage=Stage.CONDITIONAL_APPROVAL,
            payload=json.loads(payload_json),
        )
        try:
            await commit_to_sor(client, item)   # idempotent via Idempotency-Key header
        except (httpx.HTTPError, pybreaker.CircuitBreakerError):
            continue                            # leave queued; do not lose the record
        db.execute("DELETE FROM fallback_queue WHERE idem_key = ?", (idem_key,))
        db.commit()
        committed += 1
        log.info("permit.reconciled", idem_key=idem_key)
    return committed

Draining oldest-first preserves submission order, which matters when fee schedules or queue position depend on time of filing. Re-checking primary_is_available() inside the loop means a backend that wobbles again mid-reconciliation stops the drain cleanly rather than hammering a half-recovered dependency — the same restraint applied in high-throughput pipelines covered under implementing async batch processing for high-volume submissions.

Configuration Reference

Externalize every tunable so a flaky vendor scheduler and a fast internal database can use different policies without code changes.

Parameter Type Default Municipal-context notes
breaker_fail_max int 4 Consecutive backend failures before the router diverts to degraded mode. Lower it for systems with strict deadline SLAs so you fail over sooner.
breaker_reset_s int 30 Cool-down before the half-open canary write. Match to the backend’s typical maintenance-window length.
sor_timeout_s float 5.0 Per-call timeout on the authoritative write. Keep tight so a hung connection trips the breaker instead of blocking the counter.
queue_path str /var/lib/permits/fallback.db Durable SQLite file (WAL mode). Must live on persistent disk, not tmpfs, so it survives a restart.
reconcile_batch int 200 Items drained per reconciliation pass. Cap it so a long outage’s backlog does not saturate the recovered backend.
schema_cache_ttl_h int 24 Max age of the cached validation contract before degraded mode refuses new intake. Tighten when ordinances change often.
queue_retention_days int 2555 How long reconciled-then-archived records are kept (~7 years) to satisfy public-records retention law; never auto-purge below the local mandate.

Error Handling and Edge Cases

Legacy municipal infrastructure fails in specific, recurring ways that a naive failover mishandles:

  • The “succeeds late” timeout. An older backend may commit a write and then time out your client before acknowledging it. The router treats this as a failure and queues the item, so reconciliation later replays it. This is exactly why the Idempotency-Key header and the canonical-hash key are non-negotiable: the replay collides on the key and no-ops instead of double-issuing the permit.
  • Stale validation cache. If the cached schema is older than schema_cache_ttl_h, degraded mode must refuse new intake rather than accept submissions it cannot validate against current ordinances. Surface a clear counter message (“intake paused — validation rules out of date”) instead of silently queuing unvalidated data. Keep the cache fresh against versioning permit code taxonomies for annual updates.
  • Partial reconciliation. If the reconciler crashes mid-drain, some items are committed and deleted while others remain queued. Because each commit is its own transaction and each item is idempotent, simply re-running the reconciler is always safe — there is no “resume from offset” bookkeeping to corrupt.
  • Conflicting edits during the outage. A clerk edits an application in degraded mode while a separate channel already committed a change to the same record. Resolve on reconciliation with a deterministic rule — last-writer-wins on accepted_at, or quarantine for human review — and never let the reconciler guess silently.
  • Disk exhaustion on the queue host. A long outage can grow the SQLite file past available space. Monitor free space and, on the threshold, fail intake loudly rather than corrupting the queue mid-write.
def accept_with_guards(db: sqlite3.Connection, raw: dict[str, Any],
                       schema_age_h: float, free_disk_mb: float) -> WorkItem:
    """Degraded-mode intake refuses rather than risk un-auditable data."""
    if schema_age_h > 24:
        raise RuntimeError("intake paused: validation contract is stale")
    if free_disk_mb < 100:
        raise RuntimeError("intake paused: insufficient durable queue space")
    try:
        return accept_degraded(db, raw)
    except ValidationError as exc:
        log.warning("permit.rejected_degraded", errors=exc.errors())
        raise

Testing and Verification

Failover code that is not tested against failure is just hope. Drive the breaker deterministically and assert the route, rather than relying on a real flaky backend in CI.

import pytest


@pytest.mark.asyncio
async def test_router_diverts_when_breaker_open(db, mock_sor_failing) -> None:
    # Four failing commits should open the breaker and flip routing to degraded.
    for _ in range(4):
        with pytest.raises(Exception):
            await commit_to_sor(mock_sor_failing, sample_item())
    assert primary_is_available() is False
    item = accept_degraded(db, sample_payload())          # routed to local queue
    assert db.execute("SELECT count(*) FROM fallback_queue").fetchone()[0] == 1


@pytest.mark.asyncio
async def test_reconcile_is_idempotent(db, mock_sor_ok) -> None:
    accept_degraded(db, sample_payload())
    first = await reconcile(db, mock_sor_ok)
    second = await reconcile(db, mock_sor_ok)              # queue already drained
    assert first == 1 and second == 0                     # no double-commit
    assert db.execute("SELECT count(*) FROM fallback_queue").fetchone()[0] == 0

Beyond unit tests, run a failover drill: script a mock backend that returns timeouts for 60 seconds and then recovers. Confirm that intake continued throughout (no item lost), that the breaker opened and later closed on its own, that the queue drained oldest-first, and that replaying the drill produces zero duplicate permits. A passing drill is your evidence, during an audit, that no application filed during an outage was silently dropped or double-issued.

Integration Notes

Fallback routing is the platform-layer counterpart to the ingestion stack’s per-item resilience. When the ingestion pipeline’s circuit breaker opens against an upstream county source, that open state is the trigger that activates this degraded-mode path — the connection is documented from the other side in error handling and retry logic for ingestion pipelines, where a flood of failures redirects here instead of quarantining records one by one.

Two adjacent components wire in directly. Degraded-mode access must still honour who is allowed to do what, so the router defers authorization to the same policy used in implementing role-based access for clerk portals — an outage is never a reason to widen permissions. And the cached validation contract that degraded mode relies on is the local projection of the cross-referencing of state and local building codes, so the same code mappings apply whether the primary is up or down.

Frequently Asked Questions

When should the router switch into degraded mode versus just retrying?

Retry only buys time against a brief blip; switching modes is for a sustained outage. Let the circuit breaker make the call: short transient faults are absorbed by per-request retries, but once consecutive failures cross breaker_fail_max the breaker opens and the router diverts every new item to the durable queue. That keeps the counter responsive instead of stalling each clerk action behind a five-second timeout.

How do I guarantee no duplicate permits after reconciliation?

Derive a stable idempotency key from a canonical hash of the payload and send it as an Idempotency-Key on the authoritative write. A replay of a “succeeded late” item then collides on that key and no-ops at the backend instead of issuing a second permit. Pair that with one-transaction-per-item draining so an interrupted reconciler can simply be re-run.

Is it safe to accept submissions without the backend’s validation?

Yes, provided you validate against a cached copy of the same contract. Degraded mode loads the JSON schema for building permits locally and refuses intake if that cache is older than its TTL, so applications are held to the current ordinances even while the primary is unreachable — you never accept data you cannot later commit cleanly.

Where does the degraded-mode queue live so it survives a restart?

On persistent disk, never in memory. A single-file SQLite database in WAL mode is the pragmatic default for a constrained municipal server; a Redis stream or a small Postgres table works where that infrastructure already exists. The one hard requirement is durability across a process or host restart, because an outage and a reboot frequently coincide.

Explore deeper

Sub-topic Configuring Circuit Breakers for Permit Database Timeouts