Configuring Circuit Breakers for Permit Database Timeouts

12 min read

This walkthrough sits under Building Fallback Routing for Legacy System Downtime, part of the Core Architecture & Code Taxonomy for Municipal Permits. The parent guide builds the degraded-mode path that keeps a permit office running through an outage; here we tune the one component that decides when that path activates — the circuit breaker guarding calls to the permit database. Get its thresholds wrong and you either fail over on every harmless query hiccup or, worse, let a saturated legacy backend hang every clerk at the counter.

Why an Untuned Breaker Fails the Permit Office

Municipal permitting runs on strict statutory clocks — a 10-day plan-review window, a 48-hour re-inspection guarantee — so database availability is not a convenience, it is a compliance obligation. Legacy permit databases (older PostgreSQL, Oracle, or SQL Server instances behind a vendor scheduler) fail in slow, ambiguous ways: connection-pool saturation during an overnight ETL load, a query planner that suddenly chooses a sequential scan on a zoning-overlay join, or a firewall NAT table that silently drops idle sessions. A naive try/except around the query does nothing about the cascade: synchronous calls pile up against the unresponsive socket, the application thread pool drains, and a localized database stall becomes an agency-wide freeze.

A correctly configured circuit breaker turns that cascade into a fast, deterministic decision. It watches the recent failure rate, trips to an open state once a threshold is crossed, and fails fast for a cool-down window instead of queuing work behind a dependency that is already down. That fast-fail is precisely the signal the parent fallback router uses to divert intake into its durable local queue. The tuning below is what separates a breaker that protects the counter from one that flaps uselessly.

Permit-database circuit-breaker state machine Three states form a cycle. CLOSED passes calls through to the legacy permit database and counts only infrastructure faults; once the fault rate over the 60-second sliding window reaches fail_max the breaker trips to OPEN. OPEN fails fast for the cool-down window, serving the cached path, and is the signal that drives the parent fallback router into degraded mode. After reset_timeout elapses the breaker enters HALF-OPEN and admits a single canary probe — if the probe succeeds the breaker returns to CLOSED, if it fails the breaker reopens and restarts the cool-down. CLOSED pass through · count faults OPEN fail fast · serve cached read HALF-OPEN admit one canary probe trip · fault rate ≥ fail_max reset_timeout elapsed probe succeeds probe fails OPEN = fail-fast signal CircuitBreakerError diverts intake to the degraded-mode queue

Step 1: Map Driver Exceptions to Breaker Failures

A breaker only counts what you tell it to. The single most common mistake is letting it treat every exception as a backend failure: a ValidationError or a 404 from a missing application is a business outcome, not an infrastructure outage, and counting it will trip the breaker for the wrong reason. Enumerate exactly the driver-level exceptions that mean “the database itself is unreachable or unresponsive,” and exclude everything else.

import psycopg
import pybreaker
from sqlalchemy.exc import OperationalError, TimeoutError as SAQueuePoolTimeout

# Only infrastructure faults should age the breaker toward `open`. Business
# errors (constraint violations, not-found) must pass through untouched.
DB_INFRA_FAULTS: tuple[type[Exception], ...] = (
    OperationalError,            # SQLAlchemy: lost connection, server shutdown
    SAQueuePoolTimeout,          # pool checkout exceeded `pool_timeout`
    psycopg.OperationalError,    # raw driver: connection refused / reset
    psycopg.errors.AdminShutdown,
    TimeoutError,                # asyncio.wait_for around the query
)


class _DBFaultListener(pybreaker.CircuitBreakerListener):
    """Hook so we can log the exact fault that nudged the breaker."""

    def failure(self, cb: pybreaker.CircuitBreaker, exc: BaseException) -> None:
        log.warning("permitdb.fault", breaker=cb.name, fault=type(exc).__name__)

By passing exclude to the breaker (Step 3) rather than catching broadly, a psycopg.errors.UniqueViolation from a duplicate permit number stays a 409 to the caller and never moves the breaker. This keeps the failure signal clean — the breaker trips on real outages, the kind documented from the ingestion side in error handling and retry logic for ingestion pipelines, not on routine validation.

Step 2: Set a Timeout Hierarchy Below the Breaker

The breaker can only react to a failure if the call returns a failure quickly. Decouple three timeouts and keep them strictly ordered so that pool exhaustion and slow queries surface as exceptions long before an OS socket wait would block a thread indefinitely. The rule: application-level timeouts must be smaller than the kernel tcp_user_timeout.

from sqlalchemy.ext.asyncio import create_async_engine

# Order: statement < query wait < pool checkout < socket. Each layer trips the
# next-coarser guard, so a hung legacy backend fails fast instead of hanging.
engine = create_async_engine(
    "postgresql+psycopg://[email protected]/permits",
    pool_size=12,            # cap at DB max_connections minus superuser/replication slots
    max_overflow=0,          # never burst past the reserved budget
    pool_timeout=6.0,        # raise QueuePool.TimeoutError if no conn in 6s
    pool_pre_ping=True,      # validate the TCP session before handing it out
    pool_recycle=280,        # recycle under the firewall/NAT idle drop (~300s)
    connect_args={
        "connect_timeout": 5,                 # TCP connect ceiling
        "options": "-c statement_timeout=10000",  # server kills the query at 10s
    },
)

pool_pre_ping=True is the cheap insurance against the classic municipal failure mode: a load balancer or firewall silently closing an idle connection, so the first query after a quiet period blows up with a stale socket. statement_timeout pushes the deadline into the server itself, which means even a runaway zoning-overlay join is killed server-side rather than tying up a client thread. With this hierarchy, pool_timeout (6s) is the first guard to fire under saturation, well before the breaker’s per-call envelope.

Step 3: Wrap the Call in a Sliding-Window Breaker

Now combine the exception map and the timeout hierarchy inside the breaker. Prefer a sliding failure-rate window over a raw consecutive-failure counter: a fixed counter trips on three unlucky failures during an otherwise healthy minute, whereas a window with a minimum request volume ignores low-traffic noise and reacts to genuine degradation. Empirically baseline fail_max and reset_timeout against your backend’s real warm-up and maintenance behaviour rather than copying defaults.

import asyncio
import structlog

log = structlog.get_logger()

permitdb_breaker = pybreaker.CircuitBreaker(
    name="permitdb",
    fail_max=5,                  # trip after 5 infra faults inside the window
    reset_timeout=45,            # stay open 45s before a single half-open probe
    exclude=[lambda exc: not isinstance(exc, DB_INFRA_FAULTS)],
    listeners=[_DBFaultListener()],
)


@permitdb_breaker
async def fetch_permit_status(application_no: str) -> dict:
    """Authoritative read against the legacy permit DB. While the breaker is
    `open` this raises CircuitBreakerError immediately — no socket wait — which
    the router treats as 'go to the cached/degraded path'."""
    async with engine.connect() as conn:
        # Per-call envelope: a belt-and-braces ceiling under statement_timeout.
        result = await asyncio.wait_for(
            conn.execute(
                _PERMIT_STATUS_SQL, {"application_no": application_no}
            ),
            timeout=11.0,
        )
        row = result.mappings().first()
    if row is None:
        raise PermitNotFound(application_no)   # business error: NOT counted
    return dict(row)

The breaker now ages toward open only on the curated DB_INFRA_FAULTS, and a missing application raises PermitNotFound — excluded from the count — so a busy records-request day cannot trip the failover. When the window’s fault rate crosses fail_max, the next call raises CircuitBreakerError instantly, which is exactly the divert signal consumed by the parent fallback router.

Step 4: Probe Recovery and Expose the Routing Signal

Recovery must be automatic; no clerk should phone IT to “turn the database back on.” After reset_timeout, the breaker enters half_open and admits a single canary query. Success closes it and normal routing resumes; failure reopens it and restarts the cool-down. Expose the breaker’s own state as the routing predicate so the rest of the system never has to guess.

def primary_db_available() -> bool:
    """The breaker's state IS the routing signal: closed/half-open => try the
    primary; open => fail fast into degraded mode."""
    return permitdb_breaker.current_state != pybreaker.STATE_OPEN


async def get_status_or_fallback(application_no: str) -> dict:
    """Single entry point the clerk portal calls. Never blocks on a dead DB."""
    try:
        return await fetch_permit_status(application_no)
    except pybreaker.CircuitBreakerError:
        # Open breaker: serve the last cached snapshot rather than a spinner.
        log.info("permitdb.served_from_cache", application_no=application_no)
        return await cached_permit_status(application_no)

Returning a cached snapshot (or a structured 503 with a Retry-After header for writes) keeps the counter responsive during the outage. The same restraint — re-checking primary_db_available() before each authoritative write — governs how the parent router drains its queue once the backend recovers, and mirrors the throttled-recovery pattern used in implementing async batch processing for high-volume submissions.

Parameter and Flag Reference

Parameter Recommended Rationale for permit databases
fail_max 5 Faults inside the window before tripping. Lower it to 3 for systems under tight deadline SLAs so you fail over to the cached path sooner.
reset_timeout 45 s Cool-down before the half-open probe. Match it to the legacy backend’s typical warm-up or nightly maintenance-window length.
exclude infra-only predicate Count only DB_INFRA_FAULTS; never let constraint or not-found errors age the breaker.
pool_timeout 6.0 s Pool-checkout ceiling. First guard to fire under saturation, ahead of the per-call envelope.
statement_timeout 10000 ms Server-side kill switch so a runaway query dies in the database, not on a client thread.
asyncio.wait_for 11.0 s Client envelope, set just above statement_timeout so the server kills the query first and reports a clean error.
pool_pre_ping True Discards stale TCP sessions dropped by firewall/NAT idle timeouts before checkout.
pool_recycle 280 s Recycle connections under the typical 300 s idle-drop so you never hand out a dead socket.

Common Failure Patterns and Fixes

The breaker trips on business errors

If a “permit not found” or a unique-constraint violation is moving the breaker, your exclude predicate is wrong or the call site catches too broadly. Confirm the only exceptions reaching the breaker are DB_INFRA_FAULTS, and raise domain exceptions (PermitNotFound, DuplicateApplication) that the predicate filters out. A breaker that trips on a busy records-request day will hide a real outage in the noise.

Threads still hang despite a tripped breaker

This means a timeout is missing or out of order. The breaker cannot fail fast on a call that never returns. Verify the strict ordering statement_timeout (10s) < wait_for (11s) < pool_timeout (6s checkout) and that connect_timeout is set — without it, a connection refused against a downed host can block on the OS default of 75+ seconds, far longer than any application guard.

“Succeeds late” double execution

An overloaded legacy backend can commit a write and then time out before acknowledging it. The breaker counts a fault and the parent fallback router later replays the item — issuing the permit twice. Make every authoritative write idempotent with a stable key (a county application number or a canonical-payload hash) so the replay collides and no-ops rather than double-issuing.

Flapping during scheduled ETL windows

A nightly bulk import or zoning-overlay refresh saturates connections for a predictable few minutes and trips the breaker repeatedly. Raise reset_timeout to span the known window, or gate breaker evaluation behind a minimum request volume so a low-traffic maintenance period cannot trip on a handful of slow queries.

Stale connections after an idle period

The first morning query fails with a reset socket even though the database is healthy. This is a firewall/NAT idle drop. pool_pre_ping=True plus a pool_recycle set under the idle threshold eliminates it; do not raise fail_max to “absorb” these — that masks real faults instead of fixing the connection hygiene.

Audit and Logging Guidance

For compliance officers, the breaker’s behaviour during an outage is itself an auditable record that no application was silently dropped or double-issued. Log every state transition — not just failures — with enough context to reconstruct the incident timeline.

class AuditListener(pybreaker.CircuitBreakerListener):
    def state_change(self, cb, old, new) -> None:
        log.info(
            "permitdb.breaker_state",
            breaker=cb.name,
            old_state=getattr(old, "name", str(old)),
            new_state=getattr(new, "name", str(new)),
            fail_counter=cb.fail_counter,
            event_ts=datetime.now(timezone.utc).isoformat(),
        )

Pipe these events, with request correlation IDs, to the same immutable sink used for the rest of the permitting audit trail so retention matches your public-records mandate (commonly seven years). Record at minimum: the timestamp and direction of each closed → open → half_open → closed transition, the fault type that triggered it, the count of requests served from the cached path while open, and the half-open probe outcome. During a post-incident review this is the evidence that the failover engaged, recovered on its own, and committed every deferred record exactly once. The authorization context on those records still flows through the policy in implementing role-based access for clerk portals — an outage never widens who may read or write a permit.

Frequently Asked Questions

Should I use a consecutive-failure count or a sliding window?

Prefer a sliding failure-rate window with a minimum request volume. A raw consecutive counter trips on three unlucky failures inside an otherwise healthy minute, causing needless failovers, while a window ignores low-traffic noise and reacts to sustained degradation. For very low-volume internal endpoints a small consecutive count is acceptable, but pair it with a generous reset_timeout so a brief blip does not strand the counter.

How do I keep maintenance windows from tripping the breaker?

Two levers. Raise reset_timeout to span the known ETL or backup window so the breaker rides through it, and gate evaluation behind a minimum request volume so a handful of slow queries during a quiet maintenance period cannot trip the failover. If the window is fully predictable, you can also pause breaker evaluation on a schedule rather than letting it react to expected saturation.

Where does the cached fallback data come from?

From a local projection refreshed while the primary is healthy — the same cached validation contract and last-known status snapshots the parent fallback router maintains. The breaker only decides when to read from that cache; building and reconciling it is the router’s job. Validate cached reads against the JSON schema for building permits so degraded-mode responses stay as strict as live ones.

What timeout value should the per-call envelope use?

Set it just above the server-side statement_timeout — for example an 11-second asyncio.wait_for over a 10-second statement_timeout. That ordering lets the database kill the runaway query first and return a clean, attributable error, while the client envelope is only a backstop for the rare case where the server-side guard itself stalls.