Choosing Celery vs asyncio for Permit Batch Jobs

12 min read

Choosing between Celery and plain asyncio for overnight permit batch jobs is a focused architectural decision inside Implementing Async Batch Processing for High-Volume Submissions, part of the broader Automated Permit Ingestion and Parsing Workflows track: it decides whether your overnight import runs as durable distributed tasks or as in-process concurrency, and that choice shapes recoverability, observability, and how much operational weight lands on a small municipal IT team.

The decision is narrow but consequential. An overnight batch import — several thousand permit applications dropped by SFTP, portal bursts queued through the day, inter-agency reconciliations — must finish before the counter opens, survive a worker crash without double-charging a fee, and leave an audit trail a compliance officer can reconcile. Celery gives you distributed workers, durable retries, and scheduled dispatch at the cost of a broker, a result backend, and more moving parts to operate. Plain asyncio gives you thousands of cheap concurrent coroutines in one process with almost no infrastructure, at the cost of durability you must build yourself. Pick wrong and you either over-engineer a nightly job that one process could handle, or you run a business-critical import on in-memory state that vanishes when the box reboots mid-run. This guide frames the trade-offs, shows a minimal implementation of each, and gives a decision matrix you can apply to your own infrastructure.

Celery versus asyncio decision matrix for permit batch jobs A comparison matrix scores Celery and asyncio across five dimensions for overnight permit batch jobs. Durability and crash recovery favor Celery, whose broker persists tasks across restarts, while asyncio holds work in memory and loses it on crash unless checkpointed. Built-in retries and scheduling favor Celery through task retry policies and Celery beat, whereas asyncio requires hand-rolled backoff and an external cron. Observability favors Celery via Flower and task events, versus asyncio's plain logging. Operational burden favors asyncio, which needs only a Python process, while Celery adds a broker and result backend to run and monitor. A footer summarizes: choose Celery for durable distributed overnight jobs, asyncio for I/O-bound concurrency inside one process. DIMENSION Celery asyncio Durability / crash recovery survive a mid-run reboot broker persists tasks redelivered after restart in-memory work lost unless checkpointed Retries & scheduling backoff, nightly dispatch retry policy + beat built in hand-rolled + cron tenacity + systemd timer Observability per-task visibility Flower + task events state per task id structured logs you build the dashboard Operational burden what you must run broker + backend more to patch & monitor one Python process minimal infra Concurrency model how work fans out processes across hosts scale by adding workers coroutines in one loop cheap for I/O-bound work When each wins Celery → durable, distributed, scheduled overnight imports that must survive crashes and prove per-task state. asyncio → I/O-bound concurrency inside one process; often nested inside a Celery task for its lookups.

The Decision in One Frame

The two tools are not really competitors — they solve problems at different layers, and the honest answer for most municipal stacks is “both, at different levels.” Celery is a distributed task queue: it takes a unit of work, persists it in a broker, dispatches it to a worker process that may live on another host, and tracks the outcome. asyncio is a concurrency primitive: within a single process it interleaves thousands of I/O-bound operations on one event loop. Celery answers “run this reliably, later, somewhere, and tell me what happened”; asyncio answers “run these thousand lookups at once without a thread each.”

For an overnight permit import the questions that actually decide the architecture are: does the job need to survive a mid-run reboot, does it need scheduled dispatch and multi-host fan-out, and how much operational surface can the team carry. The sections below implement each approach at its simplest, then the matrix scores them.

Approach A: asyncio Alone

When the overnight job is I/O-bound — fetch a batch, enrich each record against county registries, validate, commit — and it comfortably fits one machine, plain asyncio is the lowest-friction choice. There is no broker to run and no result backend to patch. A bounded semaphore caps in-flight work so a surge cannot exhaust the database pool, exactly as the parent async-batch pipeline describes.

import asyncio

MAX_CONCURRENCY = 32  # cap in-flight records; keep <= DB pool size


async def run_overnight_batch(records: list[dict], process, pool) -> dict[str, int]:
    """Enrich + commit a batch entirely in-process, bounded by a semaphore."""
    sem = asyncio.Semaphore(MAX_CONCURRENCY)
    summary = {"done": 0, "failed": 0}

    async def handle(record: dict) -> None:
        async with sem:                       # backpressure: never exceed the cap
            async with pool.acquire() as conn:
                try:
                    await process(record, conn)
                    summary["done"] += 1
                except Exception:             # in-memory only — a crash loses this
                    summary["failed"] += 1

    await asyncio.gather(*(handle(r) for r in records))
    return summary

The catch is durability: if the process is OOM-killed or the VM reboots at 03:00, every in-flight and un-started record is gone, and the job has no memory of how far it got. You can bolt on durability — checkpoint committed keys to a table and resume from there — but at that point you are rebuilding a slice of what Celery gives you for free. asyncio shines when the work is short, idempotent, and cheap to re-run in full.

Approach B: Celery for Durable, Scheduled Dispatch

When the import must be scheduled, survive crashes, and fan out across hosts, Celery earns its infrastructure. Each batch becomes a task persisted in the broker; a worker on any host picks it up; a crash mid-task means the broker redelivers it rather than dropping it; and Celery beat triggers the run on a cron schedule without an external timer.

from celery import Celery

app = Celery("permits", broker="amqp://mq/permits", backend="redis://cache/1")


@app.task(
    bind=True,
    acks_late=True,                 # ack only after success -> crash requeues the task
    autoretry_for=(ConnectionError, TimeoutError),
    retry_backoff=True,             # exponential backoff on transient faults
    retry_backoff_max=300,
    retry_jitter=True,              # jitter avoids synchronized retry storms
    max_retries=5,
)
def import_batch(self, batch_id: str, payload_uri: str) -> dict:
    """Durable unit of work: fetch, process, commit one batch by id."""
    records = load_payload(payload_uri)        # stream raw payload from object store
    return process_batch(batch_id, records)    # commit is idempotent by batch_id


# Celery beat: schedule the overnight run without an external cron.
app.conf.beat_schedule = {
    "nightly-permit-import": {
        "task": "permits.import_batch",
        "schedule": crontab(hour=2, minute=0),  # 02:00 local, off-peak window
        "args": ("nightly", "s3://permits-raw/pending/"),
    },
}

acks_late=True is the durability lever: the task is acknowledged only after it completes, so a worker crash returns it to the queue for another worker. Pair it with an idempotent commit — keyed by batch or idempotency hash — so a redelivered task cannot double-write. Queue topology, worker lifecycle, and prefetch tuning for exactly this overnight window are covered in depth in Managing Celery Task Queues for Overnight Batch Imports.

Approach C: Celery Owns Dispatch, asyncio Owns Concurrency

The two combine cleanly, and this is what most production municipal stacks converge on. Celery provides durability, scheduling, retries, and observability at the task level; inside each task, asyncio drives the concurrent I/O so a single worker fetches hundreds of registry lookups at once instead of one at a time.

import asyncio


@app.task(bind=True, acks_late=True, max_retries=5)
def import_batch_async(self, batch_id: str, payload_uri: str) -> dict:
    """Celery task boundary (durable) wrapping an asyncio body (concurrent)."""
    # One event loop per task invocation; Celery handles the durable dispatch,
    # asyncio handles the fan-out of I/O-bound enrichment inside it.
    return asyncio.run(_import_async(batch_id, payload_uri))


async def _import_async(batch_id: str, payload_uri: str) -> dict:
    records = await load_payload_async(payload_uri)
    sem = asyncio.Semaphore(32)

    async def one(r: dict) -> None:
        async with sem:
            await enrich_and_commit(r)   # concurrent registry lookups per record

    await asyncio.gather(*(one(r) for r in records))
    return {"batch_id": batch_id, "count": len(records)}

This layering gives you Celery’s crash recovery and scheduling with asyncio’s cheap concurrency, and it is why the two rarely belong in an either/or framing once a job is both durable and I/O-bound.

Comparison Matrix

Dimension Celery asyncio Rule of thumb
Durability across restart Broker persists tasks; redelivered after a crash In-memory; lost unless you checkpoint Must survive a mid-run reboot → Celery
Retries Declarative autoretry_for + backoff + jitter Hand-rolled with tenacity Want retries for free → Celery
Scheduling Celery beat cron dispatch External cron / systemd timer Self-scheduling job → Celery
Distribution Workers across many hosts One process, one host Exceeds one machine → Celery
Observability Flower, task events, per-task state Your own structured logs Need per-task audit UI → Celery
Operational burden Broker + result backend to run/patch Just a Python process Minimal infra / small team → asyncio
Concurrency for I/O Coarse (process/prefork) Thousands of cheap coroutines High-fanout I/O in one task → asyncio
Startup complexity Broker, worker, beat, backend python job.py Prototype / one-off → asyncio

Parameter and Flag Reference

Setting Applies to Recommended Municipal-context notes
acks_late Celery True Ack after success so a crash requeues the batch instead of losing it.
max_retries Celery 5 Cap retries so a permanently bad batch lands in a dead-letter queue, not a loop.
retry_backoff / retry_jitter Celery True Exponential backoff + jitter prevents synchronized retry storms after an outage.
worker_prefetch_multiplier Celery 1 For long batch tasks, prefetch 1 so work spreads evenly and none is stranded on a crashed worker.
MAX_CONCURRENCY asyncio 32 Semaphore bound on in-flight records; keep at or below the DB connection pool size.
loop-per-task hybrid asyncio.run One event loop per Celery task; do not share a loop across prefork workers.
visibility_timeout Celery + Redis broker > job runtime Must exceed the longest batch runtime or the broker redelivers a task still running.

Common Failure Patterns and Fixes

asyncio job loses hours of work on a crash

An overnight asyncio job holds all progress in memory, so an OOM kill or reboot at 03:00 forces a full re-run — or worse, an ambiguous partial state. Make commits idempotent and checkpoint completed keys so a restart resumes, or move the durability concern to Celery where the broker persists undelivered work.

Celery task double-writes after a redelivery

acks_late guarantees at-least-once delivery, so a task that crashed after committing but before acking runs again. Guard the commit with an idempotency key and ON CONFLICT DO NOTHING so the second run is a no-op — never assume a Celery task runs exactly once.

# Idempotent commit makes at-least-once safe: a redelivered task is a no-op.
await conn.execute(
    "INSERT INTO permits (idempotency_key, payload) VALUES ($1, $2) "
    "ON CONFLICT (idempotency_key) DO NOTHING",
    key, payload,
)

Blocking calls stall the asyncio event loop

A synchronous DB driver or a CPU-bound parse inside an async def blocks the whole loop, collapsing concurrency to serial. Use async drivers (asyncpg, httpx) for I/O and offload CPU-bound work to a process pool via run_in_executor so one slow record never freezes the batch.

Celery broker becomes an unmonitored single point of failure

Adopting Celery adds a broker and result backend that a small team may forget to patch, back up, or alert on — and when the broker is down, nothing runs. Monitor broker health and queue depth as first-class signals, and size the deployment honestly: if the job fits one process, the broker may be complexity you do not need.

Redis visibility_timeout shorter than the job

With a Redis broker, a visibility_timeout below the batch runtime makes the broker redeliver a task that is still executing, producing duplicate concurrent runs. Set it comfortably above your longest observed batch runtime, and keep commits idempotent as a backstop.

Audit and Logging Guidance

Whichever model you choose, the compliance requirement is identical: every submission reaches a deterministic terminal state and that transition is recorded. With Celery, capture the task id, batch id, retry count, and final state per task — Flower and task events give per-task history for free, and those records feed the append-only audit trail compliance officers reconcile during public-records requests. With asyncio, you own that trail: emit a structured log line per record with its idempotency key, stage, and disposition, because there is no external task registry to fall back on. In both cases distinguish transient faults (retried) from permanent ones (quarantined) using the triage in Error Handling and Retry Logic for Ingestion Pipelines, and validate each record against the canonical shape in Designing JSON Schemas for Building Permits so a rejected batch is traceable rather than silently dropped. When an enrichment registry is unavailable mid-run, degrade through Building Fallback Routing for Legacy System Downtime instead of failing the whole job. Retain these run records for the state-mandated period; they are the evidence that every overnight import processed each permit exactly once or explicitly quarantined it.

Frequently Asked Questions

Is Celery just asyncio with extra steps?

No — they operate at different layers. asyncio interleaves concurrent I/O inside one process and one event loop; Celery persists units of work in a broker and dispatches them to worker processes that may run on other hosts, surviving restarts. asyncio gives you concurrency; Celery gives you durability, scheduling, and distribution. The common production answer is a Celery task whose body uses asyncio for its lookups.

When is plain asyncio the right call for an overnight batch?

When the job is I/O-bound, fits comfortably on one machine, and is cheap and safe to re-run in full — an idempotent enrichment pass, for example. You avoid running and patching a broker and result backend, and a bounded semaphore already protects the database. Reach for Celery once the job must be scheduled, must survive a mid-run crash without re-running everything, or must fan out across hosts.

How do I stop a Celery task from processing the same batch twice?

Assume at-least-once delivery and make the commit idempotent. Derive a deterministic key from the batch contents, enforce it with a unique database constraint, and write with ON CONFLICT DO NOTHING, so a redelivered task after a crash becomes a no-op. Also keep the broker visibility_timeout above your longest batch runtime so a still-running task is not redelivered underneath you.

Related in this area

Related Managing Celery Task Queues for Overnight Batch Imports