Background Jobs & Task Queues: BullMQ, Celery, SQS

Q: When should work go into a background job instead of the request cycle?

The working threshold: anything that takes longer than roughly 200 milliseconds and whose result the user does not need synchronously to proceed. Classic candidates are sending email and notifications, image and video processing, PDF and report generation, third-party API calls with unpredictable latency, data imports and exports, and — increasingly dominant in 2026 — AI work like transcription, embedding generation, and LLM enrichment. The reasons go beyond perceived speed. Work done inline holds a connection and a server thread hostage, hits load-balancer timeouts on slow days, and couples your user-facing latency to your slowest dependency's worst day. Worse, when an inline operation times out, the client retries — and now a payment posts twice or an email sends three times, because the first attempt actually succeeded after the timeout fired. The queue pattern severs all of this: the request validates input, enqueues a job, and returns immediately (HTTP 202 with a status handle the client can poll or subscribe to), while workers process at their own pace with proper retry semantics. The request/response cycle is for reading data and acknowledging intent; the queue is for doing the work.

Q: Should I use BullMQ, Celery, Sidekiq, or SQS?

Default to your stack's native tool, because ecosystem fit beats feature checklists: BullMQ for Node/TypeScript (Redis-backed, with delayed jobs, per-queue rate limiting, and parent/child flows), Celery for Python (the deepest ecosystem, canvas workflows, beat scheduling — at the cost of configuration sprawl), Sidekiq for Ruby (exceptional throughput per worker and a very mature operational story, with the caveat that advanced features live in paid tiers). Choose SQS — usually paired with Lambda consumers — when the trade you want is paying per message (around $0.40 per million after the free tier) instead of operating Redis: you get effectively infinite scale, a built-in dead-letter queue, and zero queue infrastructure to patch, accepting at-least-once delivery and Lambda's 15-minute execution ceiling. The under-considered fourth option: Postgres-backed queues (Graphile Worker, pg-boss) when volume is modest — thousands of jobs per second or less — because enqueueing in the same database transaction that writes your business data eliminates the write-vs-enqueue consistency gap entirely, which is the outbox pattern built into the queue itself. The wrong move is adopting Kafka-scale streaming infrastructure for what is actually a to-do list; a task queue and an event log are different tools.

Q: Why must background jobs be idempotent?

Because every production queue you will realistically use — SQS, BullMQ, Celery, Sidekiq — guarantees at-least-once delivery, which means duplicate execution is a designed-in behaviour, not a rare fault. The duplicates arrive through mundane mechanics: a worker processes a job successfully but crashes before acknowledging it, so the queue re-delivers; a visibility timeout expires while a slow job is still running, so a second worker picks it up in parallel; a network partition makes an acknowledgment vanish. If the job's side effect is "charge the card" or "send the email," running twice is an incident. Idempotency makes duplicate execution harmless by keying every side effect on a stable job identifier: before executing, check whether this job ID already completed (a uniqueness constraint in the database is the sturdiest check); make external calls with idempotency keys where the provider supports them (payment APIs universally do); and design state changes as absolute assignments ("set status to processed") rather than relative mutations ("increment counter") wherever possible. Exactly-once *processing* is achievable this way even though exactly-once *delivery* is not — the queue may hand you the job twice, but the second execution becomes a no-op. This is the single discipline that separates queues that survive production from queues that generate refund tickets.

Q: What is a dead-letter queue and how should I use it?

A dead-letter queue (DLQ) is where jobs go after exhausting their retry budget — the alternative being that they silently vanish, which turns transient failures into permanent, undetected data loss. The mechanics: configure a maximum attempt count (three to five is typical) with exponential backoff between attempts; when the final attempt fails, the queue moves the job — payload, error history, timestamps — to a separate holding queue instead of deleting it. SQS has this as a first-class redrive policy; BullMQ keeps failed jobs inspectable with their stack traces; Celery and Sidekiq support equivalent dead-job stores. The part teams skip is the operational half: a DLQ nobody watches is just a slower way to lose data. Minimum viable operation: an alert when DLQ depth exceeds zero (or a small threshold for noisy job classes), a dashboard showing what is accumulating, a documented replay path so that after you fix the underlying bug you can re-drive the dead jobs back through processing, and a periodic review so the DLQ does not become a landfill. The replay property is the quiet superpower: a vendor outage that failed 2,000 jobs overnight becomes a ten-minute morning recovery — re-drive the DLQ — instead of a data-reconstruction project.

Q: How do I handle retries without making things worse?

Bad retry policy converts one failure into a self-inflicted outage: a dependency struggles, a thousand queued jobs each retry immediately, and your retry storm finishes off the dependency — then your own database, as every worker slams it simultaneously. Three mechanisms make retries safe. Exponential backoff spaces attempts increasingly far apart — one second, then four, sixteen, sixty-four — giving the failing dependency room to recover instead of a sustained beating. Jitter adds randomness to each delay so that jobs which failed together do not retry together; without it, backoff just synchronizes the thundering herd into periodic waves. A capped attempt budget (typically three to five) with a dead-letter queue at the end acknowledges that some failures are not transient — a malformed payload will fail identically on attempt fifty — and routes them to human attention instead of infinite robot persistence. Two refinements worth adopting: distinguish retryable errors (timeouts, 429s, 503s) from permanent ones (validation failures, 404s) and send permanent failures straight to the DLQ without burning retries; and respect Retry-After headers when a rate-limited API tells you exactly when to come back. The combination — backoff, jitter, caps, error classification — is what lets a queue absorb a bad hour gracefully rather than amplifying it.

Q: How should long-running AI jobs like transcription or embedding be structured?

Chunk them. The monolithic version — one job that transcribes 500 audio files or embeds 100,000 documents — has a fatal profile: it holds an expensive worker (often GPU-attached) for hours, cannot be parallelized, and when it fails at 95 percent, the retry re-does everything, including the API spend. The production shape is a parent job that splits the workload into many small child jobs — one file, one batch of a few hundred documents — pushed through a concurrency-capped queue, with a completion step that fires when all children finish (BullMQ flows and Celery chords model this parent/child pattern natively). The wins compound: parallelism becomes a dial (twenty workers chew through the backlog twenty times faster), failure isolation means a bad chunk retries alone while its 9,997 siblings stand, progress is observable (children done over children total), and cost control gets granular — per-chunk budgets prevent a runaway enrichment loop from spending a thousand dollars of tokens overnight. Concurrency should be budgeted to the binding dependency: the embedding API's rate limit or the GPU pool size, not worker CPU. The boundary to know: when a single logical task itself needs many sequential steps with mid-task resumption after crashes — a long agent workflow, not a batch — a task queue stops being the right tool and durable-execution engines (Temporal-class) take over.

Q: What queue metrics should I monitor?

Three numbers predict essentially every queue incident hours before users feel it. Queue depth — how many jobs are waiting — with alerting on sustained growth rather than absolute spikes: depth that climbs for twenty minutes means arrival rate exceeds processing rate, and the only questions are why and when you run out of runway. Oldest-message age — how long the head of the queue has waited — which catches the failure depth alone misses: a poison job stuck in a retry loop at the front while depth looks modest, or a critical queue whose SLA is measured in seconds quietly aging into minutes. Processing rate and failure rate per job class — throughput tells you whether scaling workers is working; a failure-rate step-change is usually the first visible symptom of a deploy gone wrong or a dependency degrading. Around those three: DLQ depth greater than zero should page or at least ticket, worker utilization distinguishes "need more workers" from "workers are starving on a dependency," and per-class job duration percentiles catch the slow bleed of a dependency getting gradually slower. Wire these into the same observability stack as your services — queues are production infrastructure, and the dashboard-less queue is the one that becomes a 3 a.m. mystery.

Q: Can I just use Postgres as a job queue instead of Redis or SQS?

Yes, and below a few thousand jobs per second it is often the *better* choice, not a compromise. Tools like Graphile Worker and pg-boss implement real queues — retries, scheduling, backoff — on Postgres using SELECT ... FOR UPDATE SKIP LOCKED, which lets many workers claim jobs concurrently without contention. The killer feature is transactional enqueue: because the queue lives in the same database as your business data, you can write the order AND enqueue its confirmation email in one transaction — they commit or roll back together, eliminating the crash window where data exists but the job does not (or vice versa) that Redis- and SQS-based designs must patch with the outbox pattern. You also drop an entire infrastructure component: no Redis to operate, persist, and monitor. The honest limits: throughput ceilings in the low thousands of jobs per second (versus tens of thousands for Redis-based queues), queue polling adds database load that you must budget alongside application queries, and very high-volume or multi-service queue topologies outgrow it. The pragmatic rule: if you run Postgres already and your job volume is a stream rather than a flood, start there and migrate the hot job classes to a dedicated queue only when metrics — not architecture fashion — say you must.

Satyam Kumar

Zurück zum Blog

ai-architecture

Background Jobs and Task Queue Architecture: BullMQ, Celery, and SQS

By Satyam KumarJuly 6, 20267 min read

background jobs task queue bullmq celery sidekiq amazon sqs job queue architecture dead-letter queue exponential backoff idempotent jobs worker pools redis queue async processing batch processing ai architecture patterns 2026

Background Jobs and Task Queue Architecture: BullMQ, Celery, and SQS

Frequently Asked Questions

Artikel teilen

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

KI & Cloud Architekt. Ich helfe Teams, Systeme zu bauen, die auf Millionen skalieren.

LinkedIn GitHub

Background Jobs and Task Queue Architecture: BullMQ, Celery, and SQS

Frequently Asked Questions

Artikel teilen

Comments

Leave a comment