Batch LLM Inference: Throughput & Cost Architecture

Q: What is batch LLM inference and how is it different from online inference?

Batch LLM inference is processing a large, finite set of inputs through a model as an offline job optimized for throughput and cost, rather than serving one request at a time at low latency for a waiting user. The two are architecturally opposite despite both calling a model. Online inference optimizes tail latency: it serves a user staring at a spinner, runs low concurrency per request, autoscales for spiky traffic, and treats every millisecond as visible cost. Batch inference optimizes throughput per dollar: nobody is waiting on any individual result, so it maximizes concurrency to saturate GPUs or provider queues, and its only deadline is a completion window measured in hours. The distinction matters because using the wrong one is expensive in both directions — running batch work through an online endpoint pays full price and fights per-minute rate limits for latency nobody needs, while running latency-sensitive traffic through a batch pipeline makes users wait hours. The decision rule is a single question: is a human blocked on this specific result right now? If yes, online; if no — a classification backfill, a re-embedding run, document enrichment, summary generation over a catalogue, synthetic eval-set creation — it is batch-shaped work, and the batch architecture is roughly half the cost and far more reliable at scale.

Q: How much cheaper is the OpenAI or Anthropic batch API?

Both price batch processing at roughly 50 percent of the synchronous per-token rate, in exchange for a completion target of up to 24 hours rather than immediate response. OpenAI's Batch API and Anthropic's Message Batches follow the same shape: you submit a file of many requests, the provider processes them asynchronously within the SLA window, and you retrieve results when the batch completes, at about half price. The savings compound with volume in a way that changes project viability — on a job of 10 million requests, the difference between full and batch pricing is frequently the line between a project that ships and one cancelled at the budget review, not a rounding adjustment. Two benefits beyond the headline discount matter as much in practice. First, batch tiers sidestep the per-minute rate limits that wreck a naive backfill: instead of hitting a limit at hour two and retrying blindly (which multiplies the load that caused the limit and inflates cost further), you hand the provider the whole workload and let it schedule. Second, the asynchronous model removes the operational burden of pacing requests yourself. The tradeoff is genuinely just latency: if you can tolerate results arriving within hours rather than seconds — which is the definition of offline work — the batch tier is close to free money, and not using it for finite jobs is one of the most common avoidable line items in an LLM budget.

Q: How do you process millions of documents through an LLM reliably?

With a durable, resumable pipeline built on the assumption that the run will be interrupted — because over millions of items it will be, whether by a spot-instance reclaim, a transient provider error, or an out-of-memory event. The architecture has six load-bearing parts. Sharding splits the corpus into thousands of independent, resumable units, turning one impossible monolithic job into many tractable ones. A durable work queue (SQS, a task queue, or a data framework) distributes shards to workers and redelivers any that fail. Idempotent writes keyed by document ID ensure that a redelivered shard overwrites rather than duplicates its results, which is essential because the queue guarantees at-least-once, not exactly-once, delivery. Checkpointing the processed offset after each shard converts a mid-run crash into a resume from the last checkpoint rather than a restart from zero — the single most important property, since restarting a 10-million-item job that died at 6 million means paying twice. A dead-letter queue quarantines poison inputs (a malformed PDF, a document that blows the context window) so that fifty bad items do not stall ten million good ones. Finally, end-of-run reconciliation checks that inputs-in equals results-out plus quarantined-failures, catching silent data holes before the downstream job inherits them as truth. This is the same durability discipline any large distributed job needs, adapted for token-based work; skipping any one part surfaces as either duplicated cost, silent data loss, or a job that can never finish.

Q: Should I use a provider batch API or self-host batch inference?

Default to the provider batch API and self-host only when volume, model choice, or data residency forces it. Provider batch APIs (OpenAI, Anthropic, AWS Bedrock batch inference) are the right first choice for almost everyone: you upload requests, poll for completion within the SLA window, download results, and pay about half the online rate with zero infrastructure to build or operate — the fastest path to correct, cheap batch processing. Self-hosting on your own GPUs wins in three situations. First, sustained high volume, where the per-token economics of owned or spot GPUs eventually beat even the discounted managed rate. Second, open-weight models the providers do not host, where you control the serving stack. Third, air-gapped or data-residency constraints that legally forbid sending the data to an external provider at all. When you do self-host, two levers make it economical: continuous batching (vLLM and similar engines admit new sequences into the running batch the instant others finish, keeping the GPU saturated and lifting throughput several-fold over naive fixed batching), and spot or preemptible GPUs (60 to 90 percent cheaper, and ideal here precisely because a properly checkpointed batch job is interruption-tolerant by construction — the reliability property that is dangerous to rely on for online serving is free for batch). A framework like Ray Data handles sharding, GPU pooling, and fault tolerance so you are not hand-building the pipeline. The most expensive mistake is not picking wrong between these two options; it is not realising batch was an option and running the whole corpus through the online endpoint at full price.

Q: What is continuous batching and why does it matter for throughput?

Continuous batching is a serving technique that keeps a GPU maximally utilized by admitting new request sequences into the in-flight batch the moment earlier ones finish, rather than forming a fixed batch, waiting for every sequence in it to complete, and only then starting the next batch. It matters because LLM generation is variable-length: in a fixed batch, a few long-generating sequences hold the whole batch hostage while short ones sit finished but uncollected, leaving the GPU partly idle and throughput far below the hardware's ceiling. Continuous batching (popularized by vLLM and now standard in high-throughput engines) eliminates that idle time by treating the batch as a rolling admission pool — as soon as a sequence emits its final token and exits, a waiting request takes its slot, so the GPU stays saturated with useful work. The practical effect is a several-fold throughput increase over naive batching on the same hardware, which for a self-hosted batch pipeline translates directly into either fewer GPU-hours for the same job or the same job finishing far sooner. It is the single most important efficiency lever when self-hosting batch inference, and it pairs naturally with spot GPUs: high per-GPU throughput plus deeply discounted interruption-tolerant instances is what makes owned-infrastructure batch processing competitive with, and at high volume cheaper than, the discounted managed batch tiers. Continuous batching is a serving-layer concern, so it applies whether you drive it directly through vLLM or through a higher-level framework that uses vLLM underneath.

Q: When should I re-embed my entire vector database and how?

Re-embedding the whole corpus becomes necessary whenever the embedding model changes — an upgrade to a better model, a switch of provider, or a move to a domain-tuned model — because vectors produced by different models are not comparable, so a half-migrated index returns nonsense for cross-model similarity. It is the canonical batch job: large, finite, latency-insensitive, and embarrassingly parallel, which makes running it through an online embedding endpoint at full price both wasteful and fragile. The right approach is the durable batch pipeline applied to embeddings: shard the corpus, push shards through a queue, embed each batch (via a provider batch API at half price, or a self-hosted embedding model on spot GPUs with continuous batching for large corpora), write vectors idempotently keyed by document ID, and checkpoint progress so an interrupted run resumes rather than restarts. The operational subtlety unique to re-embedding is the cutover: because old and new vectors are incompatible, you build the new index alongside the old one (write to a new namespace or collection), reconcile that every document has a fresh vector, and switch reads over atomically once the backfill is verified complete — never mix embedding versions in a live index. This is the embedding-lifecycle case of batch inference, and treating it as batch rather than online routinely cuts the cost of a large re-embedding run by half or more while making it safely resumable, which matters because these runs are big enough that an unrecoverable failure at eighty percent is a genuinely expensive event.

Q: How do spot or preemptible GPUs cut batch inference costs?

Spot and preemptible GPU instances are spare cloud capacity sold at a steep discount — commonly 60 to 90 percent below on-demand price — with the catch that the provider can reclaim them on short notice. That reclaim risk makes them dangerous for online serving, where an eviction mid-request drops a user's response, but it is a non-issue for batch inference, because a correctly built batch pipeline is already interruption-tolerant by design: work is sharded, progress is checkpointed after each shard, writes are idempotent, and a durable queue redelivers anything a reclaimed worker left unfinished. In other words, the checkpointing you built to survive crashes is exactly what lets you run on the cheapest possible hardware — the reliability engineering pays for itself twice. The practical setup: run batch workers on spot GPU pools, let the queue and checkpoint store absorb evictions transparently (a reclaimed shard simply gets redelivered and resumes from its last checkpoint), and keep enough shard granularity that losing one worker costs minutes of redone work, not hours. Combined with continuous batching to saturate each GPU while you have it, spot instances are what make self-hosted batch inference economically competitive with managed batch tiers at high volume. The failure mode to avoid is the inverse: paying full on-demand price for GPUs to run a job that is inherently resumable, which wastes the entire discount that batch workloads are uniquely positioned to capture.

Q: What kinds of AI work should run as batch jobs instead of online?

Any job where the inputs are a large finite set and no human is blocked on an individual result. The recurring examples: classification or labelling backfills (categorize every historical support ticket, moderate an existing content archive); field extraction over document stores (pull structured data from a million PDFs, invoices, or contracts); re-embedding a corpus after an embedding-model change; enrichment passes (summarize every product, translate a catalogue, tag a media library); synthetic data and eval-set generation (produce graded test cases at scale); and periodic scheduled analytics (nightly sentiment over the day's reviews). The common signature is that completion in hours is acceptable and the work is embarrassingly parallel. The counterexamples that must stay online are equally clear: anything a user is actively waiting on — chat, live agents, interactive search, real-time recommendations, anything in a request path with a person at the other end. The test to apply to any LLM workload is the blocked-human question: if a person is staring at a spinner for this specific result, it is online and latency is the priority; if the result feeds a downstream job, a database, or a report that nobody is refreshing in real time, it is batch and throughput-per-dollar is the priority. Many systems have both shapes and should run two distinct inference paths accordingly — an online endpoint for the live features and a batch pipeline for the finite jobs — because forcing the offline work through the online path is the single most common source of avoidable LLM spend.