Rust vs C++ for On-Device Inference Engines

Q: Should I use Rust or C++ to build an on-device inference engine?

If you are writing an engine from scratch and have no hard dependency on a vendor NPU SDK, write it in Rust — the ownership model eliminates at compile time the two crash classes that dominate this workload, namely use-after-free on arena reuse and data races on a KV cache shared between prefill and decode threads. If you are shipping this quarter against a Qualcomm, MediaTek, or Apple accelerator SDK, you will be writing C++ regardless of preference, because those SDKs ship C++ headers and little else, and a new accelerator's day-one support is always a C++ header file. The framing that matters more than the language choice: where does the unsafe boundary sit? Most real Rust engines are a safe orchestration layer wrapped around C++ or intrinsics-heavy kernels, which means every safety claim is actually a claim about how small and well-audited that FFI seam is. A Rust engine whose hot path is four thousand lines of unsafe pointer arithmetic has bought a syntax change, not a safety property, and its crash reports look exactly like the C++ version's. The shape we recommend for greenfield work is a Rust skeleton owning the arenas, KV cache, scheduler, and session lifecycle — where the expensive, unreproducible bugs live — with hand-tuned kernels in C++ or intrinsics behind a deliberately tiny FFI seam, and unsafe line count tracked as a first-class engineering metric.

Q: Is Rust slower than C++ for AI inference?

No, and every large gap we have investigated turned out to be a measurement error. Both languages compile through LLVM, so a well-written Rust kernel and a well-written C++ kernel performing the same algorithm land within noise of each other on the same hardware. The two recurring sources of bogus benchmarks: comparing a Rust debug build against a C++ release build, which is a 10-to-30-times handicap because Rust debug builds disable optimization and keep bounds checks everywhere; and comparing two different algorithms or two different levels of hand-tuning, which measures the engineer, not the language. To benchmark honestly, build Rust with release profile, link-time optimization enabled, and panic set to abort, then compare against a C++ build with equivalent optimization flags, running the same kernel algorithm on the same device thermal state. Where a genuine performance difference can appear is at the margins of hand-tuning: decades of accumulated NEON, AVX, and Metal shader work in the C++ ecosystem means the fastest existing kernel for a given operation on a given chip is often already written in C++ or inline assembly, and matching it in Rust means either writing unsafe intrinsics yourself or calling the C++ kernel through FFI. That is an ecosystem advantage rather than a language-performance advantage, but it shows up in the same place on the same benchmark, and it is a legitimate reason to keep kernels in C++ while the engine skeleton lives in Rust.

Q: What kinds of bugs actually break inference engines in production?

Three crash classes dominate, and none of them live in the matrix multiply everyone spends their time optimizing. First, use-after-free on arena reuse: an inference engine allocates a few large arenas and reuses them for the process lifetime, and a tensor view that outlives the buffer it borrowed produces memory corruption that surfaces far from its cause. Second, data races on the KV cache: when a decode thread and a prefill thread touch the same cache block without synchronization, the resulting bug reproduces roughly once per thousand sessions, never on the developer's machine, and typically only on a specific device under thermal throttling — the most expensive bug shape in mobile engineering. Third, unbounded KV cache growth, which is not a memory-safety bug at all but kills the app just as dead, because mobile operating systems enforce memory ceilings (typically around 1.5 to 3 GB before the system starts reaping processes) by terminating you without ceremony. Rust's ownership model retires the first two at compile time: a tensor borrowing an arena cannot outlive it, and a cache shared across threads without synchronization fails to compile because it violates Send and Sync. Rust does nothing about the third, which is a design error in any language and must be fixed by capping the cache, evicting on an explicit policy, and testing at the memory ceiling on the worst device you support rather than on a development machine with a comfortable amount of RAM.

Q: Can Rust actually use CUDA, Metal, and NEON intrinsics?

Yes, but through unsafe blocks or through FFI to C++ kernels, and that distinction is the whole architectural conversation. Rust exposes SIMD intrinsics (including NEON on ARM) in its core library behind unsafe, because the compiler cannot verify alignment and lane-width invariants; calling them is entirely normal practice for high-performance Rust and it works well, provided the unsafe surface stays small and reviewed. GPU compute is a different story: CUDA and Metal are C++-centric toolchains, and while Rust bindings and compute-shader paths exist, the mature, vendor-blessed, day-one-supported path for a new accelerator is a C++ header. This is why the pragmatic architecture for a Rust inference engine is a safe skeleton wrapped around kernels that may be Rust-with-unsafe-intrinsics, or may be C++ called across FFI, chosen per kernel based on what already exists and what performs. The engineering discipline that keeps this honest: treat every unsafe block and every FFI call as a security-sensitive component, review it the way you would review cryptographic code, and track total unsafe line count as a release-gating metric. A team that does this can defend the claim that their engine is memory-safe in the parts where memory-safety bugs actually occur — the arena lifecycle, the cache, the scheduler — while acknowledging that the hot loop is exactly as trusted as it is audited.

Q: Why do llama.cpp, ONNX Runtime, and ExecuTorch all use C++?

Ecosystem gravity and hardware access, accumulated over a decade. Vendor SDKs for NPUs, DSPs, and GPUs ship C++ headers, so an inference runtime that wants day-one support for a new accelerator writes C++ or it waits. Decades of hand-tuned kernels in NEON, AVX, CUDA, and Metal already exist in C++ and are not being rewritten. And the deployment targets — Android NDK, iOS, embedded Linux, WASM — all have first-class C++ toolchains with well-understood cross-compilation stories. llama.cpp and ggml became the gravitational center of on-device LLM inference by being a dependency-free C++ core that runs everywhere; ONNX Runtime and ExecuTorch are the production paths for taking PyTorch models to phones, both C++ cores exposed through bindings to other languages. Rust alternatives are real: candle from Hugging Face and burn are actively developed, capable, and pleasant to work in. Choosing them means owning more of the stack yourself — you will write integrations that already exist in the C++ world, and you will be behind on support for whatever accelerator ships next. For a team whose product is the model experience rather than the engine, that is usually the wrong trade. For a team whose product is the engine, and whose differentiation is reliability under concurrency on a device fleet they do not control, the Rust investment can pay for itself in bugs that never happen — provided they are honest that the kernels underneath may still be C++.

Q: Should I rewrite my working C++ inference engine in Rust?

Almost certainly not, and this is the single most common expensive mistake in this space. A rewrite of a 60,000-line engine reintroduces every bug you already found and fixed over years of device-fleet exposure, in a language where your team is junior, while shipping zero customer-visible value for two or more quarters. The bugs a rewrite would prevent are hypothetical; the bugs it will introduce are certain. Worse, the safety benefit is often illusory in a rewrite because a team under schedule pressure reaches for unsafe to reproduce familiar C++ idioms, arriving at an engine that is neither idiomatic Rust nor battle-tested C++. The defensible alternatives, in order. Wrap it: put a safe Rust layer around the existing C++ core so new code gets ownership guarantees while the proven kernels stay untouched. Extend it: write new subsystems (a new scheduler, a new cache policy, a new device backend) in Rust behind a narrow FFI seam, which lets the team build Rust competence on code that is not yet load-bearing. Or harden it: address sanitizer and thread sanitizer on every pull request are the closest thing C++ has to Rust's compile-time guarantees, and a C++ codebase without them has chosen neither safety nor speed. Reserve a full Rust rewrite for the case where the engine is being rebuilt anyway for unrelated reasons, such as a fundamental change in hardware targets or execution model, and even then keep the kernels.

Q: How do I bound the KV cache to avoid out-of-memory kills on mobile?

By treating the memory ceiling as a hard architectural constraint rather than an operational surprise, because no language prevents this failure. Mobile operating systems terminate processes that exceed their allowance — commonly somewhere around 1.5 to 3 GB depending on device class and what else is running — and they do it without a catchable exception, so an unbounded KV cache that grows linearly with generated tokens will eventually kill the app on a long conversation on a cheap phone. The controls, layered. Cap the cache at a token count derived from the memory budget rather than from a comfortable number on your development device, and enforce that cap in the session lifecycle rather than hoping generation stops first. Choose an eviction policy explicitly: sliding window over recent tokens, attention-sink retention keeping the first few tokens plus a recent window, or summarize-and-drop for conversational applications. Quantize the cache itself, since KV entries tolerate lower precision better than weights do and this is frequently the single largest memory win available. Then test at the ceiling: run the longest realistic session on the worst device you claim to support, with a memory-pressure simulator running, and confirm the eviction path executes rather than the OOM killer. Track peak resident set size per release as a gating metric. This is design work that must happen before kernel optimization, because an engine that is fast and dead is worth less than one that is adequate and alive.

Q: When is a custom inference engine actually worth building?

Far less often than teams believe, and the honest first step is to disprove the need. Most engagements that open with "Rust or C++ for our engine" resolve within an hour once the latency, memory, and throughput targets are written down and measured against an off-the-shelf runtime — llama.cpp, ONNX Runtime, ExecuTorch, or a platform runtime like Core ML — because the off-the-shelf option hits the targets more often than the team expects, and the custom engine turns out to have been a preference dressed as a requirement. The legitimate reasons to build: a hardware target no existing runtime supports well (a novel NPU, an unusual embedded platform); an execution model that existing runtimes cannot express (custom speculative decoding, unusual batching or multi-model scheduling, tight integration with a sensor pipeline under a hard frame budget); a memory or binary-size ceiling below what a general-purpose runtime can achieve; or engine performance being the actual product rather than a means to it. The illegitimate reasons, which are more common: benchmark pride, distrust of dependencies, a desire to write systems code, and the belief that a general runtime's overhead is significant before anyone has measured it. Before writing a line of Rust or C++, run the model through an existing runtime, capture p99 latency and peak memory on the worst supported device, and compare against the requirement. If it passes, the language debate is moot and the engineering hours belong somewhere with a customer attached.