PEFT Beyond QLoRA: DoRA, GaLore & LoftQ Compared

Q: What is parameter-efficient fine-tuning (PEFT)?

Parameter-efficient fine-tuning is the family of techniques that adapt a large pretrained language model to a task while updating only a small fraction of its parameters, leaving the rest frozen. Full fine-tuning updates every weight, which for large models means enormous memory: beyond the weights themselves, Adam-style optimizers keep multiple states per parameter, so a 70B model demands hundreds of gigabytes across GPUs. PEFT methods sidestep this by choosing a small, expressive set of things to train — low-rank adapter matrices added to attention layers (LoRA and its descendants), bias vectors only (BitFit), multiplicative orthogonal transforms (BOFT) — or by compressing the training process itself, as GaLore does by projecting gradients into a low-rank subspace so optimizer memory shrinks while every parameter still updates. The practical result is that fine-tuning a 7-70B model becomes feasible on one or a few GPUs instead of a cluster, at some cost in final accuracy that the newer methods (DoRA, LoftQ) have substantially narrowed. The umbrella term is also the name of Hugging Face's PEFT library, which implements most of these methods behind a common interface.

Q: What is DoRA and is it better than LoRA or QLoRA?

DoRA — Weight-Decomposed Low-Rank Adaptation — is a 2024 method that has become the default upgrade over vanilla LoRA and QLoRA in 2026 pipelines. Its insight: decompose each pretrained weight matrix into a magnitude vector and a direction matrix, then let training update the two independently, applying the low-rank adapter to the direction component. Analysis in the paper shows full fine-tuning naturally makes magnitude and direction updates with a distinctive negatively-correlated pattern that vanilla LoRA cannot reproduce — and DoRA can, which is why its learning behaviour tracks full fine-tuning much more closely. Empirically DoRA consistently matches or beats LoRA across commonsense reasoning, instruction tuning, and vision-language benchmarks at essentially the same memory and trainable-parameter budget, and it stacks with quantized bases the way QLoRA does. In Hugging Face PEFT it is enabled with a single flag on a LoRA configuration (use_dora=true), and Axolotl and LLaMA-Factory expose it the same way, so the switching cost is near zero. When accuracy is your binding constraint and you were about to run LoRA or QLoRA, DoRA is close to a free upgrade.

Q: What problem does LoftQ solve in QLoRA training?

QLoRA quantizes the frozen base model to 4-bit NormalFloat (NF4) to save memory, but quantization introduces error: the 4-bit weights are not the original weights. QLoRA then initializes its LoRA adapters at zero, blind to that error, so during training the adapters must spend part of their limited low-rank capacity correcting the quantization damage before — and while — learning your actual task. LoftQ (LoRA-Fine-Tuning-aware Quantization) fixes this at initialization: it alternates between quantizing the base weights and fitting the low-rank adapters so that the combination of quantized-base-plus-adapters approximates the original full-precision weights as closely as possible before training begins. The initial quantization gap shrinks dramatically, freeing the adapters to learn the task rather than repair the compression. The benefit grows as quantization gets more aggressive — at 4-bit it is a solid improvement, and at 2-bit it is frequently the difference between training that converges and training that does not, which is what makes 70B-class models trainable on consumer-grade GPUs. In practice LoftQ is an initialization option in the Hugging Face PEFT ecosystem: you enable it on a QLoRA-style run rather than adopting a separate framework.

Q: What is GaLore and how is it different from LoRA-style adapters?

GaLore — Gradient Low-Rank Projection — takes a fundamentally different route to memory efficiency. LoRA-family methods constrain what you train: they freeze the model and learn small adapter matrices, which caps memory but also caps expressiveness at the adapter's rank. GaLore instead constrains how gradients are stored: observing that gradient matrices during LLM training are naturally low-rank, it projects them into a compact subspace during the backward pass, keeps the optimizer states (the dominant memory consumer in Adam-style training) in that small subspace, and projects updates back to apply them to the full weights. Every parameter still updates — you get full-parameter fine-tuning semantics with no adapter bottleneck — while optimizer memory drops by up to 65.5% per the paper, which also demonstrates pre-training a 7B model on a single consumer 24GB GPU. Choose GaLore when you specifically want full-fine-tuning behaviour (tasks where adapter capacity is the suspected ceiling) on constrained hardware; choose DoRA/QLoRA when adapter-style training is sufficient, since adapters remain cheaper still and produce small, swappable artifacts rather than a fully modified model.

Q: What is rsLoRA and when does rank actually matter?

rsLoRA — rank-stabilized LoRA — corrects a scaling flaw in the original LoRA recipe that quietly sabotages high-rank training. LoRA scales its adapter contribution by alpha divided by rank (α/r); the rsLoRA paper shows this causes gradient magnitudes to collapse as rank grows, so experiments with rank 32 or 64 often learn no better than rank 8 — not because the capacity is useless but because the scaling starves it. rsLoRA changes the factor to alpha divided by the square root of rank (α/√r), which keeps training dynamics stable across ranks and lets added capacity actually contribute. When does rank matter? Low ranks (4-16) suffice for style adaptation, formatting, and narrow domain vocabulary — most classic fine-tunes. Higher ranks (32-64) earn their cost when the task requires the model to internalise genuinely new, complex behaviour: intricate structured outputs, multi-step domain reasoning, or large behavioural shifts — the cases where rank-8 runs plateau below the quality bar. The practical rule: start at rank 8-16 with standard scaling; if quality plateaus and you raise rank past ~16, switch on rsLoRA scaling (a flag in HF PEFT and the major trainers) or the added rank will likely be wasted.

Q: Which PEFT method should I use with Axolotl or LLaMA-Factory today?

Both frameworks expose the modern methods as configuration flags, so the choice is strategy, not tooling. The 2026 defaults: if accuracy is your binding constraint and the model fits comfortably at 4-bit or above, run DoRA — it is a one-flag upgrade over LoRA/QLoRA that closes most of the gap to full fine-tuning. If memory is the binding constraint and you are quantizing aggressively (tight 4-bit, or 2-bit to fit a 70B on consumer hardware), run QLoRA with LoftQ initialization so the adapters are not spent correcting quantization error. If your task pushes adapter rank past roughly 16, enable rsLoRA scaling so the extra rank actually helps. If you suspect the adapter itself is the ceiling — the task needs full-parameter behaviour — GaLore gives you that with optimizer memory cut by up to 65.5%. QA-LoRA is the pick when the deployed artifact must remain quantized exactly as trained. And regardless of method: the dataset dominates outcomes more than the method choice, every run needs a frozen held-out eval gate, and if the artifact will be re-quantized for serving (GGUF export for local runtimes, for instance), evaluate the post-quantization artifact too, not just the training-precision one.

Q: Is BitFit still useful in 2026?

In a narrow lane, yes — as the fastest, lightest adaptation lever available, not as a general fine-tuner. BitFit freezes every weight matrix and trains only the bias vectors, which typically amounts to less than 0.1% of a model's parameters. That makes runs extremely fast and cheap, the artifact tiny, and the risk of damaging pretrained capabilities minimal. The original work showed it competitive with full fine-tuning on BERT-scale classification benchmarks, and the same holds directionally today: for classification-adjacent tasks — routing, intent detection, scoring, simple extraction — BitFit can be a remarkably efficient first lever, and it doubles as a useful experimental baseline: if bias-only tuning gets you most of the way, the task is easy and nothing heavier is warranted. Where it clearly caps out is complex text generation: producing long, styled, structured output requires updating representations that biases alone cannot reach, and no amount of BitFit training closes that gap. The honest 2026 placement: a diagnostic tool and a lightweight option for simple discriminative tasks, with DoRA-class methods as the real workhorses for generation.

Q: Do these advanced PEFT methods change what I deploy to production?

Mostly no — and that is part of their appeal — but the deployment-relevant differences are worth knowing. LoRA-family methods (including DoRA, rsLoRA, LoftQ-initialized runs) produce small adapter artifacts that can either be merged into the base weights for a single deployable model or kept separate and hot-swapped at serving time; serving stacks such as vLLM support multi-LoRA serving, where one base model hosts many task adapters — an architecture that pairs naturally with per-task adapters over a shared base, and with swappable language or domain adapters on-device. Two caveats. First, quantization round-trips: if you train against a 4-bit base but serve a differently-quantized artifact (a GGUF export for local runtimes, say), the served model is not bit-identical to the trained one — evaluate after the final conversion, or use QA-LoRA, whose group-wise quantization-aware training keeps the model quantized consistently from training through serving. Second, GaLore differs structurally: it modifies the full weights rather than producing an adapter, so its output is a whole new model checkpoint — no adapter swapping, standard full-model deployment. None of these methods change your serving latency; they change what artifact you ship and how carefully you must eval it post-conversion.