LLM Fine-Tuning 2026: LoRA vs QLoRA vs DoRA vs Full FT Compared

Q: What is LoRA fine-tuning and how is it different from full fine-tuning?

LoRA (Low-Rank Adaptation) freezes the base model weights and trains small low-rank decomposition matrices that approximate the weight updates, instead of updating all the model parameters. Where full fine-tuning of an 8B model trains 8 billion parameters and needs roughly 100 GB of GPU memory across multiple GPUs, LoRA trains only 0.1 to 1 percent of that parameter count and runs on a single 24 GB GPU. The key insight is that the parameter updates needed to adapt a pre-trained model to a new task have low rank — they can be approximated well by two small matrices A and B such that the effective update is W + BA. LoRA quality matches full fine-tuning on most adaptation tasks (style, instruction tuning, structured output, domain specialisation) where the fine-tuning is making the model better at applying capabilities it already has rather than teaching it fundamentally new ones. Full fine-tuning remains appropriate for genuinely new capabilities — new languages, new modalities, deep domain shifts.

Q: What is QLoRA and when should I use it instead of LoRA?

QLoRA is LoRA applied to a quantised base model — the base model is loaded in 4-bit precision (NF4 quantisation with double quantisation) while the LoRA adapters are trained in higher precision (typically BF16). This combination drops memory requirements another 4 to 8 times beyond LoRA. A 7B model can be QLoRA-fine-tuned on a 12 to 16 GB consumer GPU; a 70B model fits on a single A100 80GB or H100. Quality is typically 1 to 3 percent below full LoRA on most benchmarks, which is acceptable given the dramatic reduction in cost and barrier to entry. Use QLoRA when hardware budget is constrained (single consumer GPU), when fine-tuning large models (30B-plus) without multi-GPU clusters, or when running many parallel fine-tuning experiments where memory headroom matters. QLoRA is the most commonly deployed fine-tuning approach in 2026 simply because most teams cannot or will not provision multi-GPU clusters for fine-tuning.

Q: What is DoRA and is it better than LoRA?

DoRA (Weight-Decomposed Low-Rank Adaptation) is the 2024-released refinement of LoRA that decomposes the weight update into separate magnitude and direction components, each trained independently. The result is consistently better quality than LoRA at the same parameter budget — typically 1 to 3 percent better on instruction-following benchmarks and noticeably better on tasks requiring fine behavioural control. The compute and memory overhead versus LoRA is modest, roughly 10 to 20 percent more parameters and training time. For new fine-tuning projects in 2026 where quality matters and existing tooling supports DoRA (Hugging Face PEFT does), DoRA is becoming the default starting point. For legacy projects already using LoRA, the migration is rarely worth it — LoRA remains entirely production-viable. DoRA is supported in the standard Hugging Face PEFT library with minimal config changes.

Q: How much data do I need to fine-tune an LLM?

The conventional wisdom of needing tens of thousands of examples has shifted. For most adaptation tasks, 500 to 5,000 high-quality examples are sufficient, and adding more low-quality examples actively degrades the result. Quality means examples that genuinely represent the target behaviour, are diverse across the input space, and have outputs that are exactly what you want the model to produce — not approximately, not slightly different in formatting, exactly. The leading teams in 2026 spend more on dataset curation than on compute. The most powerful dataset construction pattern is distillation from a frontier model: use GPT-5, Claude Opus 4, or Gemini 2.5 Pro to generate high-quality outputs across your input distribution, then fine-tune your 7 to 14B open-weights model on this dataset. The production result is frontier-model-quality outputs at small-language-model economics. Always hold out 20 percent of the dataset as an evaluation set you never train on.

Q: What hardware do I need to fine-tune an open-weights LLM?

For QLoRA on 7 to 13B models, a single consumer GPU with 12 to 24 GB VRAM (RTX 4090, RTX 6000 Ada, A4500) is sufficient. For LoRA on 7 to 14B models in BF16, or QLoRA on 30 to 70B models, a single data centre GPU with 40 to 80 GB (A100 40GB, A100 80GB, L40S, H100, H200) is the sweet spot. Cloud cost runs $1 to $12 per hour depending on the GPU; a typical 4 to 12 hour fine-tuning run costs in the tens to low hundreds of dollars. Full fine-tuning of 7B-plus models, or any large-dataset multi-day run, typically requires multi-GPU configurations using DeepSpeed (ZeRO Stage 2 or 3) or FSDP, with end-to-end costs in the low single-digit thousands of dollars. The democratising change in 2026 is that meaningful fine-tuning experiments — the kind that produce production-deployable models — can be run for under $50 in cloud compute on a single GPU instance.

Q: What hyperparameters matter most for LoRA fine-tuning?

Five hyperparameters matter most. First, learning rate — for LoRA and QLoRA, 1e-4 to 5e-4 with 2e-4 a strong default; for full fine-tuning, much lower (1e-5 to 5e-5). Wrong learning rate is the most common cause of fine-tuning failure. Second, LoRA rank (r) — typical values 8, 16, 32, with 16 a strong default; higher rank captures more complex adaptations at the cost of memory and overfitting risk. Third, LoRA alpha — convention is alpha = 2 * r (e.g., r=16, alpha=32). Fourth, target modules — apply LoRA to all linear layers (Q, K, V, O attention projections plus gate, up, down MLP projections) rather than just Q and V; the quality benefit is consistent for small extra cost. Fifth, number of epochs — most fine-tuning runs should be 1 to 3 epochs; more typically causes overfitting. Always run small learning-rate sweeps before committing to a long training run, and watch held-out evaluation loss as the signal to stop.

Q: How do I serve a fine-tuned LLM in production?

Three deployment patterns dominate production. First, merged weights — merge the LoRA adapter into the base model to produce a single set of weights that runs in any standard inference stack (vLLM, TGI, SGLang, llama.cpp). Simplest pattern; right choice when only one fine-tuned variant is needed. Second, multi-LoRA serving — the base model is loaded once on the GPU and shared across requests; each request specifies which LoRA adapter to apply. vLLM, TGI, and SGLang all support this with minimal latency overhead. Right pattern for SaaS products fine-tuning per customer or per workload — one base model serves dozens of customer-specific adapters from a single GPU at substantially lower cost. Third, hot-swap adapters — the production endpoint loads new adapter versions dynamically, allowing canary deployment, A/B testing, and instant rollback without restarting the service. The pattern that turns fine-tuning from a quarterly project into a continuous capability.

Q: Should I use OpenAI fine-tuning, AWS Bedrock fine-tuning, or self-hosted fine-tuning on open-weights models?

The decision comes down to control, cost, and quality requirements. OpenAI, Anthropic, and Google fine-tuning APIs are the right choice for high-value lower-volume use cases where the proprietary model quality justifies higher per-token cost and where operational overhead must be minimised. Quality is excellent and there is no infrastructure to manage, but cost per token is much higher than self-hosted alternatives, and the fine-tuned model cannot be inspected, moved, or used in air-gapped environments. AWS Bedrock, Azure ML, Google Vertex AI managed fine-tuning sit in the middle — managed infrastructure but more flexibility and integration with cloud-native data and model registries. Self-hosted fine-tuning on open-weights models (Llama, Mistral, Gemma, Phi, Qwen) using Hugging Face TRL plus PEFT, Axolotl, or Unsloth is the right choice for sustained production volume, regulated workloads requiring air-gapped deployment, customer-tenanted SaaS using multi-LoRA serving, and any case where total cost of ownership matters. Most production deployments end up self-hosted because the per-token economics of running fine-tuned 7 to 14B models on dedicated infrastructure dominate at any meaningful scale.

Satyam Kumar

返回博客

enterprise-ai-platforms

LLM Fine-Tuning Guide: LoRA, QLoRA, DoRA, and Full Fine-Tuning Compared (2026)

By Satyam KumarApril 20, 202615 min read

llm fine tuning lora qlora dora fine tune llm parameter efficient fine tuning peft hugging face trl axolotl unsloth multi lora serving vllm llama fine tuning distillation continuous fine tuning enterprise ai platforms

LLM Fine-Tuning Guide: LoRA, QLoRA, DoRA, and Full Fine-Tuning Compared (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

LLM Fine-Tuning Guide: LoRA, QLoRA, DoRA, and Full Fine-Tuning Compared (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment