RAG vs fine-tuning: when to use which

RAG puts information into the prompt at request time. Fine-tuning changes the model's weights ahead of time. That one structural difference explains almost every practical consequence: what each can fix, how fast you can iterate, what it costs, and how you keep it current.

Stated as a rule of thumb: if your model does not know something, retrieve it. If your model knows the answer but presents it wrongly, train it.

The pattern was introduced by Lewis et al. in 2020 and the production shape has barely changed since. Your documents are split into passages, each passage is embedded as a vector and stored in an index such as Pinecone, Weaviate, or pgvector. At request time the user's question is embedded, the closest passages are retrieved, and those passages are placed in the prompt with an instruction to answer from them.

In practice the model is rarely what limits quality. Retrieval is. The failures that matter are mundane: passages chopped mid-table so a number loses its column header, near-duplicate documents crowding out the one correct version, a superseded policy sitting in the index next to its replacement with nothing to distinguish them.

• Chunking that respects document structure — headings, tables, and list boundaries — rather than splitting on a fixed character count.
• Metadata on every passage: source, version, date, and who is allowed to see it.
• Access filtering applied during retrieval, so a user only ever gets passages they are entitled to.
• A reranking step when the corpus is large, because nearest-neighbour search alone degrades as the index grows.
• An explicit instruction to say it does not know when the retrieved passages do not contain the answer.

Fine-tuning continues training a pre-trained model on your own examples, adjusting its weights so it behaves more like those examples. In practice most teams use a parameter-efficient method such as LoRA, which trains a small number of additional parameters instead of the full model and makes the whole thing affordable.

What it is good at: producing output in a rigid schema every time, adopting a house style or register, handling a classification task with your own taxonomy, and cutting prompt length by moving lengthy instructions into the weights.

What it does not do is make the model reliably know your facts. Training on documents can make it produce text that sounds like your documents while inventing specifics, which is worse than an honest gap because it reads as authoritative. It also cannot enforce per-user access control, and it cannot be updated when a policy changes without another training run.

The fastest way to decide is to stop discussing the techniques and look at your actual failures. Collect thirty real outputs you are unhappy with and sort them into two piles: the content was wrong, or the content was right but the presentation was wrong.

• Mostly wrong content — missing facts, outdated policies, invented specifics: that is a retrieval problem. Fine-tuning will not fix it and may hide it.
• Mostly wrong presentation — malformed JSON, wrong tone, ignoring your category names, too verbose: that is a behaviour problem, and a candidate for fine-tuning once you have tried a clearer prompt and a few good examples in context.
• Both, which is common: fix retrieval first. Behaviour problems are much easier to judge when the underlying facts are already right.

RAG costs more per request — you are retrieving passages and sending longer prompts — but almost nothing to change. Fixing a wrong answer can be as simple as correcting the source document and re-indexing it, and the fix is live immediately.

Fine-tuning inverts that. Each attempt costs a dataset, a training run, and an evaluation, so the loop runs in hours or days rather than minutes. In exchange, prompts get shorter and per-request cost falls, which starts to matter at high volume.

At the start of a project, iteration speed is worth more than per-request cost, because you do not yet know what good looks like. That is the real argument for beginning with RAG, rather than any claim that it is better.

If your content changes weekly, fine-tuning alone is the wrong architecture and no amount of tuning rescues it. A fine-tuned model's knowledge is fixed at training time, and there is no clean way to remove one superseded fact from a set of weights.

This is also where the compliance argument sits. A retrieved passage can be cited, so a reviewer can check where an answer came from. Weights cannot be cited. If someone needs to audit why your system told a customer something, retrieval is not a preference, it is a requirement.

The two are complementary, and the order matters. Build retrieval first and get the facts right. Then, if a behaviour gap remains — output that still needs post-processing, a tone reviewers keep correcting — fine-tune a model to close that gap while continuing to retrieve the facts at request time.

Doing it the other way round wastes the training run. You end up encoding behaviour that suits the pre-RAG prompt shape, then changing that shape when retrieval lands.

Neither technique is worth deploying without a way to tell whether it helped. Build an evaluation set from real questions with agreed correct answers, hold it fixed, and run it on every change so improvements and regressions are both visible.

• For retrieval, score it separately from generation: did the correct passage appear in the retrieved set at all? If not, the generation score tells you nothing useful.
• For generation, score the properties you actually care about — factual accuracy against the source, correct refusal when the answer is absent, and schema validity where output is structured.
• Keep a small set of adversarial cases: questions your corpus genuinely cannot answer. A system that invents an answer for those will do the same in production.
• Re-run the whole set on every model, prompt, index, or chunking change. Changes that look unrelated frequently are not.