These two approaches get compared as if they were competing solutions to the same problem. They are not. They address different failure modes, and choosing between them starts with correctly diagnosing which failure you have.

The distinction in one line:

• Retrieval-augmented generation (RAG) changes what the model knows at the moment it answers, by finding relevant material and supplying it alongside the question.
• Fine-tuning changes how the model behaves, by further training it on examples until it adopts a pattern.

Knowledge versus behaviour. Almost every correct decision follows from working out which one is missing.

Diagnosing the problem

Take a real failing example and ask: would a capable new employee get this right if you handed them the right document?

If yes, the problem is knowledge. The model does not have access to your policy, your product catalogue, your contract terms, your current pricing. No amount of behavioural training fixes that — you need to get the information in front of it. That is RAG.

If no — if a capable person with the document would still produce the wrong shape of answer, because it needs a particular format, a particular tone, a particular set of conventions specific to your organisation — then the problem is behaviour. That is where fine-tuning earns its place.

In our experience the large majority of business cases turn out to be knowledge problems dressed as behaviour problems. Teams report “the model does not understand our business” and reach for fine-tuning, when what is actually missing is retrieval over their own documentation.

Why RAG is usually the right starting point

Your information changes

Prices change, policies get updated, products are discontinued, contracts are renegotiated. With RAG you update the source document and the next answer reflects it. With fine-tuning, updating knowledge means retraining — so your model is stale between training runs, and quietly confident about facts that expired last quarter.

For anything that changes more often than annually, this consideration alone usually settles it.

You can show your sources

A RAG system knows which documents it used, so it can cite them. Users can verify. Reviewers can audit. When an answer is wrong you can inspect what was retrieved and understand why.

A fine-tuned model produces knowledge with no traceable provenance. When it is wrong, you have very little to work with diagnostically. For any regulated context, or any case where users need to trust the output, citation is close to essential.

Access control still works

This one is frequently discovered too late. If you fine-tune on your internal corpus, that information is absorbed into the model’s weights. Anyone who can query the model can potentially surface any of it, and you cannot meaningfully apply per-user permissions after the fact.

With RAG, retrieval respects your existing entitlements — the system searches only what that user is allowed to see. For any organisation with confidential material or differentiated access, this is a decisive architectural advantage, not a detail.

It is faster and cheaper to iterate

RAG improvements are engineering changes: better chunking, better search, reranking, richer metadata. You can try something and measure it the same day. Fine-tuning requires a curated dataset, a training run and evaluation for each iteration.

When fine-tuning genuinely earns its place

A consistent output format that resists instruction

When you need output in a very specific structure every single time, and prompt instructions get you most of the way but not reliably enough. Fine-tuning on several hundred correct examples produces far more consistent adherence than a long instruction ever will.

A specialised tone or house style

Organisations with a distinctive voice, or a domain with strict conventions in how things are phrased, often find that describing the style is much harder than demonstrating it. If you can show a hundred good examples more easily than you can write the rules, that is the signal for fine-tuning.

Domain language and classification at volume

Highly specialised vocabulary — clinical, legal, industrial — where general models misinterpret terms of art. And high-volume classification, where a smaller fine-tuned model can match a much larger general model’s accuracy on your narrow task at meaningfully lower cost per request.

That last point is a genuine and often-overlooked benefit: fine-tuning can be a cost optimisation, letting a smaller model do a specific job well.

Compressing an unwieldy prompt

If your instructions have grown to a large block of rules sent with every request, fine-tuning can absorb much of that into the model, reducing per-request cost and latency. This is an optimisation to apply once behaviour is settled — not a starting point.

Using both

They compose well, and mature systems often use both: a fine-tuned model that reliably produces your required format and tone, retrieving current information at query time so its facts stay fresh and attributable.

The sensible sequence is almost always:

1. Start with a good prompt and a solid RAG implementation.
2. Measure against a real evaluation set.
3. Identify residual failures and classify each as knowledge or behaviour.
4. Fix knowledge failures by improving retrieval — this is where most of the remaining gains are.
5. Only if behavioural failures persist, and you have the examples to train on, consider fine-tuning.

Skipping to step five is the most common and most expensive mistake in this area.

Comparing the running costs

The cost profiles differ in ways that matter at production volume, and they are frequently compared on the wrong basis.

Fine-tuning carries an upfront cost — dataset preparation, which is mostly human effort, plus the training run itself — and repeats that cost each time you retrain. Its ongoing per-request cost can be lower than the alternative, because the behaviour is baked in and requests carry fewer instructions.

RAG has a modest setup cost but a permanent operational one: hosting and updating a search index, and larger requests, because retrieved material is sent with every question. More retrieved context means higher per-request cost and higher latency.

The crossover depends almost entirely on volume and on how much context each answer needs. At low volume, RAG is cheaper overall because you never pay for training. At high volume with a narrow, stable task, a fine-tuned smaller model can be substantially cheaper per request — enough that the training cost is recovered quickly.

Two practical points. First, model this against your expected production volume, not your pilot volume; the ratio between them is often large enough to reverse the conclusion. Second, do not let cost drive the initial architecture. Get something correct first, measure it, then optimise — including by reducing how much context RAG retrieves per request, which is usually the cheapest available saving and improves answer quality at the same time.

Evaluate retrieval separately

Measure retrieval and generation independently. For a set of real questions with known-correct source documents, check whether retrieval surfaced the right material at all. If it did not, no change to the model or prompt will help — and teams routinely spend weeks tuning generation to fix what is a retrieval failure.

What actually makes RAG work

RAG is easy to demonstrate and harder to do well. Most disappointing RAG systems fail in retrieval, not generation — the model answers reasonably given what it was handed, but the wrong material was handed to it.

The parts that repay attention:

• Chunking. Splitting documents arbitrarily severs context. Split along the document’s own structure — sections, clauses, headings — so each chunk is self-contained.
• Hybrid search. Semantic search alone misses exact identifiers: part numbers, clause references, error codes. Combining keyword and semantic search materially improves recall.
• Reranking. Retrieve generously, then rerank to select what actually gets used. Usually a larger accuracy gain than swapping models.
• Metadata filtering. Filter by date, department, document type and permissions before ranking, so stale or unauthorised material never reaches the model.
• Source quality. RAG over contradictory, outdated documentation produces contradictory, outdated answers. Very often the highest-value work in an AI project is cleaning up the knowledge base — which is unglamorous, and benefits your human staff regardless of the AI.

How we approach it

Our AI development engagements begin by diagnosing whether you have a knowledge problem or a behaviour problem, using your real failing examples rather than assumptions. In most cases the answer is a well-built RAG system over a tidied knowledge base — and where your source data needs collecting or structuring first, our data engineering team handles that.

Talk to us about what your system is getting wrong, and we will tell you which approach fixes it.