When to fine-tune, when to use RAG, and when to just fix the prompt

Every AI engineering team eventually has a version of this argument. It is Wednesday afternoon. An engineer proposes fine-tuning the customer-support bot on six months of chat transcripts. Someone asks about RAG. "We tried, not good enough." Someone asks about few-shot examples. Silence. The team spends the next six weeks fine-tuning. The fine-tuned model beats the baseline by three points. In week seven, someone runs the eval nobody thought to run first, and the base model with four handpicked few-shot examples lands within a point of the fine-tune on the customer-facing metric.

The lesson is not that fine-tuning is bad. The lesson is that the team never weighed fine-tuning against the two cheaper alternatives, and the six weeks it cost to find that out are gone. Every technique for adapting an LLM to a specific problem (prompt engineering, RAG, fine-tuning) has a defensible use case. The failure mode is not picking the wrong one; the failure mode is picking one before you have a reason to prefer it over the other two.

Issue 003 covered the evaluation framework that tells you whether a change worked. This issue is about which change to try in the first place. What follows is a decision framework for the three techniques, the cost and latency numbers that make the choice concrete, and the combinations that beat any of them alone.

Why teams default without weighing

Three tribes tend to form around AI engineering work, each with a default technique they reach for before considering the alternatives. Recognising which tribe you sit in is the fastest way to notice you are defaulting rather than deciding.

The prompt-first tribe is usually engineers who came from product or full-stack backgrounds. They reach for prompt engineering because it is fast, cheap, and requires no new infrastructure. Their weakness is undershooting: they push prompts past the point of returns because trying RAG or fine-tuning feels like admitting the prompt was not enough. The tell is a system prompt with fifteen bullet points that started as three.

The RAG-first tribe is usually engineers who came from search, infrastructure, or data-adjacent work. They reach for retrieval because it slots into architectures they already know how to build. Their weakness is over-engineering: they build hybrid retrieval with reranking for a use case where the answer fits in a five-line prompt, and they spend three weeks on chunking strategy before checking whether the base model would have known the answer anyway.

The fine-tune-first tribe is usually engineers who came from ML research or MLOps. They reach for training because it is the technique they have the deepest tools for. Their weakness is over-committing: they build a fine-tuning pipeline before establishing what problem the fine-tune is meant to solve, and they discover halfway through that the base model with a better prompt was fine.

There is no clean way to make any of these tribes stop preferring their default; the goal is to force a moment of explicit comparison before the default runs. The framework below is that moment written down.

The three techniques, briefly

Prompt engineering conditions the base model with instructions, examples, and formatting rules delivered as text at inference time. This includes zero-shot prompts, few-shot examples, chain-of-thought scaffolding, and structured output constraints. Nothing about the model changes; only what the model sees changes. Setup takes hours to days; per-request cost is whatever the tokens cost you.

RAG retrieves relevant content at inference time and inserts it into the prompt before generation. The retrieval layer is the bi-encoder from Issue 002, optionally with the hybrid retrieval from Issue 006, the chunking from Issue 007, and the reranker from Issue 005. Everything RAG does could in principle be done by pasting the entire corpus into every prompt; the point of RAG is to do that at scale. Setup takes one to four weeks; per-request cost is retrieval plus the LLM call.

Fine-tuning updates the model weights on a task-specific training set. This ranges from lightweight adapter methods (LoRA, QLoRA) that train a small delta on top of the base model, to full fine-tuning that updates every parameter. API providers (OpenAI, Google, and increasingly others) offer fine-tuning as a hosted product; the open-source ecosystem offers the same on any GPU you can rent. Setup takes one to eight weeks; per-request cost varies dramatically with whether you host the fine-tuned model yourself or pay an API for hosted fine-tuned inference.

The decision framework

The framework runs three questions in a specific order. Start at the top; take the first "yes" answer.

The tree looks trivial and hides everything real. Two things about it are worth spelling out.

The sixty-percent threshold in the first question is doing most of the work. It exists because prompt engineering scales predictably up to a plateau and then hits diminishing returns. If a decent zero-shot or few-shot prompt gets you to sixty percent on your golden set, you are within reach of eighty or ninety by iterating on the prompt, adding examples, and tightening the output schema. If you are at ten percent, no amount of prompt engineering will save you; the model does not know what you need it to know, and you have to reach for the knowledge (RAG) or the behaviour (fine-tune) directly. The threshold is a starting point to tune against your own eval; the point is to make the "we should just prompt harder" call an evidence-based one.

The second question distinguishes knowledge from behaviour. A model that returns the wrong fact does not know the fact; RAG fixes this. A model that returns the right fact in the wrong tone knows the fact just fine; fine-tuning fixes the tone, or a prompt with tone examples does. Getting this distinction wrong is the single most common cause of teams spending six weeks fine-tuning what a retrieval pipeline would have solved in a day.

The third question exists because fine-tuning without a dataset is wishful thinking. You need at least a thousand curated examples of the desired output for LoRA to reliably move the needle, and closer to ten thousand for full fine-tuning to justify its cost. If you do not have the examples, you have to either generate them (from a stronger model, from human labellers, from production traces) or reframe the problem as one prompt engineering or RAG can handle.

Cost and latency: the tables that decide the argument

The framework above tells you what to try. The table below tells you what each option costs. Numbers are approximate July 2026 figures for a mid-scale LLM application; your own numbers will differ, and pricing keeps drifting downward across every row.

Two patterns are worth pulling out of the table.

RAG's setup cost is dominated by the embedding pass over your corpus, which is a one-time cost that scales linearly with corpus size. For a ten-million-token corpus, that is roughly $0.20 at the cheap end of the providers and $1.30 at the expensive end. For a hundred-million-token corpus, roughly $2 to $13. The ongoing cost of RAG is dominated by the LLM call itself, because retrieval is cheap by comparison. If retrieval is expensive in your setup, revisit the chunking and vector-database choices from Issues 002, 006, and 007.

Fine-tuning has bimodal economics. If you fine-tune a small open-source model with LoRA and host it yourself, the per-request cost can drop by a factor of ten compared to a hosted large model, and the training run pays back quickly at scale. If you fine-tune a hosted model through an API, the training run is paid up front and the per-request cost is usually the same as (or slightly higher than) the base model. The choice between those two economics is often a bigger decision than the choice to fine-tune at all.

When to combine, not choose

The framework presents the three techniques as alternatives, but production systems routinely combine them.

RAG plus prompt engineering is the default combination. Retrieve relevant content, insert it into a well-designed prompt with instructions and format rules, generate. Almost every production RAG system does this; the question is only how tight the prompt is.

Fine-tuning plus RAG is often the answer for teams that have hit ceilings on both techniques alone. Fine-tune a small model to learn the desired tone, output format, or refusal patterns, then use RAG to give it factual grounding at inference time. The fine-tune handles behaviour; the RAG handles knowledge.

Fine-tuning to distil is the pattern that changes unit economics. Use a large frontier model with a strong prompt to generate a training set, fine-tune a much smaller model on that set, and deploy the small model in production. The small model runs at ten to fifty times lower cost per token, and the fine-tune inherits the frontier model's behaviour on the specific task. The tradeoff is that the small model matches the frontier only on the distribution you distilled; anything outside that distribution is a coin flip, and you will not notice until a user finds the edge.

Common mistakes

Fine-tuning without an eval. A fine-tune that improves training loss on your dataset is not the same as a fine-tune that improves production quality on your users' queries. Without the golden-set evaluation from Issue 003, you have no way to tell whether the six weeks of training helped, hurt, or moved the metric sideways. Run the eval before, during, and after every fine-tune, and treat "no measurable improvement" as a legitimate outcome that should stop the project.

Using RAG for style problems. If your model returns technically correct answers in the wrong tone, RAG cannot help you. Retrieval augments what the model knows; it does not change how the model writes. Style problems belong to prompt engineering (add examples of the desired tone) or fine-tuning (train on outputs in the desired tone), not to the retrieval layer.

Jumping to fine-tuning before exhausting prompts. Fine-tuning is the technique with the highest fixed cost and the slowest feedback loop, and it is also the technique with the smallest chance of being the right first move. If you have not tried a well-structured system prompt with four to eight curated few-shot examples on your golden set, you have not earned the right to fine-tune yet. This rule fails roughly one percent of the time and saves months in the other ninety-nine.

Treating the choice as permanent. The three techniques are not mutually exclusive and the right combination changes as the product and the model landscape do. A team that started with prompt engineering on GPT-4o in 2024 might now be running a fine-tuned small model with RAG; the fine-tune replaced the prompt, and the RAG covers what the fine-tune could not. Re-evaluate the choice every six to twelve months. The frontier moves.

The takeaway

The three techniques for adapting an LLM to your problem are not tribes to belong to; they are tools to compare against a specific problem and a specific eval. The decision framework runs three questions in order (does the base model get you close, is the gap knowledge or behaviour, do you have the training data), and the cost and latency table tells you what each answer costs to try. Combinations beat individual techniques for most real applications, and the choice is rarely permanent. If the team is spending more than a week on a technique choice, someone has skipped the eval. Write down what "good" would look like on a real benchmark before writing another line of retrieval or training code.

Production checklist

• Write down the eval before writing any adaptation code. If you cannot describe what a passing response looks like on twenty concrete examples, stop and do that first.

• Establish the baseline: run the base model with a decent zero-shot prompt against the golden set from Issue 003. The number you get is what every subsequent technique has to beat.

• Try prompt engineering next, including four to eight curated few-shot examples. Iterate until you plateau, and note where the plateau lands.

• If the plateau is knowledge-shaped (missing facts, outdated information, need for citations), add RAG using the retrieval stack from Issues 002, 006, and 007.

• If the plateau is behaviour-shaped (wrong tone, inconsistent format, refusal patterns), consider fine-tuning, but confirm you have at least a thousand curated examples of the desired output before spending on training.

• Combine techniques deliberately. RAG plus prompt is the default; fine-tune plus RAG is the answer for teams that have hit ceilings on both alone.

• Track per-technique cost and latency as separate metrics: setup time, ongoing per-request cost, added latency. The table above is a starting point; your production numbers are the source of truth.

• Re-evaluate the choice every six to twelve months. The frontier moves, and the technique that was clearly right last year may not be this year.

Further reading

• Anthropic, "Prompt engineering overview" - docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview

• OpenAI, "Fine-tuning documentation" - platform.openai.com/docs/guides/fine-tuning

• Hamel Husain, "Your AI product needs evals" - hamel.dev/blog/posts/evals

• Eugene Yan, "Patterns for building LLM-based systems and products" - eugeneyan.com/writing/llm-patterns