"Should we use RAG or fine-tune it?" is the most common question I get about a model that needs to know something. It is usually the wrong question, because the third option has quietly become viable: context windows are now a million tokens, and a lot of corpora fit.
The three are not competing answers to one question. Retrieval and a long context window both change what the model sees. Fine-tuning changes how the model behaves. Picking between them starts with which of those two things you actually need, and only then becomes arithmetic.
The three options answer different questions
Fine-tuning changes the weights. You collect examples of the task done well, run a training job, and get a model that does that task the way your examples did it. What it is not good at is learning facts. Ovadia and colleagues tested exactly this in Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, and found that "LLMs struggle to learn new factual information through unsupervised fine-tuning", while "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge".
Retrieval keeps the knowledge outside the model. The original RAG paper framed it as combining parametric with non-parametric memory, and the practical consequence is the one that matters in production: you can change a fact by re-indexing a document instead of retraining anything.
Long context is retrieval without the retrieval. You put the whole corpus in the prompt and let attention do the selecting. It is the least code by a distance, and for a small corpus it is the right answer.
What each one costs
Here is a concrete workload: a 2-million-token corpus (company documentation, perhaps 5,000 pages), 100,000 queries a month, Claude Sonnet 5, a 1,200-token instruction and question, 500 tokens of output. Retrieval puts 6,000 tokens of retrieved text in each prompt.
Sending the whole corpus every time costs $4.01 a query, or $400,740 a month. Cache 90% of it, and the corpus is billed at a tenth of the input price most of the time: $0.77 a query, $76,740 a month. Retrieval costs $0.02 a query, and $2,112 a month including embeddings, reranking and $50 of vector store.
The index itself is almost free, which surprises people who have not priced embeddings lately. Embedding all 2 million tokens once costs four cents with OpenAI's text-embedding-3-small.
| Embedding model | $ per 1M tokens | 2M-token corpus, once |
|---|---|---|
| OpenAI text-embedding-3-small | $0.02 | $0.04 |
| OpenAI text-embedding-3-large | $0.13 | $0.26 |
| Voyage 4-lite | $0.02 | $0.04 |
| Voyage 4 | $0.06 | $0.12 |
| Voyage 4-large | $0.12 | $0.24 |
| Gemini Embedding 2 | $0.20 | $0.40 |
| Voyage rerank-3 (reranking, not embedding) | $0.05 | per 1M tokens scored |
The expensive part of retrieval was never the embeddings. It is the engineering: chunking, an index, a retrieval eval set, and the ranking work that decides whether the right chunk reaches the prompt at all.
The break-even corpus size
Because the long-context bill scales with the corpus and the retrieval bill does not, there is a corpus size where the lines cross. It moves with your query volume, and with whether the corpus is cached.
At 100,000 queries a month, retrieval is cheaper above about 36,000 tokens of corpus with 90% caching, or about 6,900 tokens without it. At 5,000 queries a month the break-even moves to about 61,000 tokens. At 500 queries a month it is about 298,000 tokens, and a 200,000-token corpus costs $41.70 a month to send whole against $60.31 for retrieval. The long context window wins outright at that volume, before you count a day of engineering.
This is why Anthropic's own guidance says that "if your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods". It is good advice for a corpus that size at ordinary volumes. Multiply the volume by two hundred and the same advice costs you $76,000 a month.
quick check
Your corpus is 2M tokens and you serve 100,000 queries a month on Claude Sonnet 5. You switch from sending the whole corpus (90% cached) to retrieving 6,000 tokens a query. Roughly what happens to the monthly bill?
The corpus is 2M tokens on every query against 6,000 retrieved ones. Even at the cached rate that is the whole bill: $76,740 becomes $2,112, of which $162 is embeddings, reranking and infrastructure.
Work out your own numbers
The defaults are the workload above. The two inputs that move the answer most are queries a month and the share of the corpus you can keep cached.
calculator · list prices, checked 12 Sep 2026
Retrieval or the whole corpus in the prompt?
cost a month
Counts tokens, embeddings and reranking only. It does not count the engineering time retrieval costs you: chunking, indexing, an eval set, and the ranking work that decides whether the right chunk is in the prompt at all. Fine-tuning is not in here because it answers a different question: how the model behaves, not what it knows.
Fine-tuning: what it is actually for
Fine-tuning earns its place when the problem is shape rather than substance: classify into your taxonomy, always answer in your JSON schema, write in a house style, do one narrow task on a small model that a big model does well.
Three things to know before you plan around it.
The platforms are moving. OpenAI's pricing page now says it "is winding down the fine-tuning platform" and that it "is no longer accessible to new users". Its listed training prices are still there for existing users, $5 per million training tokens on gpt-4.1-mini and $25 on gpt-4.1, so a ten-million-token training set is a $50 job. But building a roadmap on a service being wound down is not a plan.
A fine-tune is a snapshot. Every fact you train in is correct as of the training run. Retrieval makes a new fact live at the next index run, and a long context window at the next request. If your knowledge changes weekly, a fine-tune is stale weekly.
You still need the data. The cost that matters is not the training job, it is collecting and curating a few thousand good examples, plus the eval set to prove the new model is better than the old one. That eval set is the same work you need to check any change to an AI feature, which is one reason to build it early.
The honest combination, when you need both, is fine-tune for behaviour and retrieve for facts. A small fine-tuned model that reliably produces your output format, fed retrieved context, is cheaper and steadier than a large model doing both jobs from a prompt.
Where long context fails, and it is not the price
Cost is the easy part of this decision. Quality is the part that bites later.
A long prompt is not read evenly. Liu and colleagues, in Lost in the Middle, found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts". Anthropic describes the same effect as context rot: "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases".
So a corpus that technically fits is not the same as a corpus the model uses well. As the window fills, the odds that the one relevant paragraph gets the attention it needs fall, and they fall silently: you get a confident answer built on the wrong part of the document.
Retrieval has the mirror-image failure. If the right chunk is not retrieved, the model cannot use it, and again you get a confident answer from whatever was retrieved instead. This is measurable, which is the good news. Anthropic's contextual retrieval work reports failed retrievals dropping from 5.7% to 3.7% with contextual embeddings, to 2.9% adding BM25, and to 1.9% with reranking on top. The lesson is not the exact numbers, it is that retrieval quality is a number you can track and improve, so track it.
Latency differs too. Sending two million tokens means the model must read two million tokens before the first word comes back, cached or not. Retrieval reads six thousand. For anything interactive, that gap matters more than the money; the same reasoning applies to voice agents, where the budget is measured in milliseconds.
How to choose
- Is it behaviour or knowledge? Behaviour, and prompting has not fixed it: fine-tune. Knowledge: keep reading.
- Does the corpus fit comfortably in the window? Under about 200,000 tokens, start by putting it all in the prompt. Ship it in an afternoon and find out whether the feature is any good before building an index.
- Now multiply by volume. Corpus tokens times queries a month is the number that decides whether step 2 stays affordable. Run it through the calculator above before it is a surprise.
- Does the corpus change under you? Constant change favours retrieval, which re-indexes a document, over anything that needs a training run.
- Measure retrieval separately from generation. Keep a set of questions with known answers and known source documents, and track how often the right document is retrieved at all. Without that, a bad answer is unattributable.
- Cache whatever is stable. If you go long-context, the corpus is the most cacheable prefix you will ever have, and caching moves this bill by a factor of five.
If you are costing a whole feature rather than one request, the per-user version of this arithmetic is in LLM cost per user, and the build-cost version in what an AI chatbot actually costs to build.
questions people ask
Is RAG cheaper than a long context window?
Above a certain corpus size, yes, and the gap grows quickly. In the worked example, a 2M-token corpus at 100,000 queries a month costs $76,740 to send whole with 90% caching, against $2,112 with retrieval. Below about 36,000 tokens at that volume, sending everything is cheaper.
Should I fine-tune a model on my company documents?
Usually not. Research comparing the two found that models struggle to absorb new facts through unsupervised fine-tuning and that retrieval beats it, for both familiar and new knowledge. Fine-tune for behaviour, retrieve for facts.
When should I just use a long context window?
When the corpus is small, stable and your query volume is modest. Anthropic suggests that under 200,000 tokens you can simply include the whole knowledge base in the prompt, and at a few hundred queries a month that is cheaper than running an index as well.
How much does it cost to embed a corpus?
Very little. Embedding 2 million tokens costs $0.04 with OpenAI's text-embedding-3-small, $0.26 with 3-large, and $0.40 with Gemini Embedding 2. The real cost of retrieval is engineering time and the vector store, not the embeddings.
Does a bigger context window make RAG obsolete?
No. It removes the need for retrieval on small corpora, which is a real simplification. Long prompts still cost more per query, answer more slowly, and recall less reliably from the middle of the context.
Can I use retrieval and fine-tuning together?
Yes, and for many products it is the right answer: fine-tune a small model to produce your format and follow your task reliably, then feed it retrieved context for the facts.
The short version
Decide what you need first. Fine-tuning changes behaviour and is the weakest option for facts, so use it for format, tone and narrow tasks, not for company knowledge.
For knowledge, retrieval and long context are the same idea at different scales. Multiply corpus tokens by queries a month: if the product is small, put the whole thing in the prompt, cache it and move on. If it is large, the long-context bill grows on every query while the retrieval bill stays flat, and retrieval wins by a factor that gets embarrassing quickly.
Then spend the saving on retrieval quality. Both approaches fail the same way, with a confident answer drawn from the wrong text, and only one of them gives you a number to watch.