RAG vs fine-tuning vs long context: which to use in 2026

The RAG vs fine-tuning decision comes down to what you need to change. If the model lacks facts, supply them at request time, either with retrieval-augmented generation (RAG) or by putting the whole source in the prompt. Fine-tuning is for changing behaviour, such as a fixed output format, a house tone or a narrow set of labels. For a knowledge base under about 200K tokens, start by sending all of it in a cached prompt, and add retrieval when measurements of cost, latency or accuracy call for it.

Window sizes and cache prices changed the arithmetic this year. A 1M-token context window is now standard among frontier models, and a cache read costs 2.5% to 10% of the normal input price on the Anthropic, OpenAI and Google models in our October price comparison. In the worked example below, a 150K-token knowledge base costs $0.309 a question on Claude Sonnet 5.5 sent uncached, $0.028 cached and $0.016 with RAG. Accuracy hasn’t kept pace with window size, and studies from 2023 to 2026 find that models use long inputs less reliably as they grow. Fine-tuning has also become harder to buy, with OpenAI closing it to new users and Mistral deprecating its service.

The sections below cover what each approach changes, cost per question, latency, the accuracy evidence, fine-tuning and who still sells it, and hybrids. A decision table and a checklist close the post. List prices are as of 8 October 2026, from our LLM API price comparison. Fine-tuning prices are as of 16 October 2026.

What RAG, fine-tuning and long context each change

Retrieval and long context both change what the model sees, and neither touches its weights. With RAG, a retriever picks passages for each request, so updating knowledge means re-indexing what changed, and the retriever can apply each user’s permissions before text reaches the model. With long context, the whole source goes into every prompt, so no retriever can miss the passage that holds the answer.

Fine-tuning changes the weights, and what it changes reliably is behaviour. Two EMNLP 2024 papers tested it as a way to add knowledge. Gekhman and colleagues found that examples carrying new facts were learned much more slowly than examples consistent with what the model knew, and once learned, they linearly increased its tendency to hallucinate.¹ Ovadia and colleagues found that RAG consistently outperformed unsupervised fine-tuning, for knowledge seen in training and for entirely new knowledge.²

RAG vs long context: cost per question at October 2026 prices

Take one workload and price it three ways. The assumptions are ours, chosen to resemble a documentation or support assistant:

• A 150,000-token knowledge base, about 83,000 English words on Anthropic’s current tokenizer, which fits roughly 555,000 words in 1M tokens.³
• 2,000 tokens of instructions, a 100-token question and a 500-token answer. Reasoning tokens are left out, since they would add the same amount to every option.
• 1,000 questions a working day, 22 working days a month.
• Cached long context: instructions and documents form a 152,000-token cached prefix, and the cache goes cold and is rewritten 20 times a day.
• RAG: ten retrieved 500-token chunks per question, instructions cached, and each query embedded with OpenAI’s text-embedding-3-large at $0.13 per million tokens.⁴ Vector database hosting and engineering time are left out.

Claude Sonnet 5.5 costs $2 per million input tokens, $10 per million output tokens, $0.10 per million for cache reads and $2.50 per million for 5-minute cache writes.⁵ Per question:

• Whole knowledge base, uncached: (152,100 × $2 + 500 × $10) ÷ 1M = $0.3042 + $0.0050 = $0.309 a question.
• Whole knowledge base, cached: (152,000 × $0.10 + 100 × $2 + 500 × $10) ÷ 1M = $0.0152 + $0.0002 + $0.0050 = $0.0204. Cache writes add 20 × 152,000 × $2.50 ÷ 1M = $7.60 a day, or $0.0076 a question. Total $0.028.
• RAG: (2,000 × $0.10 + 5,100 × $2 + 500 × $10) ÷ 1M = $0.0002 + $0.0102 + $0.0050 = $0.0154. Instruction cache writes ($0.10 a day) and the query embedding ($0.000013) bring it to $0.016.

The same workload on four models, at list prices as of 8 October 2026:

GPT-6.1 Sol’s prices, cache writes included, match Sonnet 5.5’s at this prompt size.⁴ For Gemini 3.8 Flash we assumed Google’s default implicit caching, which has no write premium, and billed the 20 daily refreshes at the full input price. Google doesn’t guarantee cache hits, and the model’s introductory price doubles on 1 January 2027.⁶⁷

Caching cuts the cost of sending everything by about 91% on Sonnet 5.5, and RAG stays 1.8 times cheaper. At 22,000 questions a month that is $616 against $341, which leaves a retrieval pipeline $275 a month to cover building, hosting and evaluating it.

Price thresholds can change the answer. The 150,000-token prompt crosses Claude Haiku 5.5’s 100K threshold, above which every token, cache reads included, costs five times as much.⁵ Cached long context costs 13.5 times RAG there. GPT-6 models reprice above 272K tokens and Gemini 3.1 Pro above 200K.⁴⁷

For comparison with a measured number, the study assistant we run on an AI study platform sends a whole course of up to 60k tokens with each question. It costs about $0.018 a question, as we described in grounding a study assistant in live web search and the course itself.

Latency: long prompts, cache hits and retrieval steps

The model reads the whole prompt before it writes, so prompt length shows up as time to first token. Google’s long-context guide says every request carries some fixed latency and that longer queries generally take longer to the first token.⁸

Cache hits remove much of that reading. In Anthropic’s launch figures for prompt caching, a 100,000-token cached prompt cut time to first token from 11.5 s to 2.4 s.⁹ Those are vendor figures, on the models of that time. OpenAI says caching cuts the time spent processing input but gives no number.¹⁰

Retrieval adds its own steps before the model starts, because the query has to be embedded and the index searched, and the results reranked if you use a reranker. An agent that searches several times adds a model round trip per search. We haven’t measured these steps for this post, so time them in your own stack against the prompt-reading time they save.

Reasoning effort often outweighs both. In Artificial Analysis’s tests, Claude Opus 5.5’s first token came after 8.7 s at low effort and 731.5 s at max, as our price comparison reports. Our study assistant, with the whole course in context, answered fact questions in 3.5 to 6.5 s and summarised a whole course in 15 to 17.5 s.

What the research says about long-context accuracy

Position matters. In Lost in the Middle, models were most accurate when the relevant text sat at the start or end of the input. GPT-3.5-Turbo’s multi-document QA accuracy dropped by more than 20% depending on position, and with 20 or 30 documents its worst position scored below its 56.1% closed-book score, which it reached with no documents at all.¹¹

Length hurts, even on simple tasks. Chroma tested 18 models and found performance grew less reliable as input length grew. On LongMemEval, every model did significantly better with focused prompts of about 300 tokens than with full prompts of about 113k tokens.¹²

Long context wins on average, at a price. Li and colleagues compared long context with RAG on Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo.¹³ With enough resources, long context had the higher average score on all three, 49.70 against 37.33 on Gemini-1.5-Pro and 48.67 against 32.60 on GPT-4O, while RAG cost far less. The two gave identical predictions for over 60% of queries, and RAG’s failures clustered in multi-step, general, long and implicit questions.

Anthropic’s 200K rule of thumb. In 2024 Anthropic wrote that a knowledge base under 200,000 tokens, about 500 pages, can go into the prompt whole, with prompt caching to make that faster and cheaper.¹⁴ On its current tokenizer, 200K tokens is roughly 150,000 words.³ Its documentation also warns that accuracy and recall degrade as the token count grows.¹⁵

2026 results. LongBench Pro tested 46 models from 8K to 256K tokens and found effective context length typically shorter than the claimed length.¹⁶ ATLAS scored 26 models from 8K to 1M tokens. Seven moved two or more places between their 8K-to-128K and 8K-to-1M scores, and the leader changed from Gemini 3.1 Pro Preview at 128K to Claude Opus 4.6 at 1M.¹⁷ Vendor results on synthetic retrieval look stronger. OpenAI reports 96.3% for GPT-6 Astra between 512K and 1M tokens on its MRCR v2 test, which hides eight near-identical requests in a long conversation and asks for one, against 73.8% for GPT-5.6 Sol.¹⁸¹⁹ ATLAS’s authors caution that strong retrieval need not transfer to downstream use.¹⁷

Layout helps. Google reports about 99% accuracy on a single “needle” and lower accuracy when the model must find many pieces of information at once.⁸ Google and Anthropic both advise putting the question after the documents, which Anthropic says improved response quality by up to 30% in its tests.²⁰

Our own evidence is small. With the whole course in context, our study assistant answered 8 of 8 fact questions correctly, enough to show the design works for fact lookups and too few for an accuracy rate.

Does RAG still matter with 1M-token context?

For many products, yes. A 1M-token window holds about 555,000 English words on Anthropic’s current tokenizer, many document collections are larger, and effective context is shorter than the window. Retrieval also does things a window can’t. It can filter by each user’s permissions before text reaches the model, pick up a document edited an hour ago without rewriting a cache, and point to the passage behind each claim.

Agents with long windows still gain from it. Cursor reported that adding semantic search alongside grep raised its agent’s accuracy on codebase questions by 12.5% on average, and by 6.5% to 23.5% depending on the model, on Cursor’s own benchmark.²¹

Retrieval’s main failure is missing the passage. Embedding similarity measures how related two texts are, and related texts can differ in exactly the detail that matters. In our textbook-to-course engine, distinct topics scored 0.67 to 0.91 on cosine similarity with text-embedding-3-large and reworded duplicates 0.70 to 0.94, so no threshold separated them. Anthropic’s contextual retrieval prepends a short explanation to each chunk before indexing. Combined with BM25, it cut top-20 retrieval failures by 49%, and by 67% with a reranker.¹⁴ Keyword search also suits text dense with exact terms, which is why our study assistant uses BM25 above its 60k-token cap.

When to fine-tune an LLM

Fine-tune when prompts and examples can’t get the behaviour you need, or can’t get it cheaply enough at your volume. Three uses hold up.

• Format and tone. OpenAI’s guide says fine-tuning can make a model format responses consistently. It lists room for more examples than fit in a prompt, shorter prompts and a smaller, cheaper model as the advantages over prompting.²²
• Narrow classification. With stable categories and thousands of labels, a fine-tuned classifier is hard to beat. In tests we summarised in our post on decision models, ModernBERT-large fine-tuned on 25,000 labelled reviews matched Jev’s 96.5% on IMDb, and a DeBERTa prompt-injection classifier on a laptop CPU beat Jev by 89.0% to 86.4%.
• Distillation. Training a small model on a large one’s outputs can carry its behaviour across. Hsieh and colleagues’ fine-tuned 770M-parameter T5 beat a few-shot prompted 540B PaLM on a benchmark using 80% of the available data.²³

Most of this now runs on LoRA, which trains small low-rank matrices beside frozen weights. Its authors reported 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning of GPT-3 175B, with no added inference latency.²⁴

Cheap cache reads have weakened the cost case for fine-tuning. Take a classifier on Gemini 3.1 Flash-Lite on Google Cloud, which costs $0.25 per million input tokens, $0.025 cached and $1.50 output on the global endpoint.²⁵ Assume a 6,000-token few-shot prefix, a 300-token input and a 5-token label. Cost per million requests is tokens per request times the price per million tokens:

• Prompted, uncached: 6,300 × $0.25 + 5 × $1.50 = $1,582.50.
• Prompted, prefix cached: 6,000 × $0.025 + 300 × $0.25 + 5 × $1.50 = $232.50.
• Tuned, 400-token prompt: a tuned Gemini 3 endpoint costs 1.5 times the base price and is served only from US and EU multi-region endpoints.²⁵²⁶ We priced it at 1.5 times the non-global rate of $0.275 and $1.65, so 400 × $0.4125 + 5 × $2.475 = $177.38.

Training on 3,000 examples of 400 tokens for 3 epochs is 3.6M training tokens, or $10.80 at $3 per million. Caching shrinks the tuned model’s saving from about 89% to about 24%.

Who still sells fine-tuning, and what it costs

As of 16 October 2026:

Google Cloud and Together AI count training tokens as dataset tokens times epochs.²⁵³⁴ A tuned closed model is also tied to its base model’s lifespan. OpenAI serves tuned models until their base models are deprecated,²² and Bedrock blocks new tuning jobs once a model enters its Legacy stage.³⁵ With open weights you keep the adapter and can serve it on any host that supports LoRA, or on your own GPUs.

Hybrids: retrieval into long context, and small models that route

Retrieval that returns whole sections. Classic RAG passes a few short chunks; Li and colleagues used five 300-word chunks per query.¹³ With a long window you can retrieve whole sections or documents, tens of thousands of tokens, and let the model read them in order. Our study assistant does the simple version, sending the whole course up to 60k tokens and letting BM25 choose passages above that.

RAG with a long-context fallback. Li and colleagues’ Self-Route answers with RAG first, lets the model judge whether the retrieved text is enough, and sends the full context only when it isn’t. It cut cost by 65% on Gemini-1.5-Pro and 39% on GPT-4O, with performance comparable to long context.¹³

A small model that routes. A fine-tuned classifier or a decision model can send each question to the cached corpus, to retrieval, or to a person or a larger model. The decision models post linked above compares fine-tuned encoders, which suit stable categories, with decision models such as Jev and Clef, which read the options in each request. One cascade there matched GPT-5.6 Terra’s accuracy at 26% to 28% of its cost.

A tuned model on top of retrieval. Fine-tuning can fix the output schema and tone while retrieval or a cached prompt supplies the content, which keeps facts that will go stale out of the training set.

How the numbers here were put together

• Prices are standard pay-as-you-go rates from our 8 October 2026 comparison. Cache-write prices come from Anthropic’s and OpenAI’s pricing pages, and OpenAI’s were checked against its page as archived on 16 October 2026. Fine-tuning status and prices come from provider pages as of 16 October 2026.
• Worked costs are our arithmetic from the stated assumptions. They leave out reasoning tokens, vector database hosting, engineering time, batch discounts and taxes.
• Field notes come from two earlier posts about an AI study platform we build and run.
• Research results come from the cited papers, on the models they tested, most of them older than today’s frontier models. Vendor results are labelled as such.
• We ran no new accuracy, latency or cost tests for this post.

RAG vs fine-tuning vs long context: decision table and checklist

Before you build:

1. Count your corpus with each candidate provider’s token counter. Anthropic’s current tokenizer produces about 30% more tokens for the same text than its previous one.⁵
2. Compare that count, and your largest prompts, with the price thresholds: 100K tokens on Claude Haiku 5.5, 200K on Gemini 3.1 Pro and 272K on GPT-6 models.
3. Write 50 to 100 real questions with known answers, including answers that sit mid-document, questions that span documents and on-topic questions that should be answered. Our old, stricter course prompt refused 6 of 24 on-topic questions.
4. Run the cached whole-source baseline first, with the question after the documents. Record accuracy, p50 and p95 time to first token, and cost from billed tokens including cache writes.
5. Add retrieval when the baseline fails on cost, latency, accuracy, permissions or freshness, and measure retrieval recall separately from answer accuracy.
6. Log cache hit rates. A cold cache bills writes at 1.25 times the input price for Claude’s 5-minute cache and on OpenAI’s GPT-5.6 and later models.⁵¹⁰
7. Fine-tune only for behaviour, after prompts with examples fall short. Hold part of a labelled set back for evaluation, and price the tuned model’s inference as well as its training.
8. Before tuning a closed model, check the provider’s fine-tuning status and the base model’s retirement date. Re-run your evaluation at your real prompt length whenever you change model.

Frequently asked questions