Fine-tuning vs RAG: which one does a small team actually need?

The difference

RAG comes from a 2020 paper by Patrick Lewis and co-authors that paired a text generator with a retriever over a document index. The model's knowledge stays the same; what changes is what it reads before answering. Remove an outdated document and the next answer changes immediately.

Fine-tuning changes the model itself by training it on examples of inputs and ideal outputs. OpenAI's model optimization guide lists when it helps: when you have more examples than fit in one prompt, when you want shorter prompts, and when you want to train on sensitive data without sending it with every request. It also notes that fine-tuning can be a time-consuming process.

The same guide, read on 4 November 2026, says "OpenAI is winding down the fine-tuning platform": it is no longer accessible to new users, existing users can create training jobs for the coming months, and fine-tuned models remain available until their base models are deprecated.

Why a small team should care

• Fine-tuning doesn't keep facts current. A model trained on last quarter's prices still says last quarter's prices.
• Most "the AI doesn't sound like us" problems are instruction problems. A style guide and three example replies usually fix them.
• You can't choose without measuring. Anthropic's prompt engineering overview starts with success criteria and a way to test them; that test also tells you whether anything heavier is needed.

A worked example

Illustration: a made-up business. Greenfield Insurance Brokers, a 14-person brokerage, wants an AI that drafts client emails in its careful tone and quotes the right policy limits. A consultant proposes fine-tuning on 2,000 past emails.

The operations lead first writes 20 test cases from real client questions. With a plain prompt, 9 pass. Adding a tone guide and three example emails brings it to 14; the failures are all wrong policy limits. Uploading the current policy summaries as searchable documents brings it to 19. The remaining failure is a policy nobody had written down. Fine-tuning would have taught the old limits from the old emails.

How to decide

1. Write 20 test cases from real work, with what a passing answer must contain.
2. Run a clear prompt and record the pass count.
3. Add examples and a style guide; run again.
4. Add your current documents (RAG) if failures are about facts; run again.
5. Consider fine-tuning only if a measured gap remains that instructions and documents can't close, and check that your provider still offers it.

Common confusions

• "Fine-tuning teaches the model our knowledge base." It learns patterns from examples; current facts belong in documents it reads.
• "RAG is a product you buy." It is a pattern; a project with uploaded files already does it. See RAG and vector databases.
• "We'll measure later." The eval set is how you know which one worked. See evals.

Questions

What is the difference between fine-tuning and RAG?

RAG leaves the model unchanged and gives it relevant passages from your documents each time it answers. Fine-tuning trains the model further on your examples, changing how it behaves. RAG is for facts that change; fine-tuning is for a behaviour or format you can't get with instructions and examples.

Can I still fine-tune OpenAI models?

OpenAI's model optimization guide, read on 4 November 2026, says it is winding down the fine-tuning platform: it is no longer open to new users, existing users can create training jobs for the coming months, and fine-tuned models stay available until their base models are deprecated.

Should a small team fine-tune a model?

Rarely. Start with clear instructions and examples, add your documents through RAG if the task needs your facts, and measure both with a small eval set. OpenAI's guide notes that prompt engineering may be all you need.