When a local model does not know enough about your business, two approaches come up immediately: retrieval-augmented generation, usually called RAG, and fine-tuning. People often treat them as competitors. In practice they solve different problems, and choosing the wrong one wastes weeks.
What RAG is good at
RAG keeps your knowledge in documents and retrieves relevant passages at question time. The model reads those passages and answers based on them. This works well when:
- Your information changes often, such as price lists, policies or product specs.
- You need answers to cite sources so people can verify them.
- Different users should see different documents, which you can enforce at retrieval time.
- You want to update knowledge by editing files rather than retraining.
RAG struggles when the answer depends on patterns spread across many documents, or when the model needs to adopt a specific style or format consistently.
What fine-tuning is good at
Fine-tuning, including lightweight methods like LoRA, adjusts the model itself using example inputs and outputs. It works well when:
- You want a consistent tone, format or structure in every answer.
- The task is narrow and repetitive, such as classifying tickets into fixed categories.
- You want a smaller model to perform a specific task as well as a larger general one.
Fine-tuning is a poor way to teach facts that change. Once the facts change, the model still remembers the old version until you train again, and it will not cite where an answer came from.
A simple decision rule
Ask whether your problem is mainly about knowledge or mainly about behavior. Knowledge problems point to RAG. Behavior problems point to fine-tuning. Many real projects need both: RAG for facts, a light fine-tune or a carefully written system prompt for format.
Try the cheap option first
Before fine-tuning, try a better system prompt and a few examples in the prompt. This often closes much of the gap at zero training cost. Before building a complex RAG pipeline, test with a small set of clean documents to confirm retrieval finds the right passages.
Hardware considerations
RAG mainly needs memory for the chat model, the embedding model and the context. Fine-tuning needs additional memory during training, even with efficient methods. If you plan to fine-tune locally, check the LoRA VRAM guide before buying hardware. For RAG, the local RAG knowledge base tutorial explains the moving parts.
Measure the outcome
Whichever path you choose, prepare a fixed set of test questions with expected answers before you start. Run them before and after each change. Without that, it is very hard to tell whether the project is actually improving.
Choosing between RAG and fine-tuning becomes much easier once you name the problem precisely. Most teams discover that their first need is cleaner documents and a clearer prompt, and only later a trained adapter.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.