Retrieval-augmented generation (RAG) lets a local language model answer questions using your own documents. Most attention goes to the chat model, but retrieval quality usually decides whether a RAG system feels useful. Two components drive retrieval: the embedding model and the vector store. This guide explains how to choose both for a local setup.
What the embedding model does
An embedding model turns a passage of text into a vector, a list of numbers that captures its meaning. At question time, the question is embedded the same way, and the system finds passages with the most similar vectors. If embeddings are poor, the right passage is never retrieved, and no chat model can fix that.
Choosing an embedding model
- Language coverage: for Chinese or mixed Chinese and English documents, choose a model trained for multilingual retrieval. English-only models perform poorly on Chinese text.
- Size and speed: embedding models are small compared with chat models and often run on a CPU. Larger embedding models may retrieve better but take longer to index large collections.
- Maximum input length: match chunk sizes to what the model handles well.
- Consistency: once you index documents with one model, you must re-index if you switch.
Popular local options include the BGE family and nomic-embed-text, both of which can be served through common runtimes including Ollama.
Chunking matters as much as the model
Split documents along natural boundaries such as headings and paragraphs, with a modest overlap. Very small chunks lose context; very large chunks dilute similarity. Keep metadata such as file name, section and date for citations and filtering.
Choosing a vector store
- Small personal collections: an embedded store such as Chroma or SQLite with a vector extension is simple and adequate.
- Team or larger collections: Qdrant, Milvus or PostgreSQL with pgvector offer filtering, persistence and scaling.
- Hybrid search: combining keyword search (BM25) with vectors often improves results for names, codes and exact terms.
Evaluate with real questions
Collect twenty to fifty real questions with known answers. For each configuration, check whether the correct passage appears in the top results. Improve retrieval first, then tune the chat model and prompt.
Keep it private
All of these components run locally. Store indexes on encrypted disks and apply access control at retrieval time if several users share the system.
References
- MicroAI One DIY lab, local AI projects including RAG: https://microaione.com/lab/
- MicroAI One tutorials: https://microaione.com/tutorials/
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.