A local RAG assistant often needs two models at once: an embedding model to search documents and a chat model to answer. Both want memory, and they are rarely sized together.
List what must be resident
- Chat model weights at your chosen quantization.
- KV cache for the context length and number of concurrent users.
- Embedding model weights.
- Runtime overhead and the desktop environment if the GPU also drives a display.
Approaches
- Everything on the GPU: simplest and fast, if it fits with margin.
- Embeddings on CPU: embedding models are often small enough to run acceptably on CPU, especially when indexing is a batch job.
- Load on demand: swap models in and out. This saves memory but adds latency, so use it for infrequent tasks.
- Separate machines: one for indexing and one for serving.
Practical sizing
Start by loading the chat model alone and noting memory use at your intended context. Add the embedding model and check again. Leave headroom for spikes during long prompts; running at the edge invites out-of-memory errors.
Schedule heavy jobs
Index building competes with users. Run it overnight or on a different device whenever possible.
Watch the runtime settings
Some servers keep models loaded for a set time. Adjust that policy so a rarely used embedding model does not stay in memory all day.
Document your configuration
Note model names, quantizations, context settings and the resulting memory use. That record makes upgrades far less painful.
To estimate how much your card can hold, use the VRAM table and the device and model match tool. Troubleshooting out-of-memory errors is covered in the troubleshooting tutorials.
A short checklist
Before going live, confirm that both models load after a reboot, that memory use stays stable during a long conversation, that indexing a large folder does not freeze chat, and that logs show no out-of-memory warnings. Repeat the check after every software update, because default memory behavior can change between versions, and write the results in your notes for the next person.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.