Chat gets the glory; embedding jobs quietly saturate disks and CPUs overnight. Sizing only for interactive VRAM leaves batch RAG builds fighting for scraps.
Separate the profiles
Interactive chat wants low latency and reserved GPU memory. Embedding pipelines often batch well on GPU but can also run on CPU when overnight wall-clock is acceptable. Decide which jobs may touch the GPU during meetings.
Sizing questions
- How many documents per week need (re)embedding?
- Do you rebuild indexes fully or incrementally?
- Is the bottleneck encode speed, disk IO, or vector DB writes?
- Will mobile capture add bursty image OCR before text encode?
Map answers to match/vram-table, hardware/budget-5000, and storage layout practices. Install paths: tutorials/install. Architecture help: solutions/inquiry.
Extra
Run a pilot on a representative folder and extrapolate carefully—do not multiply peak GPU util by document count without measuring batching. Keep embedding models in their own directory. Document whether GPU embedding is allowed at 10 a.m. Community tales: community. Lab notes: lab. Desktop products: microai-one.
If chat and embed always collide, you do not have a prompt problem—you have a schedule and capacity problem.
Scheduling template
Publish a calendar: evenings for heavy re-embeds, lunch for light syncs, meetings for chat priority. If a full rebuild is unavoidable in daytime, announce it. Measure encode rate on a sample and extrapolate with a safety factor rather than best-case batch sizes from a blog. Revisit when document volume doubles.
Hardware split examples
Some teams keep a quieter secondary GPU or a CPU-heavy box just for embeds. Others accept slower overnight CPU embeds to protect chat latency. Write the choice explicitly including power cost. When evaluating a purchase, model the embed backlog after a major doc migration, not only steady state. Include disk endurance if you rewrite indexes often. Coordinate with IT on UPS sizing if overnight jobs stretch. Re-evaluate after the corpus doubles or when chat SLOs tighten. Link capacity sheets to match so newcomers see the same numbers.
Capacity narrative for leadership
Explain embed backlog in business days, not only in tokens per second. If a migration will block chat GPUs for a weekend, say so early. Prefer a cheaper second device over angry meetings. Track energy use for overnight jobs if the office pays metered power. Revisit architecture when RAG becomes customer-facing, because rebuild SLOs tighten. Keep diagrams of data flow next to the hardware sheet so new architects do not rediscover collisions. Schedule a joint review between search owners and chat owners every quarter.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.