Framing "Embedding model on shared vs dedicated GPU"
Dedicated if RAG is hot; else schedule off-peak. Local knowledge bases fail in predictable ways: outdated documents outrank current ones, tables get flattened into nonsense, and exact codes never match because embeddings blur them. Each failure has a simple, boring fix.
Fix the documents
Assign an owner for each collection. Store a document date and version as required fields. Convert tables to structured text or keep them as separate records instead of letting a paragraph splitter cut them in half. Remove duplicates so the same paragraph does not dominate results.
Fix retrieval
Use hybrid retrieval: keyword search for identifiers and rare terms, vectors for meaning. Filter by metadata before ranking, for example by department, product line or validity date. Return a few passages with their sources so users can check the answer themselves.
Fix expectations
The assistant should summarise and point to sources, not invent policy. Publish a short list of things it will not do: legal interpretation, medical advice, payroll decisions. Show the source document name and date with every answer, and make it easy to report a wrong one.
A two-week pilot
- Pick one team and one document collection.
- Write thirty real questions with expected sources.
- Run the pilot, collect feedback, and fix ingestion issues first.
- Only then expand to more collections or users.
A worked example
A small manufacturer wanted engineers to ask questions about internal specifications. The first version used only vector search and often returned the wrong revision of a document. The team added document dates and revision numbers as metadata, filtered out superseded revisions before ranking, and kept a keyword index for part numbers. They also rewrote their chunking so that tables were stored as separate records. After these changes, they re-ran the same evaluation questions and checked whether the correct source appeared in the retrieved passages. Only when retrieval was reliable did they spend time adjusting the answer prompt.
Further reading from MicroAI One: MicroAI One software, and local RAG knowledge-base guide.
Bottom line
Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.
Quick reminders
- Keep a printed fallback for offline demos.
- Pin driver and runtime versions together.
- Write the date and owner on every change record.
- Photograph cable routing before closing the case.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.