Why "CSV exploration with a local LLM" matters for local RAG
Schema first; never upload customer CSVs to public chats. Retrieval-augmented generation is usually the first serious project a team builds on a local model, because it keeps documents in-house and avoids fine-tuning. It is also where small design choices decide whether people trust the answers.
Ingestion comes first
Bad input produces confident wrong answers. Before thinking about embeddings:
- Extract structure: headings, tables and lists should survive conversion from PDF or Office files.
- Attach metadata to every chunk: source file, section, version and document date.
- Reject documents that have no owner or no date; stale policies are a liability.
- Keep the original files so you can re-ingest when your splitter improves.
Chunking and retrieval
Start with moderate chunk sizes and some overlap, then evaluate on your own questions instead of copying settings from a blog. Chinese text needs a splitter that respects sentences rather than English whitespace rules. Pure vector search misses exact identifiers such as part numbers, contract IDs or SKUs, so keep a keyword index alongside the vector store and merge the results. When policies change over time, bias retrieval toward the newest valid version.
Evaluation you can repeat
Build a set of real questions with known answers and the documents that should support them. After each change to chunking, embedding model or prompt, run the set and look at both the answer and the retrieved passages. If the right passage was not retrieved, no prompt will fix it.
Operations
- Back up the vector store and test a restore.
- Separate collections per department or client so access rules are easy to enforce.
- Log which sources were cited for each answer.
- Decide what the assistant must refuse to answer, and publish that list.
A worked example
A small manufacturer wanted engineers to ask questions about internal specifications. The first version used only vector search and often returned the wrong revision of a document. The team added document dates and revision numbers as metadata, filtered out superseded revisions before ranking, and kept a keyword index for part numbers. They also rewrote their chunking so that tables were stored as separate records. After these changes, they re-ran the same evaluation questions and checked whether the correct source appeared in the retrieved passages. Only when retrieval was reliable did they spend time adjusting the answer prompt.
For related guides, see MicroAI One software; you may also find local RAG knowledge-base guide useful.
Bottom line
Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.
Quick reminders
- Use one models directory shared by every runtime.
- Keep the previous model file until the new one passes your eval set.
- Review the setup again after every major model upgrade.
- Pin driver and runtime versions together.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.