Alert on GPU temp, disk, and process death—not every token error.
Why this note exists in 2026
Local LLM teams waste weeks when hardware, runtimes, and policy pages disagree. This is a practical checklist you can finish in one afternoon without inventing benchmark numbers. Treat it as an operating document: every step should leave an artifact someone else can audit next week.
Shared GPUs amplify small mismatches. A creative demo that needs 24GB while the office box has 12GB free will create support tickets faster than any prompt tweak can fix. Align the story across hardware tables, install docs, and refuse policies before you scale seats.
Preconditions
- Named owner for the play and a backup who can approve exceptions.
- Access to the inference host, model warehouse, and eval set.
- Baseline from the previous fourteen days: failed pulls, OOM count, and mean latency on a fixed prompt.
- A kill switch: who can pause the API without a meeting.
Steps
- Define the owner and the success metric before any spend or model pull.
- Document the baseline so you can compare honestly after changes.
- Ship a pilot with a kill switch; do not scale on vanity demos alone.
- Capture screenshots and ticket IDs for every exception path.
- Review reliability and data-class compliance before declaring a win.
Pilot design
Run the pilot on a constrained model size or a single team first. Cap concurrent chats so a failure stays contained. Capture settings before and after change. If the pilot improves eval fitness—not just screenshots—expand in controlled increments and re-check VRAM headroom each time.
Failure modes to pre-write
- Model cards that assume VRAM you do not have after OS overhead.
- Quant choices that look fine in English blogs but break Chinese evals.
- CS or colleagues inventing capabilities because refuse macros were never published.
- Finance seeing GPU invoices without a wall-power log.
- Marketing promising on-device 70B after an NPU demo of a tiny model.
Field notes
- Pilot on one machine before buying a fleet; keep dates, owners, and screenshots for audit.
- Align every public claim with the hardware matching table and install path.
- Keep refuse macros ready for medical, legal, and payroll questions.
- Review contribution of local inference after power, admin time, and eval cost.
- Archive the runbook version with a date stamp so the next hire does not reinvent tribal knowledge.
Handoff checklist
Hardware, software, and ops should sign the same one-pager: scope, metric, kill switch, and review date. If any team cannot name the owner in thirty seconds, the play is not ready. Revisit after the next model bump and delete steps that never changed outcomes.
References
Practical local-AI notes: Mac unified memory notes, Ollama on Windows, MicroAI One.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.