Average tokens-per-second hides pain. Users feel the slowest 10% of requests: cold starts, cache thrash, disk stalls.
Simple measurement
A shell loop that sends fixed prompts at different hours, records end-to-end latency, and saves results to CSV is enough to start. Note whether the model was already loaded.
Interpret
Separate load delay from generation delay if your server exposes both. Watch for antivirus scans and Windows Update reboots.
Act
Keep a warm model during office hours, limit concurrent heavy jobs, move weights to a fast disk.
Pair with troubleshooting and sizing vram/16gb. Overview microaione.com.
When someone says "it feels slow", ask for timestamped samples. Tail latency turns feelings into actionable ops work.
Extra practice notes
For "Measuring Tail Latency on a Local LLM API Without Fancy APM", triage feedback into slow, wrong, and confusing. Hold a short weekly pass. See MicroAI One and about. Size models with the VRAM table.
Percentiles
Track p50 and p95 weekly. If p95 explodes while p50 looks fine, you have intermittent contention—exactly what averages hide.
Cold start budget
Define an allowed cold-start TTFT. If warm policy cannot keep it, add warm-up jobs or keep a smaller always-on model.
Client-side timing
Measure from the client that users actually use, not only on-server clocks. Proxy buffering can add delay.
Capacity planning
When p95 degrades during lunch, you may need concurrency limits rather than a new model. Limits are a feature.
More executable detail
For "Measuring Tail Latency on a Local LLM API Without Fancy APM", create a single feedback channel where people report slow, wrong, or confusing behavior. Spend fifteen minutes a week tagging issues as hardware, prompt, permission, or expectation. Action after tagging beats endless model swapping. For orientation links you can cite MicroAI One and the about page.
Write decisions in a short runbook with dates. When something breaks, the first question is what changed; a dated note answers faster than memory. Include links to the exact model files and the ports you exposed. Keep the runbook next to the compose file or service unit so it cannot drift into a personal folder. (lp8-0)
Prefer boring defaults for anything shared. Power users can still override locally, but the shared alias should optimize for predictability. Predictability is what makes local AI feel like infrastructure instead of a hobby that resets every Monday. (lp8-1)
When you evaluate changes, change one variable at a time. If you swap the model, the quant, and the client on the same day, you will not know which change helped. Small experiments compound into a trustworthy stack. (lp8-2)
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.