Speed claims for local LLMs are everywhere, and they are rarely comparable. One person reports tokens per second with a tiny prompt; another measures with a long document; a third uses a different quantization. If you want to know how your own setup performs, or compare two configurations fairly, a simple, consistent method matters more than any number you read online.
Separate the two phases
Inference has two distinct phases:
- Prompt processing (prefill): the model reads your input. Speed here is measured in tokens per second of input and depends heavily on compute.
- Generation (decode): the model produces new tokens one by one. This speed depends mostly on memory bandwidth.
Report both. A machine can be fast at one and slow at the other. Macs and many integrated GPUs, for example, often show a bigger gap between the two than high-end discrete GPUs.
Fix the variables
For any comparison, keep these the same:
- Exact model file and quantization.
- Context length setting and actual prompt length.
- Number of tokens generated.
- Runtime and version, with the same settings such as GPU layers, flash attention and cache type.
- Sampling settings; use a fixed seed where possible.
- Thermal state; run a warm-up first, then measure several runs.
Use built-in tools
llama.cpp ships a benchmarking tool that reports prefill and generation speeds for chosen prompt and output lengths. Ollama prints timing statistics when run in verbose mode. These are better than stopwatch measurements because they exclude load time.
Record the environment
Write down GPU, VRAM, driver version, CPU, RAM, operating system, power limit and whether the machine was also driving a display. Small differences in these explain many disagreements between users.
Test quality as well as speed
Faster is not better if answers degrade. Keep a small set of prompts from your real work and compare outputs between configurations, especially when changing quantization.
Be honest about limits
A single run is noise. Report the median of several runs, note anything unusual, and avoid generalizing from one model to all models.
Share your results
Consistent, well-documented measurements are valuable to the community, because hardware combinations are too numerous for any single reviewer to test.
References
- MicroAI One DIY lab, hands-on local AI projects: https://microaione.com/lab/
- MicroAI One testing methodology: https://microaione.com/methodology/
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.