A speed figure can change a lot with even small differences in conditions. This guide shows how to measure under similar conditions with llama.cpp, MLX and Ollama, and which values to write down. It cannot guarantee identical results on every setup, and any numbers shown are examples.
Command options may differ between versions. If something does not work, check the options of your version with --help. Command text is not translated, and anything written as MODEL should be replaced with your own model file or name.
Write these down first
If you note the values below along with the speed, you can copy them straight into the submission form, and comparing with other devices gets easier.
Device name, chip or GPU, memory size (GB)
Model name and quantization (e.g. Q4_K_M, 4bit)
Engine name and version or build number
Context length (tokens) and number of concurrent requests
Generation speed (tok/s), prompt processing speed (tok/s), number of tokens generated
If possible, peak memory (GB), power (W) and the command you used
Tips for keeping conditions fixed
Fix the context lengthPick one context length such as 8192 and note it. Speed often drops as context grows, so values at different lengths are hard to compare.
Keep concurrent requests at 1Send one request at a time and make sure no other program is using the model. With more concurrent requests, per-request speed and total throughput diverge.
Match the quantizationTo compare with other devices, use the same quantization of the same model. Similar names can still differ by who made the file, so note where you downloaded it.
Fix the power stateOn a laptop, plug in the charger and turn off low-power or battery-saver modes. Noting whether it was plugged in helps others interpret the result.
Measure after temperatures settleSkip the first run or two, repeat several times and use results once the values settle. Keep the pauses between runs consistent, neither very long nor very short.
Repeat with the same promptFix the prompt and output length, repeat 3 to 5 times and use the median. A single run is heavily affected by chance.
Close other workClose anything that uses GPU or memory, such as browser tabs, video playback or other models. If memory runs short and spills to disk, speed drops sharply.
llama.cpp (llama-bench)
llama-bench, included with llama.cpp, measures prompt processing and generation separately several times and shows the average. It is the best place to start.
-p is the number of prompt tokens to read, -n the number of tokens to generate, -r the number of repetitions and -ngl the number of layers on the GPU. In the result table, the pp512 row is prompt processing speed and the tg128 row is generation speed, in t/s (= tok/s).
Values to record tg128 t/s → generation speed, pp512 t/s → prompt processing speed. The ± value shows the spread between repetitions. For tokens generated, enter the -n value (128).
By default this shows speed with a nearly empty context. To see speed at a longer context, the -d option sets how much context is already filled, but this option may vary by version, so check --help. In that case, enter the -d value as the context length.
Use this when you want to measure through an actual server. Start the server with context 8192 and 1 slot, then send one request with the prompt cache disabled. Read the values from the timings field of the response.
Example commands · llama.cpp
llama-server -m ./models/MODEL.gguf -c 8192 -ngl 99 --parallel 1
curl http://127.0.0.1:8080/completion -d '{"prompt":"Explain how a transformer language model works.","n_predict":256,"temperature":0,"cache_prompt":false}' | jq '.timings'
Values to record timings predicted_per_second → generation speed, prompt_per_second → prompt processing speed, predicted_n → tokens generated. For context length enter the -c value, and 1 for concurrency.
MLX (mlx_lm)
The mlx-lm package for Apple silicon Macs. When a generation command finishes, it prints prompt speed, generation speed and peak memory together.
--max-tokens fixes the output length and --temp 0 reduces randomness. Read the 'Prompt: … tokens-per-sec' and 'Generation: … tokens-per-sec' lines printed at the end.
Example commands · MLX
pip install -U mlx-lm
python -m mlx_lm.generate --model mlx-community/MODEL --prompt "Explain how a transformer language model works." --max-tokens 256 --temp 0
Values to record Generation line tokens-per-sec → generation speed, Prompt line tokens-per-sec → prompt processing speed, the number before tokens on the Generation line → tokens generated, Peak memory → peak memory use (GB). The first run may be slow because the model is loading, so use results from the second run on.
Recent mlx-lm versions include mlx_lm.benchmark, which runs with a set prompt length (-p) and generation length (-g) several times (-n). After a warm-up it prints each trial and an 'Averages' line. Older versions may not have it, so check that it exists in your installed version.
Values to record Averages line prompt_tps → prompt processing speed, generation_tps → generation speed, peak_memory → peak memory use. For tokens generated enter the -g value, and keep the -p value you used in the command field together with the command. The output format may differ by version.
Ollama
With the --verbose option Ollama prints speed statistics after the answer, and its API response includes token counts and durations.
Running once with --verbose prints 'prompt eval rate' and 'eval rate' in tokens/s at the end. It is simple, but it is hard to fix context and output length, so use it as a rough check.
Example commands · Ollama
ollama run MODEL:TAG --verbose "Explain how a transformer language model works."
Values to record eval rate → generation speed, prompt eval rate → prompt processing speed, eval count → tokens generated. For models with thinking turned on, check whether those tokens are included.
Through the API you can fix the context length (num_ctx), output length (num_predict) and randomness (temperature) in one request. The generation speed is calculated from eval_count and eval_duration (nanoseconds) in the response. jq is required.
Example commands · Ollama
OLLAMA_NUM_PARALLEL=1 ollama serve
curl -s http://localhost:11434/api/generate -d '{"model":"MODEL:TAG","prompt":"Explain how a transformer language model works.","stream":false,"options":{"num_ctx":8192,"num_predict":256,"temperature":0}}' | jq '{gen_tps: (.eval_count / .eval_duration * 1e9), prompt_tps: (.prompt_eval_count / .prompt_eval_duration * 1e9), eval_count, prompt_eval_count}'
Values to record Printed gen_tps → generation speed, prompt_tps → prompt processing speed, eval_count → tokens generated, num_ctx → context length. If prompt_eval_count is very small, a cache from an earlier request was used and the prompt speed is not reliable.
To keep concurrency at 1, start the server with the OLLAMA_NUM_PARALLEL=1 environment variable and make sure no other request arrives at the same time. The default may vary by version and environment.
Common mistakes
Not deciding whether thinking tokens countModels that output their reasoning first may count those tokens as generated tokens. The speed itself may be similar, but the time to the final answer differs a lot, so note in the command field whether they were included.
Prompt cache was onIf the same prompt is repeated, the engine may reuse what it stored instead of reading it again, which makes prompt processing speed look unrealistically high. Turn the cache off or use a different prompt each time.
Including the first request in the averageThe first request is slow because the model is being loaded or the GPU is waking up. Run once beforehand as a warm-up and use the values from after that.
Confusing tokens with charactersA token is not a character or a word. The same sentence has a different token count per model, so use the speed computed from the token count your engine reports, not a character count you did yourself.
Comparing different quantizations or contextsValues under different conditions, such as Q4 vs Q8 or context 4096 vs 32768, cannot be compared on the same line. To compare, change only one thing and keep the rest the same.
Mixing up what concurrency meansIf you measured with 2 or more concurrent requests, state in the command field whether the speed you entered is per request or the total. Without that it cannot be compared.
Using a heat-throttled value as representativeSome devices slow down from heat when run for a long time. For a representative value use the median of repeated runs, and note it if you see speed steadily dropping.
Done measuring?
Capture the result screen and submit it along with the values you noted above. After review, it is published under the 'User-submitted · evidence verified' grade.