The TPS Method
How we measure local LLM speed
A reproducible approach to measuring local LLM inference speed across the hardware × software matrix real people actually run.
What we track
Five metrics, measured together
Five metrics, measured together. Six dimensions, varied independently.
- Generation throughput, measured over the sustained decode phase after the prompt is processed, excluding the first token.
- Wall-clock latency from submitting the prompt to the first streamed token — the delay a user actually feels before text starts to appear.
- The highest whole-package power draw sampled during the run, capturing the real energy cost of sustained generation — not just idle or nameplate figures.
- How hard the run pushed against available VRAM / unified memory, reported against the 40% and 80% pressure thresholds where throughput starts to degrade.
- How fast the model ingests input context, measured over the prefill phase before the first token is generated.
Six dimensions, varied independently
The same five metrics are captured across the whole matrix that determines local inference speed. Change one dimension, hold the rest, and the leaderboard tells you what that change actually cost.
- OS macOS · Windows · Linux
- Hardware Apple Silicon · x86 + dGPU · Strix Halo · …
- Hosting tool Ollama · LM Studio · llama.cpp · …
- Inference engine llama.cpp · MLX · vLLM · …
- GPU backend Metal · CUDA · ROCm · Vulkan
- Model Llama · Qwen · Gemma · quant & size
How credibility works
Every number carries a trust tier
A community leaderboard is only as good as its worst unmarked number. Every submission carries a trust tier so you always know how much to believe it.
- UnverifiedDefault tier. No auto-parseable evidence attached yet — take the number as a claim, not a fact.
- CommunityAuto-promoted after net +10 community upvotes with no open flags. The crowd vouches for it.
- Tool outputA recognised benchmark tool's raw output file was attached and parsed — the numbers came from the tool, not typed by hand.
- TPS ApprovedA moderator reproduced or verified the run. The highest trust tier on the board.
Hybrid demotion — fast signals, human confirmation
- Unverified A fast community signal applies a soft warning immediately — but never silently rewrites the earned tier.
- A moderator confirms every tier change. Community votes can raise a flag; only a human moves a result down to Community suspect or hides it.
- Mod notes are public. When a result is demoted, the reason is visible on the submission.
- Appeals are visible. The submitter can respond, and the exchange stays on the record.
How to reproduce
Run every number yourself
Every metric on this page comes from commands you can run yourself. This section doubles as the how-to for submitting your own benchmark.
The standard run
- Prompt: the standard prompt (about 500 tokens; the exact count depends on the model's tokenizer) or the tool's 512-token prefill.
- Output: 128 tokens where the tool can cap it. Thinking off in Ollama and MLX. llama-bench and llama-benchy fix the token counts, so thinking doesn't change their speeds.
- Runs: llama-bench and llama-benchy average 5 runs after their own warm-up. Ollama and MLX: one warm-up with a different prompt, then one recorded run.
- Conditions: on mains power, with no other heavy programs running. Put the server settings (flash attention, KV-cache type, context length) in the notes.
- Honest runs on the same machine still differ by up to about 10% in generation speed.
Per-tool commands
llama-bench -m ./models/your-model.gguf -p 512 -n 128 -r 5 --output json > result.jsonUses a synthetic 512-token prompt, warms up itself and averages 5 runs. Attach result.json.
llama-benchy --base-url http://localhost:8080/v1 --model your-model --pp 512 --tg 128 --runs 5 --format json --save-result benchy.jsonBenchmarks a running OpenAI-compatible server, not a model file: start llama-server, Ollama or vLLM first and point --base-url at it. --model is the name the server knows the model by. It warms up itself and averages the 5 runs. Attach benchy.json.
ollama stop your-model
ollama run your-model --think=false "hi" </dev/null
ollama run your-model --verbose --think=false "$(cat tps-prompt.txt)" </dev/null 2> result.txtUnload, warm up with a different short prompt, then run the standard prompt once and attach result.txt. Never run the standard prompt twice in a row: Ollama caches it and the second prompt rate is far too low. Ollama can't cap the answer at 128 tokens, so its generation rate varies by about 10% between runs. Leave out --think=false if the model has no thinking mode.
mlx_lm.generate --model mlx-community/your-model --prompt "hi" --max-tokens 16
mlx_lm.generate --model mlx-community/your-model --prompt "$(cat tps-prompt.txt)" --max-tokens 128 --chat-template-config '{"enable_thinking":false}' --verbose True > result.txtWarm up once, then run the standard prompt and attach result.txt. Not yet tested by us on a Mac.
Per-OS power & memory capture
- macOS
sudo powermetrics --samplers cpu_power,gpu_powerpackage power & memory pressure - Windows
Intel PCM · nvidia-smi --query-gpu=power.draw --format=csvNVIDIA GPU power, one reading: run it while the benchmark runs - Linux
sudo perf stat -e power/energy-pkg/ …RAPL energy via perf — CPU package only, not the GPU - AMD (ROCm)
rocm-smi --showpowerGPU package power, no sudo needed
Submission API for CI pipelines
Limitations & credits
What this doesn't measure
The TPS Method measures speed, not quality. It says nothing about how good a model's answers are, how well it holds up over long-context coherence, or how it scales under batched, concurrent load. A fast wrong answer is still wrong — read these numbers as one axis of a bigger decision.
Credits & inspiration
The TPS Method is inspired by systematic local-LLM benchmarking work done in public by others.