How to measure an LLM's energy per token
On this page
With Ollama, in one commandWith vLLM, llama.cpp, LM Studio or TGICompare two setups fairlyFind the cheapest power limitWhat to reportNextEnergy per token is the fairest way to compare models, quantisations, batch sizes and servers on your own hardware: it counts the energy the GPU actually used and divides it by the tokens it actually produced.
With Ollama, in one command
pip install landauer-gap
joules bench ollama:llama3.1:8bjoules starts Ollama if needed, downloads the model the first time, sends one warm-up request that is not counted (so loading the model is left out), then measures the GPU (and CPU and DRAM) energy over five measured requests (--runs changes it) and divides by the output tokens. It prints joules per token, tokens per second and the average power.
With vLLM, llama.cpp, LM Studio or TGI
Any server with an OpenAI-compatible API works:
joules bench openai:my-model --url http://localhost:8000/v1Compare two setups fairly
Energy per token changes with batch size, quantisation, context length and the GPU's power limit. To compare two setups, change one thing at a time and alternate the runs:
joules compare "python serve_test.py --batch 1" "python serve_test.py --batch 8"joules compare runs a warm-up, alternates the commands, and reports the median energy and time of each with a p-value, and warns when one side ran hotter or at a different clock.
Find the cheapest power limit
Most GPUs run near their maximum power by default, and the last stretch of power often buys little speed. joules tune tries several power limits and reports the one with the cheapest token that is at most 10% slower:
sudo joules tune ollama:llama3.1:8bIt puts the limit back afterwards.
What to report
- Energy per output token, and tokens per second, side by side
- The GPU, its power limit and whether idle was subtracted
- The prompt length and output length, since both change the result
- The number of runs and their spread, not one run