What a million tokens really costs on your own GPUs
On this page
The boardThree things every row showsYour price, not oursAgainst the API priceEvery load, every setupHow every row is measuredGet your setup on the boardEvery row here is a model we served on rented GPUs while joules read the GPUs' own energy counters. Tokens per second and energy per token are measured, not quoted or estimated. The dollar figures use the price we paid for those GPUs that hour, and the formula below turns them into your price. A new row is added after every test.
The board
One row per setup: what a million output tokens cost with one user and at the busiest load we measured, and how fast the first token came back at that load.
| Model | GPUs | How it was served | GPU price per hour | 1 user | Busiest load measured | First token, busiest (p50 / p95) | GPU energy per token, busiest |
|---|---|---|---|---|---|---|---|
| Kimi K2 Thinking (1T parameters, 32B active) | 8× H200 | vLLM 0.32, 256-token answers | $37.37 | $76.93 | $1.61 at 256 users | not measured | 0.79 J |
| Kimi K2 Thinking (1T parameters, 32B active) | 8× H200 | vLLM 0.32, simulated mixed traffic: random arrivals, documents of mixed length | $27.38 | not measured | $1.28 at 1,024 users | 0.51 s / 0.86 s | 0.87 J |
| Qwen2.5-7B-Instruct | 1× H100 NVL | Hugging Face generate, 256-token answers | $2.34 | $7.03 | $0.12 at 256 users | not measured | 0.076 J |
The first token is what people notice. With 1,024 users at once, it came back in 0.51 s, and in 0.86 s for the slowest 1 in 20 requests. After it, the answer streams faster than anyone reads. Every new row measures it.
Next on the board: OpenAI's open model gpt-oss-120b on one H200.
Three things every row shows
- The users set the cost, not the GPU. The same 8 H200s cost $76.93 per million tokens with 1 user and $1.61 with 256, 48 times less, in the same hour. A GPU hour costs the same whether it serves 1 user or 1,000.
- Electricity is the small part. At every load measured, electricity was 1 to 3% of the cost of a token. The rest is the GPU hour.
- Busy is not the same as useful. With 1 user, the 8 H200s still drew 2,424 W and made only 135 tokens a second. Power and utilization look busy long before the GPUs are full.
Your price, not ours
Tokens per second is what was measured. To use your own GPU price:
cost per 1M tokens = price per hour × 1,000,000 ÷ (tokens per second × 3,600)
The price per hour is for all the GPUs together. Example: at $2.00 per H200 hour ($16 for 8), the 1,024-user row above would be $16 × 1,000,000 ÷ (5,960 × 3,600) = $0.75 per million tokens.
Against the API price
Kimi K2's API list price is $0.60 per million input tokens and $2.50 per million output tokens. Counting the prompts each test sent, that is $2.58 per million output tokens for the 256-token answers and $3.68 for the simulated mixed-traffic documents.
| Setup | Self-hosted GPU cost | API | Self-hosted against the API |
|---|---|---|---|
| 1 user | $76.93 | $2.58 | 30× more |
| 256 users | $1.61 | $2.58 | 37% less |
| 1,024 simulated users, mixed traffic | $1.28 | $3.68 | 65% less |
Self-hosting wins only while the GPUs stay busy. Quiet hours cost the same as busy ones, so the hours at each load decide the real cost. These are GPU rental costs only, not staff or the rest of a server.
Every load, every setup
Kimi K2 Thinking on 8× H200, 256-token answers
vLLM 0.32.0 with tensor parallel 8 on a Vast machine in Georgia at $37.37 an hour. Every answer was exactly 256 tokens to a 34-token prompt. Three rounds per load, medians reported. Idle: 648 W for the 8 GPUs with no model, 1,016 W with the model loaded.
| Users at once | Output tokens/s | GPU energy per token | Average power, 8 GPUs | GPU cost per 1M tokens | Electricity per 1M tokens |
|---|---|---|---|---|---|
| 1 | 134.9 | 17.96 J | 2,424 W | $76.93 | $0.75 |
| 8 | 716.1 | 5.13 J | 3,676 W | $14.50 | $0.21 |
| 32 | 2,092.8 | 1.79 J | 3,752 W | $4.96 | $0.07 |
| 128 | 4,674.3 | 1.01 J | 4,727 W | $2.22 | $0.04 |
| 256 | 6,432.3 | 0.79 J | 5,076 W | $1.61 | $0.03 |
Kimi K2 Thinking on 8× H200, simulated mixed traffic
The same model and server on a different Vast machine, in Japan at $27.38 an hour. A load generator played the users. Each request was a synthetic document of 1 to 12 paragraphs plus a question, about 580 prompt tokens on average, asking for a fixed answer of 64 to 512 tokens. Requests arrived at random, 2 per user. One measurement per load, and no request failed. Idle: 625 W with no model, 979 W with the model loaded.
| Users at once | Output tokens/s | GPU energy per output token | Average power, 8 GPUs | GPU cost per 1M output tokens | First token, p50 / p95 | Whole answer, p50 / p95 |
|---|---|---|---|---|---|---|
| 256 | 2,618.8 | 1.77 J | 4,622 W | $2.90 | 0.27 s / 0.43 s | 16.7 s / 32.1 s |
| 512 | 4,182.6 | 1.19 J | 4,978 W | $1.82 | 0.37 s / 0.47 s | 24.6 s / 45.0 s |
| 1,024 | 5,960.4 | 0.87 J | 5,189 W | $1.28 | 0.51 s / 0.86 s | 36.4 s / 66.2 s |
Throughput was still rising at 1,024 users, so this machine can take more.
Qwen2.5-7B-Instruct on 1× H100 NVL
Hugging Face generate in bf16 with fixed batches, at $2.34 an hour. Every answer was exactly 256 tokens. Three rounds per load, medians reported. Idle with the model loaded: 89 W. The full write-up: What a GPU hour buys.
| Users at once | Output tokens/s | GPU energy per token | Average power | GPU cost per 1M tokens | Electricity per 1M tokens |
|---|---|---|---|---|---|
| 1 | 92.3 | 2.634 J | 243 W | $7.03 | $0.11 |
| 8 | 556.0 | 0.419 J | 233 W | $1.17 | $0.02 |
| 32 | 1,865.7 | 0.147 J | 275 W | $0.35 | $0.006 |
| 128 | 4,413.0 | 0.086 J | 378 W | $0.15 | $0.004 |
| 256 | 5,231.2 | 0.076 J | 395 W | $0.12 | $0.003 |
How every row is measured
- Energy: joules reads each GPU's own energy counter through NVML for the whole test, one phase per load. GPU board energy only, not the CPU, memory or the power at the wall.
- Idle first: each machine is measured doing nothing, with and without the model loaded, before any load.
- Warm-up: each load runs once unmeasured before it is measured.
- Known work: answer lengths are fixed, so every load does a known number of tokens.
- Prices: the GPU price is what the rental cost that hour. Electricity is the measured energy at $0.15 per kWh.
- Not comparable across rows: different models, GPUs, servers, prompts and prices. Compare loads within one setup, and use the formula above for your own price.
Get your setup on the board
Running a model on your own GPUs? Measure it with joules, following the LLM energy guide. Or send us your model, GPUs and traffic and we will measure it and send you the full report.