Self-hosting a 1-trillion-parameter model: when 8 H200s beat the API
On this page
How it was measuredTest 1: the same machine, 1 to 256 usersTest 2: 256 to 1,024 simulated users, mixed trafficWhat the numbers showAgainst the API priceYour price, not oursMeasure your ownWe served Kimi K2 Thinking, a model with 1 trillion parameters (32 billion active), on 8 rented NVIDIA H200s with vLLM, and measured what a million tokens cost at every load, with the GPUs' own energy counters running the whole time.
The result: the same 8 GPUs, in the same hour, cost $76.93 per million tokens with 1 user and $1.61 with 256 users at once, 48 times less. The API for the same model costs $2.58 for that work. So self-hosting was 30 times more expensive than the API when quiet and 37% cheaper when busy. In a second test, with 1,024 simulated users sending documents of mixed length, it reached $1.28 against the API's $3.68, and the first token still came back in 0.51 s.
The GPU hour costs the same whether it serves 1 user or 1,000. What you pay per token depends on how busy you keep it.
How it was measured
- Model: Kimi K2 Thinking (1T parameters, 32B active), served by vLLM 0.32.0 with tensor parallel 8.
- GPUs: 8× NVIDIA H200, rented on Vast. Test 1 ran on a machine in Georgia at $37.37 an hour; test 2 on a machine in Japan at $27.38 an hour.
- Energy: joules read each GPU's own energy counter through NVML, one phase per load. GPU board energy only, not the CPU or the power at the wall.
- Idle first: each machine was measured doing nothing, with and without the model loaded, before any load.
- Warm-up: each load ran once unmeasured before it was measured.
- Cost: the GPU price is what the rental cost that hour. Cost per million tokens is that price divided by the tokens served in an hour. Electricity is the measured energy at $0.15 per kWh.
Test 1: the same machine, 1 to 256 users
Every answer was exactly 256 tokens to a 34-token prompt, so each load did a known amount of work. Three rounds per load, medians reported.
| Users at once | Output tokens/s | GPU energy per token | Average power, 8 GPUs | GPU cost per 1M tokens | Electricity per 1M tokens |
|---|---|---|---|---|---|
| 1 | 134.9 | 17.96 J | 2,424 W | $76.93 | $0.75 |
| 8 | 716.1 | 5.13 J | 3,676 W | $14.50 | $0.21 |
| 32 | 2,092.8 | 1.79 J | 3,752 W | $4.96 | $0.07 |
| 128 | 4,674.3 | 1.01 J | 4,727 W | $2.22 | $0.04 |
| 256 | 6,432.3 | 0.79 J | 5,076 W | $1.61 | $0.03 |
Idle: 648 W for the 8 GPUs with no model, 1,016 W with the model loaded.
Test 2: 256 to 1,024 simulated users, mixed traffic
The same model and server on a different machine. A load generator played the users. Each request was a synthetic document of 1 to 12 paragraphs plus a question, about 580 prompt tokens on average, asking for an answer of 64 to 512 tokens. Requests arrived at random, 2 per user. One measurement per load, and no request failed.
| Users at once | Output tokens/s | GPU energy per output token | GPU cost per 1M output tokens | First token, p50 / p95 | Whole answer, p50 / p95 |
|---|---|---|---|---|---|
| 256 | 2,618.8 | 1.77 J | $2.90 | 0.27 s / 0.43 s | 16.7 s / 32.1 s |
| 512 | 4,182.6 | 1.19 J | $1.82 | 0.37 s / 0.47 s | 24.6 s / 45.0 s |
| 1,024 | 5,960.4 | 0.87 J | $1.28 | 0.51 s / 0.86 s | 36.4 s / 66.2 s |
Throughput was still rising at 1,024 users, so this machine could take more.
More users made tokens cheaper and each answer slower. The first token stayed under a second, which is what people notice. The wait moved to the stream: at 1,024 users each answer streamed at about 8 tokens a second, close to reading speed.
What the numbers show
1. The users set the cost, not the GPU. From 1 to 256 users, tokens per second rose 48 times on the same machine, and the cost per token fell by the same 48 times. The hourly price never changed.
2. Busy is not the same as useful. With 1 user, the 8 H200s drew 2,424 W, well over twice their idle power with the model loaded, and made only 135 tokens a second. Power draw and utilization look busy long before the GPUs are full.
3. Electricity is the small part. Electricity was 1 to 2% of the cost of a token in test 1. Energy per token fell 23 times from 1 user to 256, but the bill is the GPU hour.
4. Self-hosting has a break-even, and it is an average. For 8 H200s at $37.37 an hour to beat the API's $2.58, they must serve about 4,000 tokens a second averaged over every hour, quiet hours included. In test 1 that load lies between 32 and 128 users at once, about 63% of the busiest throughput we measured. In test 2, at $27.38 an hour against $3.68, the bar is about 2,070 tokens a second, 35% of the throughput at 1,024 users. A cluster that is busy at noon and empty at night pays for the night too.
Against the API price
Kimi K2's API list price is $0.60 per million input tokens and $2.50 per million output tokens. Counting the prompts each test sent, that is $2.58 per million output tokens for test 1 and $3.68 for test 2's longer documents.
| Setup | Self-hosted GPU cost | API | Self-hosted against the API |
|---|---|---|---|
| 1 user | $76.93 | $2.58 | 30× more |
| 256 users | $1.61 | $2.58 | 37% less |
| 1,024 simulated users, mixed traffic | $1.28 | $3.68 | 65% less |
These are GPU rental costs only, not staff, storage or the rest of a server.
Your price, not ours
Tokens per second is what was measured. To use your own GPU price:
cost per 1M tokens = price per hour × 1,000,000 ÷ (tokens per second × 3,600)
The price per hour is for all the GPUs together. At $2.00 per H200 hour ($16 for 8), the 1,024-user load would cost $16 × 1,000,000 ÷ (5,960 × 3,600) = $0.75 per million tokens.
Measure your own
The question that decides self-hosting is not "how fast is the model" but "how busy are my GPUs, hour by hour". joules measures the energy of any run from the GPUs' own counters, and from joules 0.8.1, joules agent on a vLLM or SGLang server sends tokens per second and cost per million tokens to WattOps Live every few seconds, beside the API price you set. The LLM energy guide shows how to run a test like this one.
Every result we measure goes on the cost per token board, one row per test. Next: OpenAI's gpt-oss-120b on one H200.