fp16 vs fp32 on a Tesla T4: 81 to 84% less GPU energy

Last updated 17 October 2026

How much GPU energy does fp16 save over fp32? Spec sheets give peak FLOPS and TDP, not joules per job, so we measured it on a Tesla T4: the same matrix multiply job in fp16 and in fp32, five runs each, under three different setups.

The result: fp16 used 81 to 84% less GPU energy and finished 4.3 to 4.8× faster. All three setups agree, and the spread between runs was as low as 0.1%.

To run it yourself in a few clicks, open the Kaggle notebook.

The job

An 8192 × 8192 matrix multiply, repeated 100 times, in PyTorch on one Tesla T4 (a Kaggle notebook with the GPU T4 x2 accelerator, GPU 0 only):

import sys, torch
dtype = {"fp16": torch.float16, "fp32": torch.float32}[sys.argv[1]]
a = torch.randn(8192, 8192, device="cuda", dtype=dtype)
for _ in range(100):
    b = a @ a
torch.cuda.synchronize()

Each run is a whole process (Python start-up, CUDA set-up and the 100 multiplies), so the numbers are energy per job, the way a real script uses the GPU.

How it was measured

• GPU energy comes from the GPU's own energy counter, read through NVML. It is the GPU board only, not the CPU and not power at the wall.
• Alternating runs: fp16, fp32, fp16, fp32 and so on, so any drift in temperature or clocks hits both sides equally.
• Five runs each, with the median reported, and the interquartile range (IQR) as the spread.
• A Mann-Whitney test between the two sides. With five runs each, p = 0.008 is the smallest p-value possible: every fp16 run beat every fp32 run.
• Three setups, to check that the result does not depend on how warm the GPU is: no cooling between runs; every run starting at 65 °C or cooler; and 65 °C plus three warm-up runs per side before the measured ones.

All of it is one command, joules compare, from the open-source joules CLI.

Results

p = 0.008 in all three setups.

Why the gap is so large

• Tensor cores. The T4 is a Turing GPU. Its tensor cores run fp16 matrix multiplies, but Turing has no TF32, so fp32 runs on the ordinary CUDA cores. fp16 finishes the same job 4.3 to 4.8× sooner.
• Energy is power × time. fp16 also drew less power on average (49 to 56 W against 62 to 65 W), but most of the saving comes from the shorter run.
• The 70 W cap. The T4 is limited to 70 W. fp32 ran throttled (by the power or thermal limit) 92% of the time, fp16 about 60%. So on this card the comparison is partly a comparison of how each precision copes with the cap.

What this does and does not tell you

• It is one GPU model and one kind of work: large, dense matrix multiplies. Real models also spend time on memory-bound layers, data loading and communication, where the gap is smaller.
• On an A100, H100 or L4, fp32 matrix multiplies can use TF32 on the tensor cores, so expect a much smaller gap between fp32 and fp16 there. Measure it rather than assume it.
• fp16 trades precision for speed. Mixed precision (fp16 or bf16 with fp32 accumulation) is the usual way to get the energy saving and keep training stable.

Run it yourself

The quickest way: open the Kaggle notebook, click Copy & Edit, set the accelerator to GPU T4 x2 and click Run All. It installs joules, runs the comparison and prints the table.

Or, on any machine with an NVIDIA GPU, save the script above as bench.py, then:

pip install -U landauer-gap
joules compare --names fp16,fp32 --baseline fp32 \
  --runs 5 --warmup 3 --cool-to 65 --gpus 0 \
  "python3 bench.py fp16" "python3 bench.py fp32"

It prints the table above for your GPU: median GPU energy, time, average power, clocks, throttling, temperature and the p-value, with a warning when the two sides did not run under the same conditions. Add --markdown results.md to save the table. Nothing leaves the machine unless you add --track.

The joules guide lists every command. To see the same numbers for every job on a GPU fleet, with cost per job and the GPUs that sit idle, try WattOps.