One GPU, two jobs: splitting the energy by SM time, not by memory

Measurement · 24 October 2026 · 5 min read

On a shared GPU server, the energy counter tells you what the whole card used. It does not tell you which job used it. Most tools that try to split it look at the memory each process holds, because the driver reports memory for every process. But memory held is not work done. We tested a better split on an NVIDIA H100 NVL: two jobs on one GPU, one that computes all the time and holds little, one that holds 10 GB and barely computes.

The result: the job that computed got all 35,611 J of the GPU's energy, and the job that only held memory got 0 J, split by each process's SM time as the driver reports it. A split by memory held would have charged most of the energy to the job that did almost nothing.

Why the split matters

When several teams, users or jobs share a GPU, the energy on the meter has to be divided among them before it can mean anything: for showing each team its cost, for finding the job that wastes a card, or for an energy report that holds up. The division is only as good as what it is based on.

• Memory held is easy to read and often wrong. A process that loads a large model and waits holds a lot of memory and does little work. A process doing heavy math on small data holds little and does most of the work.
• GPU utilisation (the number in nvidia-smi) is for the whole card, not per process, and it counts a kernel as running even when it uses a fraction of the chip.
• SM time is the time each process actually kept the GPU's streaming multiprocessors, its compute units, busy. The driver reports it per process (NVML's per-process utilisation on Maxwell and newer GPUs, or its accounting mode). It follows the work.

The test

Two Python programs on GPU 0 of an H100 NVL, started together and run for 90 seconds each:

# heavy.py: computes all the time, holds little memory
import time, torch
x = torch.randn(8192, 8192, device="cuda:0", dtype=torch.float16)
end = time.time() + 90
while time.time() < end:
    for _ in range(20):
        y = x @ x
    torch.cuda.synchronize()

# light.py: holds about 10 GB, computes only now and then
import time, torch
hold = torch.empty(10 * 1024**3, dtype=torch.uint8, device="cuda:0")
x = torch.randn(1024, 1024, device="cuda:0", dtype=torch.float16)
end = time.time() + 90
while time.time() < end:
    y = x @ x
    torch.cuda.synchronize()
    time.sleep(0.2)

joules agent watched the GPU the whole time, reading the energy counter and the per-process SM time once a second:

joules agent --gpus 0 --interval 1

The test checked that the process IDs matched the driver's list of processes on the GPU (they did). The agent split each second's energy between the processes by their SM time.

The result

light.py's work, a few microseconds of compute every fifth of a second, rounds to 0% of the SM time, so its share of the energy is 0. That is the right answer: it held the card's memory but did not use its compute. A split by memory would have given it most of the energy, because it held many times more.

Where the split is exact, and where it is close

joules agent says, for every job, which split it got, so the number always comes with how far to trust it:

SM time is close rather than exact because a process's memory traffic, and its share of the card's idle power, are not in its SM time. Each job also reports its coverage, the share of its run the agent actually sampled; below 95% the job is flagged. For an exact number per job, give each job whole GPUs and keep coverage at 100%.

How to use it

1. On a shared server, run joules agent on each GPU node. It finds the jobs on the GPUs (Slurm jobs, Kubernetes pods, containers or plain processes) and keeps a running list of their energy at http://127.0.0.1:9877/jobs, with Prometheus metrics as well.
2. Turn on accounting mode where you can (nvidia-smi -am 1, as root). The driver then keeps per-process totals, not only samples, and the agent uses them.
3. Read the split next to the number. A job's attribution says how its energy was divided, and attribution_j how many joules came from each method. A report that shows a job's energy should show its split too.
4. Look for jobs like light.py. A process that holds a GPU's memory and does almost no work keeps the GPU from anyone else. Its energy is small, but its cost is the whole GPU it blocks: the same lesson as what a GPU hour buys.

Method notes

• One test, on one H100 NVL, in one session, with joules 0.8.0. In 0.8.0 light.py's split reads "CPU time", because it received no GPU energy; the next release labels it "SM time", with 0 J. The energy split itself is the same in both.
• The agent's list also showed the machine's other processes (a notebook server, a web server and others) with 0 J of GPU energy: they were not on the GPU.
• The driver's per-process SM time is a sampled measure. A job that runs only very short kernels, like light.py, can read 0% even though it did a little work. Its true share here is a tiny fraction of a percent.
• GPU board energy only, not the CPU, memory or the power at the wall.