Comparing a runtime with a baseline

Guide · Last updated 21 October 2026 · 13 min read

This guide is for teams who want to show that one way of doing the same work uses less energy than another: a new runtime against the standard one, a sparse kernel against a dense one, incremental updates against a full recompute. It covers what to measure, how to make the comparison fair, and the three ways joules can measure it: whole commands (joules compare), phases inside a long-running program (the agent's phase marks), and phases inside a Python program (joules.Session).

Everything here runs on your machines. Nothing is sent anywhere unless you add --track.

What to report

Per completed unit of work (one solve, one incremental update, one batch):

Report the median of several runs and its spread. joules gives the interquartile range as a share of the median, and a Mann-Whitney p-value: the chance of seeing a difference at least this large if the two were really the same. With 5 runs each the smallest possible p is 0.008; with 3 runs it is 0.1, which can never be convincing.

Whole commands: joules compare

pipx install landauer-gap          # or: curl -LsSf https://wattops.dev/install.sh | sh
joules doctor                      # what it can measure on this machine

joules compare --names hst,baseline --runs 7 --warmup 1 --subtract-idle 10 \
  --markdown results.md --csv runs.csv --out compare.json \
  "./bench --runtime hst --input big.mtx" "./bench --runtime baseline --input big.mtx"

What it does:

1. Measures the idle machine for 10 seconds (--subtract-idle), so every run also has energy above idle.
2. Runs one unmeasured round of each command (--warmup), so caches, JIT compilation and clocks settle.
3. Runs the commands interleaved, A B A B (or A B B A with --order abba, which also cancels a steady drift such as a room warming up), so drift and heat hit both sides equally.
4. Stops if a command fails: a failed run is not a fast run.
5. Prints, per command, the median energy, time, average power and energy × time with their spread; the GPU's clock as a share of its maximum, the share of time it was throttled, and the hottest temperature; and for each command against the baseline (--baseline, default the first) the change with its p-value.
6. Warns when the conditions were not the same: the GPU throttled much more under one command, ran at a different clock, or the runs started at very different temperatures.

--markdown appends a table you can paste into a paper, a README or a pull request. --csv writes one row per measured run (energy per device, GPU readings, start temperature) for your own analysis. --json and --out give everything.

Quote each command. --shell runs them through the shell when they need pipes, && or environment variables. --quiet hides their own output.

Controlling the GPU

A runtime can look faster or more frugal simply because the card boosted or throttled differently under it. Run the comparison twice:

• Default settings, the result your users will see.
• Controlled, where the gain cannot come from the card. Either fix the power limit with --limits, or lock the SM clock with --lock-clocks.

Report both. Locked clocks give a clean comparison of the work itself: neither side gets more boost, and run-to-run spread from the card changing its clock goes away. Real jobs run boosted, so the default-settings result is the one your users will see.

sudo joules compare --names hst,baseline --runs 5 --warmup 3 --lock-clocks base --gpus 0 \
  "./bench --runtime hst" "./bench --runtime baseline"

--lock-clocks MHZ locks the SM clock of every GPU in --gpus (nvidia-smi -i N -lgc MHZ,MHZ) before the first run, warm-ups included. --lock-clocks base uses the card's base clock as the driver reports it (its default applications clock), which the card can hold under its power cap. Afterwards joules always unlocks them (nvidia-smi -i N -rgc), also after a failed run or Ctrl-C. It needs root (an administrator terminal on Windows). If a GPU refuses (no permission, MIG mode, a driver or cloud host that does not allow it), joules stops before running anything and says why in one line. Containers and cloud notebooks usually refuse even root: a Kaggle T4 notebook did in October 2026, so lock clocks on a machine where you control the GPU (your own, or a cloud VM). joules doctor says whether locking is permitted on the machine without changing any clock.

The result says the clock it was locked at: in the header of the table, in --markdown and --csv, and in the label of each run sent with --track (hst @ 1590 MHz locked). A card can still drop below a locked clock when it hits its power or thermal cap; joules then says so, naming the side and its average clock, because that cap is the only thing besides the work that can still make the two sides differ. Lock lower (base) to hold it. --lock-clocks works with --limits (clocks locked first, then the power limits stepped) and with --cool-to.

--limits repeats the whole comparison at several power limits, as a percentage of the card's maximum, and puts the original limit back at the end, also after an error or Ctrl-C. It needs permission to set the limit, so run it with sudo:

sudo joules compare --names hst,baseline --runs 5 --limits 100,85,70,55 --cool-to 50 \
  "./bench --runtime hst" "./bench --runtime baseline"

The result is one block per limit, which is an energy-against-time curve for each runtime: a strong figure, and hard to argue with if the new runtime wins at every limit.

--cool-to 50 waits before each run until every NVIDIA GPU is at or below 50 °C (at most --cool-timeout seconds), so no run starts on a hot card. It never warms a cool one, so the first runs after an idle spell still start cooler than the rest; --warmup 3 brings the card to its working temperature before the measured runs. Pick a target the card reaches between runs: a passively cooled T4 in a notebook sits near 65 °C, a desktop card with fans cools to 50 °C. --gpus 0 waits for the GPU your work runs on only. --rest 10 pauses 10 seconds between runs instead.

Making runs long enough

A meter cannot resolve what happens faster than it updates. Aim for at least 30 seconds per run: loop a short kernel many times inside one run rather than timing one call. NVIDIA's energy counter (Volta and newer) updates every few to 100 milliseconds, RAPL about every millisecond. Older NVIDIA cards only report power, which joules samples every --interval (0.1 s) and integrates.

Phases inside a long-running program: the agent

When the work is one long process with phases inside it (load, full recompute, a stream of incremental updates), run the agent on the machine and have the program mark each phase over HTTP:

joules agent --interval 0.2          # listens on 127.0.0.1:9877

curl -s -X POST localhost:9877/phase/start -H 'content-type: application/json' -d '{"name": "incremental update"}'
# {"id": "p1", "name": "incremental update", "started_at": 1792145658.29}
curl -s -X POST localhost:9877/phase/end -H 'content-type: application/json' -d '{"id": "p1"}'
# {"id": "p1", "name": "incremental update", "duration_s": 0.538, "energy_j": 169.5, "avg_w": 315.0,
#  "devices": {"GPU 0 NVIDIA GeForce RTX 3070 Ti": 134.5, "CPU package 0": 35.0}, "gpu": {...}}

Each mark reads every meter at that moment, so a phase's energy is the change in each device's running total between its two marks, at the counter's own resolution, whatever --interval is. Phases may overlap and nest. End a phase by its id, or by name for the most recent open phase with that name. The end reply has the phase's energy per device and, for NVIDIA GPUs, what the card did during it: clock and its share of the maximum, busy and memory-controller percentages, the share of time throttled and why, the temperature at the start and end, and the power limit. Health is read twice a second, so a phase shorter than that has energy and time but no health.

The phase endpoints accept JSON only and refuse any request a web browser makes (one with an Origin header), so a web page you visit cannot mark phases on your machine. The agent listens on 127.0.0.1 unless you pass --host 0.0.0.0.

From C, C++ or Rust, the same two HTTP calls work with any client. From the HST Console or any dashboard, GET /phases gives the medians to show next to your own telemetry: joules per incremental update next to joules per full recompute, on the same timeline.

Phases inside a Python program: joules.Session

import joules

with joules.Session(gpus=[0]) as s:
    for batch in batches:
        with s.phase("incremental update") as p:
            runtime.update(batch)
        log(p.energy_j, p.duration_s, p.gpu)       # known as soon as the block ends
    with s.phase("full recompute"):
        runtime.recompute()
    print(s.summary())                              # per name: count, median, quartiles, spread
    s.save("phases.json")                           # every phase, for your own analysis

with joules.measure("one solve") as p:              # a single block
    solve()
print(p.energy_j)

The same mechanism as the agent, in your own process, with no server. s.begin(name) and s.end(id) work where a with block does not fit. On a machine with no energy counters Session() raises joules.NoMeters with the reason.

To keep a session, give it a project: joules.Session(track="hst-study", name="incremental vs full"). When the block ends, the whole session is saved to your WattOps account as one run (its energy, time, cost, CO₂ and GPU health) with every phase's count, totals, median and spread, and joules prints a link to it. On a paid plan, its PDF button on Tracking opens the report to read, print or download at any time. It uses the key from joules login (or $LANDAUER_API_KEY); if the upload fails, the program carries on and joules says why. Without track= nothing leaves the machine.

Per-job energy on a shared machine: when the split is exact

joules agent reads each device's energy counter once an interval and gives that energy to the jobs on the device. The counter itself is exact to the hardware's resolution. What can be approximate is the split between jobs. Every finished job says which split it got (the attribution field in /jobs, on Tracking and in its PDF), and how much of its wall time the agent actually sampled (coverage).

A job's label is the split that gave it the most GPU energy. /jobs also lists the joules per split (attribution_j).

Sample coverage. The share of a job's wall time in which the agent read every meter on time. An interval that came late (the agent was starved of CPU, or the machine suspended) counts only up to two intervals, and an interval with a failed meter read does not count. The job's energy is never lost when that happens, because the counters keep counting, but the split over the missed time is coarser. Below 95% the agent prints a warning, and Tracking and the PDF flag the run.

For an exact per-job number, give each job whole GPUs (MIG slices are measured as the whole GPU), keep the agent's interval at 1 second or less, and check that coverage stays at 100%. On shared GPUs, turn on accounting mode or use a driver with per-process utilisation, and report the split next to the number.

Hardware notes

• NVIDIA. Energy counter on Volta and newer, including the RTX 30, 40 and 50 series, RTX PRO Blackwell and B200; clocks, temperature, throttle reasons and power limit on every card the driver supports. Readable by any user; setting a power limit needs root.
• AMD Ryzen and EPYC, Intel Core and Xeon (Linux). RAPL, one meter per CPU package (socket), covering the whole package rather than single cores, so keep other work off the socket during a run, or pin your benchmark and stay off the other cores. Intel also reports DRAM; AMD does not. Reading RAPL needs root, or sudo chmod a+r /sys/class/powercap/intel-rapl:*/energy_uj once.
• Macs. powermetrics (CPU, GPU and Neural Engine on Apple Silicon), which needs your password once. It reports power, which joules samples and integrates, so prefer runs of several seconds or more.
• Wall power. The meters cover the chips, not the power supply, fans or the rest of the machine. To make claims about wall power, joules calibrate compares the meters with a plug-in wall meter over the same minutes (--wall-wh for a meter that shows a running total, --wall-csv for one that logs) and prints the share of the wall reading they saw.

Keeping the results

Add --track PROJECT (after joules login) and every measured run lands on your private Tracking page at wattops.dev under its command's name (hst, baseline, or hst @ 70% power with --limits), with what the GPU was doing. Only the names you chose and the measurements are sent, never the commands' arguments, code, data or output. Each run is one API call: the Free plan takes 10 a minute and 100 a month, and joules waits when the per-minute limit says so rather than lose a run.

On the Tracking page, pick the project and the Compare runs panel lines the two up:

• a headline, such as "hst used 42.9% less energy than baseline (p = 0.008) and took 33.3% less time";
• one row per power limit with each side's median energy and time, the change and its p-value, energy × time, and the share of time each side's GPU was throttled;
• the same warnings as the command line (different throttling or clocks, too few runs, different hardware);
• energy against time with every run and each limit's medians, so a sweep reads as one curve per runtime;
• Copy as Markdown for a paper or a README.

It works for any two run names, not only those from joules compare: joules run --label hst --track study -- … and --label baseline give the same panel. History, charts and CSV export come with it, and on a paid plan an energy report as a PDF to share with reviewers.