Choosing Bit-Widths and Clock Speeds Together on Real Silicon

When a neural network is quantized, meaning its numbers are stored in fewer bits, the obvious gain is that less data moves between memory and the processor. Moving data costs far more energy than doing arithmetic on it. A second, less obvious effect follows: quantization changes how much computation the processor does per byte it fetches. A layer that waited on memory at 8 bits may become limited by arithmetic at 4 bits, or the reverse. This matters because GPUs and CPUs can be locked to a fixed clock speed, and the most energy-efficient clock depends on that balance. Memory-limited work wastes power at a high core clock. Arithmetic-limited work benefits from it, up to the point where the chip must raise its voltage steeply. Today two separate communities make these two choices. Mixed-precision search methods pick per-layer bit-widths while assuming the clock never changes. Frequency tuners pick a clock while assuming precision never changes.

This proposal tests whether that separation costs real energy, and closes the gap if it does. We will search over per-layer bit-widths and a single statically locked (core clock, memory clock) pair at the same time. The cost model comes from measurements on the target chip, not from a cycle-level simulator. The search itself is tractable. For any fixed precision assignment, energy as a function of clock is U-shaped, so only a few dozen clock pairs need checking. At each clock pair, choosing bit-widths reduces to a small integer program. The work succeeds if the jointly chosen configuration beats the best "quantize first, then tune the clock" pipeline at matched accuracy. It also succeeds if it shows, with measurements, that simulator-ranked precision policies fail to hold their ranking once clocks are allowed to move.

Core Claim

Our hypothesis is that the energy-optimal static clock moves when the precision assignment moves. If so, any two-stage pipeline lands at a suboptimal operating point. Two lines of measurement motivate this. Clock-locking studies found a U-shaped energy curve against core frequency. The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study profiled P100, V100 and RTX 2080Ti on CNNs at uniform precision. It reported inference energy savings up to 26.4%. It also found that core and memory clocks should be scaled in opposite directions for compute-bound versus memory-bound work. Going green: optimizing GPUs for energy efficiency through model-steered auto-tuning located the voltage-frequency "ridge point", the clock above which voltage must rise sharply. On an A100 FP32 matrix multiply, its best setting was 50.9% more energy-efficient than running flat out, despite being 27.5% slower. Neither study varied precision. On the other side, HAWQ ranks layers by loss curvature to decide which layers can tolerate few bits. Hardware-Centric AutoML for Mixed-Precision Quantization (HAQ) trains a reinforcement-learning agent on latency and energy feedback from the BitFusion and BISMO simulators. Both treat the clock as fixed. HAQ's feedback also comes from simulators that, per Accurate and stable empirical CPU power modelling for multi- and many-core systems, assign fixed energy per event and miss the $V^2 f$ power scaling across clock states.

The closest work, then, does one half of the problem on silicon or both halves in simulation. What is new here is the joint choice, made against measured silicon cost. The proposal also commits to a falsifiable prediction: across precision assignments, the measured energy-optimal clock pair should shift by a margin larger than the clock-step granularity. If it does not, the joint search is unnecessary and we will say so.

Research Program

The objective is to choose a bit-width vector $\mathbf{b} = (b_1,\dots,b_L)$ over $L$ layers and a locked clock pair $f = (f_c, f_m)$, core and memory, to minimize measured inference energy:

$$E(\mathbf{b}, f) = \sum_{l=1}^{L} t_l(b_l, f)\, P_l(b_l, f) \quad \text{s.t.} \quad \sum_{l} \Omega_l(b_l) \le \Delta,\;\; \sum_l t_l(b_l,f) \le T.$$

Here the symbols mean the following. The quantity $t_l$ is the measured latency of layer $l$, and $P_l$ is its measured average board power. The term $\Omega_l(b_l)$ is a sensitivity proxy for the accuracy cost of quantizing layer $l$ to $b_l$ bits. We use the curvature score from HAWQ, and block-reconstruction error in the style of BRECQ as a check. The constant $\Delta$ is an accuracy budget, and $T$ is a latency service level.

The naive joint space has $B^L$ precision assignments times the number of clock pairs. FracBits counts $4^{50} \approx 10^{30}$ assignments for a 50-layer network. Two structural facts collapse this. First, with $f$ fixed, the objective is additive over layers, so bit selection becomes a multiple-choice knapsack solvable exactly by integer programming in seconds. Second, the U-shaped energy curve reported in the DVFS study means we can restrict the clock search to a coarse grid of roughly 20–40 lockable pairs, then refine locally. We solve one integer program per pair and take the minimum.

The engineering core is the silicon cost table. For each layer shape, candidate bit-width and clock pair, we run the actual deployed kernel and record latency and energy through NVML, using Kernel Tuner's energy observers as in Going green. The kernels are TensorRT INT8/FP16 for CNNs and arbitrary-bit kernels in the style of ABQ-LLM for weight-only LLM quantization. Measuring the deployed kernels captures the packing, unpacking and misaligned-access penalties documented in Dustin and Channel-Wise Mixed-Precision Quantization for Large Language Models. Simulators omit exactly those penalties.

Per-layer measurements do not sum perfectly to end-to-end cost, because of caching across layers, kernel launch gaps and thermal drift. This composition error is our concrete version of the simulation-to-silicon gap. We quantify it directly. If it exceeds a few percent, we fit a small residual correction from a few hundred end-to-end runs. This borrows the spirit of Powmon's physics-informed counter models and of Air Learning's practice of feeding measured hardware behaviour back into the optimizer.

The precision "bytes moved" axis gives us the von Neumann lens. For every configuration we log DRAM traffic per inference. This lets us state how much of the joint gain comes from reduced data movement and how much from the clock moving to match the new compute-to-memory balance.

Analog in-memory accelerators are outside the experimental scope. Their dominant cost is the converter circuitry, per Mixed-Precision Deep Learning Based on Computational Memory, and we have no lockable-clock silicon for them. If the GPU result holds, a natural follow-up would apply the same joint idea to converter precision and digital-periphery clock in IBM AIHWKit-style models. We flag that follow-up as simulation-only.

Baselines

The naive reference point is uniform INT8 at the vendor default clock with the stock governor. This is what a practitioner gets with no effort. Every other method is reported as energy per inference relative to it, at matched task accuracy.

The strongest existing competitor is the two-stage pipeline. It runs the best published precision search first, then the best published clock tuner. Concretely, this means HAWQ-sensitivity bit allocation solved by the same integer program, at the default clock, under the same accuracy budget. That is followed by Going-green-style ridge-point clock search on the resulting model. We also run a HAQ-style variant whose cost signal is a simulator or roofline estimate instead of measurement. This isolates the simulator's contribution.

The strongest reported numbers are as follows. For precision search, HAQ reports 1.4× to 1.95× latency reduction and up to 1.9× energy saving over fixed 8-bit, measured on simulators. For clock tuning, the DVFS study reports up to 26.4% inference energy saving at uniform precision. Going green reports a mean 42.0% energy efficiency gain at a 24.3% mean performance loss on its kernels.

Fairness rules apply across all methods. Every method gets the same models, the same calibration data, the same sensitivity proxy and the same accuracy budget. It also gets the same measurement harness, with clocks locked identically and thermals stabilized before each run. Every method also gets an equal number of silicon measurement runs for tuning.

Beating the two-stage pipeline means at least 10% lower measured energy at equal accuracy and latency budget, on at least two of the three GPU generations. A gap smaller than our run-to-run measurement variance counts as a null result.

The key ablation removes coupling while keeping everything else. We run the joint integer program but force the clock pair to the one that is optimal for uniform INT8. If this ablation matches the full method, the coupling hypothesis is wrong.

Why not just…?

Why not just quantize first and tune the clock afterwards? This is the real threat. If the energy-optimal clock barely moves across precision assignments, two stages lose nothing. We have no direct evidence either way. The suggestion that precision shifts the compute-to-memory balance is an inference from the DVFS study's opposite core/memory responses, not a measured result. The proposal is therefore built so that the first experiment settles this question. If the optimal clock does not shift, the paper becomes a negative result plus a measured cost table, and we will not inflate it.

Why not just set a power cap and let firmware choose the clock? Going green compared power capping directly with explicit clock locking. It found that power caps make the controller throttle unpredictably and give worse efficiency than locking near the ridge point. We still include a power-cap sweep on top of the two-stage precision policy as a baseline, because firmware might happen to track the shifted optimum.

Why not just keep using the simulator, which is cheaper? Air Learning measured a 15.49% energy gap between a simulator-trained policy and the same policy on a Raspberry Pi. The CPU power-modelling paper attributes simulator power errors to fixed per-event energy weights that ignore voltage scaling. Our experiment comparing simulator-ranked and silicon-ranked precision policies measures this gap directly for mixed precision. If the rank correlation is high, the simulator suffices for precision search, and only clock selection needs silicon.

Why not just use uniform INT4 with a tuned clock? BRECQ reports that uniform 4-bit can collapse top-1 accuracy on sensitive networks. A uniform setting also cannot place memory-heavy layers at different precision from compute-heavy ones. We still include uniform INT8 and INT4 with clock tuning as baselines, because this is the cheapest serious competitor.

Proposed Experiments

The first experiment decides whether the project continues. On A100, V100 and an RTX-class card, we take ResNet-50 and ResNet-18 on ImageNet. We sample about 200 bit assignments that satisfy the accuracy budget and, for each, sweep the lockable clock pairs. We then plot the measured energy-optimal pair against each assignment's measured DRAM traffic. The joint project proceeds only if the optimum moves by more than one clock step across this population. It also requires that holding the INT8-optimal clock costs more than 10% energy for a meaningful fraction of assignments.

The main comparison runs the full method against all baselines at two accuracy budgets. The anchor points are HAWQ's ResNet-50 result of 75.48% top-1 at 7.96 MB and BRECQ's ResNet-18 result of 69.6% at 4-bit weights and activations. For language models, a Llama-class model with weight-only mixed precision via AMQ-style search is evaluated on perplexity. No reported energy numbers were found on Lacuna for these exact settings, so all energy baselines will be re-measured by us.

For latency-bound serving we use MLPerf Inference LoadGen in the Offline and Server scenarios. MLPerf reports that Server mode loses over 50% of throughput relative to Offline, so the latency constraint $T$ is likely to bind in Server mode. We expect this to be where joint selection matters most.

The simulator-gap check computes Spearman rank correlation between simulator-predicted and measured energy across the sampled assignments, separately at default and at tuned clocks. If the correlation falls at tuned clocks, this would show that clock changes widen the simulator's error. Finally, we report the per-layer-to-end-to-end composition error with and without the residual correction, together with an energy breakdown separating the gain from fewer bytes moved and the gain from clock repositioning.

One limitation must be stated plainly. Locking clocks requires root access through NVML. The results therefore apply to operators who control their own hardware, not to unprivileged cloud tenants.