Optimizing Neural Networks for Discrete Hardware Latency Steps

This direction is merged into another direction: FLOPs-latency discrepancy. Its last version stays readable and citable here.

Version 1 of 130 papers44 conceptsfrozen 2026-10-04

In the pursuit of efficient deep learning, we often use FLOPs (floating-point operations) or parameter counts as proxies for speed. The intuition is simple: a model with 50% fewer operations should run roughly twice as fast. However, researchers have increasingly observed that this linear relationship is a myth. On actual hardware, latency does not glide down a smooth slope; it descends a staircase.

This "staircase" effect means that reducing a model's size might yield zero performance gain until a specific threshold is crossed, at which point the latency drops abruptly. Conversely, as Guoyang Chen and colleagues discovered in their work on Latency Hysteresis, a smaller model can sometimes even run slower than a larger one due to memory alignment and hardware quirks.

The Mechanism: Parallel Waves and the Tail Effect

To understand why latency is discrete, we must look at how modern hardware, like GPUs and edge accelerators, schedules work. These devices do not process operations one by one; they process them in parallel "waves" or "warps."

Imagine a bus with 32 seats. If you have 33 passengers, you need two buses. If you manage to "prune" 10 passengers, you still have 23 people—and you still need one full bus. The total travel time remains the same. It is only when you prune that 33rd passenger that you can eliminate the second bus entirely, leading to a massive jump in efficiency.

Fuxun Yu and Longfei Shangguan refer to this as the GPU tail effect. When a workload doesn't perfectly align with the hardware's parallel capacity, the "tail"—the final, partially filled wave of computation—takes just as long to execute as a fully saturated wave. This creates a plateau in the latency curve where reducing computation provides no wall-clock benefit.

Exploiting the Staircase

If latency is a staircase, the most efficient models are those that sit exactly at the edge of a step. Elliot J. Crowley and Amos Storkey leveraged this in their work on Performance Enhanced Students. Instead of pruning a model to an arbitrary size, they "nudge" layer widths to the maximum capacity allowed within a specific latency step. This allows the model to retain more parameters (and thus more accuracy) without any penalty in inference speed.

This principle applies across different architectures:

  • Vision Transformers: Nicholas John Eliopoulos showed that in token pruning, pruning "one more token" can be the difference between hitting a hardware "sweet spot" and wasting cycles.
  • CNN Decomposition: Lizhi Xiang and Miao Yin found that in Tucker decomposition, increasing the rank of a layer often costs zero marginal latency because of how thread waves are scheduled.
  • Large Language Models: In speculative decoding, Qidong Su observed that the optimal number of tokens to speculate is heavily dependent on batch size due to these same discrete processing thresholds.

Because these "steps" are specific to the hardware—varying between an NVIDIA T4 GPU, a mobile phone's DSP, or a custom ASIC—optimization must be hardware-aware.

Traditional pruning methods often fail because they use linear proxies. In contrast, frameworks like SPDY by Elias Frantar and Dan Alistarh use dynamic programming to find sparsity profiles that guarantee actual speedups. Similarly, Archtree by Edouard Yvinec performs on-the-fly exploration on the target device to navigate the non-linear latency landscape.

The shift in perspective is profound: we are moving away from "making models smaller" toward "fitting models into hardware shapes." As Xinglong Sun demonstrated with MDP, by modeling these non-linearities using precomputed lookup tables, we can find subnetworks that are both faster and more accurate than those found by any FLOP-counting heuristic.

In summary, the "latency staircase" is not just a hurdle; it is an opportunity. By understanding the discrete nature of hardware execution, we can stop leaving performance on the table and start building models that are perfectly harmonized with the silicon they run on.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimizing_neural_networks_for_discrete__96eaab2d,
  title = {Optimizing Neural Networks for Discrete Hardware Latency Steps},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimizing-neural-networks-for-discrete-hardware-latency-steps/rdr_96eaab2d26764a0a9c4312b3082cc285},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimizing_neural_networks_for_discrete__96eaab2d_v1,
  title = {Optimizing Neural Networks for Discrete Hardware Latency Steps},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimizing-neural-networks-for-discrete-hardware-latency-steps/rdr_96eaab2d26764a0a9c4312b3082cc285/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}