Optimizing Real-World Latency Over Theoretical FLOPs

This direction is merged into another direction: FLOPs-latency discrepancy. Its last version stays readable and citable here.

Version 1 of 128 papers34 conceptsfrozen 2026-10-04

For years, the primary metric for "efficiency" in deep learning was the reduction of floating-point operations (FLOPs) and parameter counts. This led to the ubiquity of depthwise separable convolutions and group convolutions. However, a growing body of research suggests that these theoretical metrics are often decoupled from actual execution speed on hardware.

Researchers like Jierun Chen and Shiu-hong Kao argue in Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks that we have been chasing the wrong metric. Instead of minimizing total operations, we should maximize arithmetic intensity—the ratio of computation to memory access—to achieve higher throughput (FLOPS) on real-world hardware.

The Efficiency Paradox of Compact Operators

The central tension in modern architecture design is between "theoretical efficiency" and "hardware affinity." Compact operators like depthwise convolutions reduce FLOPs by breaking a standard convolution into smaller, fragmented pieces. While this looks good on paper, it creates two major bottlenecks:

  1. Memory Access Overhead: As noted by Lei Xing and Longjun Liu in their work on L-Mobilenet, fragmented operations increase the frequency of memory reads and writes. On embedded systems and mobile CPUs, the energy and time cost of moving data often dwarfs the cost of the actual computation.
  2. Low Arithmetic Intensity: Standard $3 \times 3$ convolutions have high data reuse. Once a patch of data is loaded into a processor's cache, it is used for multiple calculations. Depthwise operations, by contrast, underutilize the parallel processing capabilities of GPUs and TPUs, leading to what Mingxing Tan and Quoc V. Le identify as poor hardware utilization in EfficientNetV2.

Leveraging Library Optimizations

A critical reason standard convolutions outperform "efficient" alternatives is the existence of highly optimized low-level libraries like cuDNN and MKL. These libraries use specialized algorithms, such as the Winograd transform, to accelerate $3 \times 3$ convolutions.

In EfficientRep, Kaiheng Weng and Xiangxiang Chu demonstrate that by aligning architectural choices with these hardware-preferred patterns, one can achieve state-of-the-art accuracy with significantly lower latency. Similarly, Zhaofan Qiu and Ting Yao show in LPS-Net that standard convolutions on modern GPUs can reach hundreds of frames per second because they "saturate" the hardware's compute units more effectively than depthwise separable layers.

Structural Re-parameterization: The Best of Both Worlds

If multi-branch architectures (like ResNet) are better for training but simple stacks (like VGG) are faster for inference, why not have both? This is the intuition behind Structural Re-parameterization, pioneered by Xiangyu Zhang and Xiaohan Ding in RepVGG.

During training, the model uses multiple branches (e.g., $3 \times 3$, $1 \times 1$, and identity) to capture diverse features. Before deployment, these branches are mathematically collapsed into a single $3 \times 3$ convolutional layer. This allows the model to benefit from the optimization of dense operations during inference without losing the representational power of complex architectures.

Fusing for Speed

Another strategy to bridge the gap between FLOPs and latency is the elimination of "irregular" operations that break the computational flow. Haichuan Yang and Yonggan Fu introduced DepthShrinker, a framework that removes redundant activation functions to merge sequences of compact operators into single, dense convolutions.

By converting a series of low-reuse operations into one high-reuse operation, the framework improves hardware throughput even if the total FLOP count remains similar. This philosophy is echoed in EfficientNetV2, where "Fused-MBConv" layers replace depthwise convolutions in early stages to maximize training and inference speed on GPUs and TPUs.

Beyond Convolutions: Hardware-Native Sequences

This hardware-aware shift is also transforming sequence modeling. In TCNCA, Aleksandar Terzic and Michael Hersche replace complex attention mechanisms with dilated convolutions. They argue that while some attention approximations have lower theoretical complexity, dilated convolutions are more "hardware-native," mapping directly to optimized GPU kernels and resulting in a $7\times$ speedup on long sequences.

Similarly, Xin Yang and colleagues demonstrate in cMoLLM that using convolutional primitives for routing in Mixture-of-Experts models allows the use of standard hardware-optimized libraries, ensuring that the theoretical benefits of conditional computation actually translate into practical performance gains.

Conclusion

The research direction represented by CSPNet, FasterNet, and RepVGG marks a transition from mathematical minimalism to hardware pragmatism. By prioritizing arithmetic intensity and library compatibility over raw FLOP reduction, these architectures prove that the fastest path to an answer isn't always the one with the fewest steps—it's the one that keeps the hardware running at full speed.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimizing_real_world_latency_over_theor_ad57cfb7,
  title = {Optimizing Real-World Latency Over Theoretical FLOPs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimizing-real-world-latency-over-theoretical-flops/rdr_ad57cfb7a7dc4ba1a2bdf967e30d47f2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimizing_real_world_latency_over_theor_ad57cfb7_v1,
  title = {Optimizing Real-World Latency Over Theoretical FLOPs},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimizing-real-world-latency-over-theoretical-flops/rdr_ad57cfb7a7dc4ba1a2bdf967e30d47f2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}