For years, the deep learning community relied on a simple proxy to measure a model's efficiency: the number of floating-point operations (FLOPs) or multiply-accumulate operations (MACs). The underlying assumption was intuitive—fewer mathematical operations should mean faster execution and lower power consumption.
However, as models migrated from theoretical papers to real-world silicon, engineers noticed a frustrating paradox: a model designed to minimize FLOPs can often run significantly slower than a theoretically heavier counterpart. This phenomenon, known as the FLOPs-latency discrepancy, arises because hardware-agnostic metrics ignore the physical realities of modern computing hardware. To design truly efficient models, we must look past operation counts and understand the bottlenecks of memory, parallelism, and software execution.
The Hidden Cost of Data Movement
FLOPs only count arithmetic operations occurring inside a processor's algebraic units. They completely ignore the cost of moving data through the memory hierarchy (from main memory to caches and registers). In modern hardware, data movement is often orders of magnitude more expensive—both in time and energy—than computation itself.
As Ningning Ma et al. demonstrated in ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design, the Memory Access Cost (MAC) heavily dictates actual runtime. Element-wise operations—such as activations, dropouts, and residual additions—require virtually zero FLOPs but demand substantial memory bandwidth. When a model is bottlenecked by memory bandwidth, the processor sits idle waiting for data, meaning that reducing FLOPs yields no practical speedup.
The authors of All You Need Is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification encountered this exact bottleneck when experimenting with manual tensor shifts. While shifting data across memory addresses bypasses traditional convolutional math entirely, it creates severe memory access overheads that can render the model slower than a standard convolution on physical hardware.
Hardware Parallelism and the Penalty of Fragmentation
Modern accelerators, such as GPUs and TPUs, are massive parallel processing engines optimized for dense, uniform matrix operations. They thrive on regularity. When an architecture introduces sparse operations, branching paths, or fragmented tensors to save FLOPs, it often inadvertently starves the hardware.
Consider group convolutions or sparse attention mechanisms. Mathematically, they cut down the required computation by breaking large matrices into smaller, isolated blocks. However, Guo-Jun Qi et al. showed in Interleaved Group Convolutions that these sparse permutations introduce significant structural overhead. Because standard hardware libraries are highly optimized for dense matrix multiplications, these fragmented operations fail to utilize the hardware's parallel processing capabilities, leading to low hardware utilization.
Conversely, models that embrace dense, hardware-friendly structures can achieve superior wall-clock speed despite having higher FLOP counts. As argued by Tal Ridnik et al. in TResNet: High Performance GPU-Dedicated Architecture, prioritizing dense convolutions allows a model to fully saturate the GPU's capabilities, outperforming low-FLOP architectures in actual throughput.
In a single-author study, Sheng Li 0007 reinforced this design principle in Searching for Fast Model Families on Datacenter Accelerators. The paper demonstrates that models with higher FLOP counts can achieve lower latency simply by maximizing the utilization of the underlying matrix multiplication units, proving that hardware alignment matters more than abstract operation counts.
The Software Execution Gap and Kernel Overhead
Beyond physical hardware architecture, the software stack—compilers, runtimes, and specialized execution kernels—plays a critical role in the latency equation. Every time a model executes a layer, the host CPU must launch a "kernel" on the accelerator. If a model consists of many small, fragmented layers rather than a few large ones, the kernel launch latency can become the primary bottleneck.
Zhongchen Zhao et al. highlighted this in Polyline Path Masked Attention for Vision Transformer, explaining that theoretical reductions in algorithmic complexity are frequently wiped out by implementation overheads like excessive GPU kernel launches. To bridge this gap, algorithmic innovations must be paired with low-level kernel engineering. For example, Keisuke Kamahori et al. developed a custom Triton-based attention kernel in LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation to ensure that their theoretical memory savings translated into actual wall-clock speedups.
Furthermore, hardware compilers and schedulers optimize layer configurations in ways that hardware-agnostic metrics cannot foresee. As Andrew G. Howard et al. observed in NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications, treating the underlying hardware as a black box leads to suboptimal deployment because indirect metrics fail to capture cache hierarchies and compiler-specific optimizations.
Designing for Real Silicon
To overcome the FLOPs-latency gap, the field has shifted toward hardware-aware and platform-aware paradigms. Instead of optimizing for abstract mathematical metrics, modern frameworks incorporate direct hardware measurements into the design loop.
Jin-Dong Dong et al. pioneered this with PPP-Net: Platform-aware Progressive Search for Pareto-optimal Neural Architectures and DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures. By using multi-objective neural architecture search (NAS) that measures actual inference time on target devices, they mapped out the true Pareto front of accuracy and latency across diverse hardware platforms.
Similarly, Jiyang Xie et al. integrated hardware constraints directly into model scaling in ScaleNet: Searching for the Model to Scale, ensuring that scaled-up models do not run into unexpected memory bandwidth walls. By shifting the optimization target from theoretical FLOPs to physical wall-clock execution, researchers can design architectures that are truly optimized for the silicon they run on.