Bridging the Gap Between FLOPs and Hardware Latency

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,146 papers1,508 conceptsfrozen 2026-10-04

For years, the primary metric for "efficiency" in deep learning has been the FLOP (Floating Point Operation). The logic was simple: if you reduce the number of mathematical operations, the model should run faster. However, as researchers have moved from theoretical proofs to real-world deployment, a frustrating reality has emerged: models with fewer FLOPs often run slower than their "heavier" counterparts.

This discrepancy arises because modern hardware—specifically GPUs and specialized AI accelerators—does not treat all operations equally. A research direction has formed around Hardware-Aware Design, which seeks to move beyond FLOPs to account for memory bandwidth, kernel overhead, and arithmetic intensity.

The Myth of the Efficient FLOP

The fundamental reason FLOPs are a poor proxy for speed is that they ignore the "cost of moving data." In modern computing, moving a byte of data from memory to the processor often consumes orders of magnitude more energy and time than the actual calculation performed on that byte.

As Ranajoy Sadhukhan, Beidi Chen, and colleagues point out in Kinetics: Rethinking Test-Time Scaling Laws, traditional "compute-optimal" scaling laws fail because they ignore the memory access bottleneck. A small model that requires frequent, fragmented memory accesses can easily be outperformed by a larger, more "dense" model that utilizes hardware more efficiently.

This is often described through Arithmetic Intensity: the ratio of total operations to total memory traffic. Hardware like GPUs are designed for high arithmetic intensity—performing many calculations on a single piece of data while it is in the local cache. When we "optimize" a model by reducing FLOPs, we often inadvertently lower the arithmetic intensity, forcing the hardware to wait on memory.

The Cost of Architectural Fragmentation

Many "efficient" architectural innovations—such as group convolutions, depthwise separable convolutions, and dynamic branching—achieve low FLOP counts by breaking large operations into many smaller ones. While mathematically elegant, this creates a "fragmentation" problem.

  1. Kernel Launch Overhead: Every time a GPU performs an operation, the software must "launch" a kernel. If a model like the CSWin Transformer or IGC relies on frequent tensor splitting, shuffling, and concatenation, the time spent managing these small tasks can exceed the time spent on the actual math. Xiaoyi Dong and Ting Zhang have noted that these memory-intensive manipulations are often poorly optimized in standard libraries.
  2. Memory Discontinuity: Hardware is fastest when it reads data in long, contiguous blocks. Kun Ding and Guojin He addressed this in PackDet by introducing a "Packing Operator" to stitch fragmented feature maps together, reducing the latency caused by managing multiple small, disconnected computations.

To solve this, researchers are moving away from manual design toward Hardware-Aware Neural Architecture Search (NAS). Instead of optimizing for validation accuracy alone, these systems use actual hardware latency as a feedback signal.

In SqueezeNext, Amir Gholami and his team demonstrated that by designing specifically for the constraints of embedded systems—avoiding operations that are slow on specialized accelerators—they could outperform models like MobileNet that were theoretically efficient but practically slower. Similarly, Mingxing Tan and Quoc V. Le developed EfficientNetV2 by using a training-aware NAS that prioritized operations with high hardware utilization, leading to 11x faster training speeds.

Even the "elegance" of human design is being quantified. Minghao Guo and Dahua Lin introduced IRLAS, which uses Inverse Reinforcement Learning to teach search agents the "topological elegance" of networks like ResNet. These human-like structures tend to be faster because they favor the contiguous data flows that hardware was built to handle.

Redefining Scaling Laws

The ultimate goal of this research direction is to rewrite the "physics" of AI scaling. Traditional scaling laws suggest a linear relationship between compute, data, and performance. However, as Benjamin Van Roy and Hong Jun Jeon argue in their Information-Theoretic Analysis, these laws overlook communication overhead in distributed systems.

Future frameworks, such as the Mechanistic Architecture Design (MAD) proposed by Michael Poli and Armin W. Thomas, use small-scale "unit tests" to predict how hybrid architectures (like those mixing attention with convolutions) will perform on actual silicon at massive scales. By shifting the focus from "how many operations" to "how many tokens per second," the field is finally aligning the math of neural networks with the reality of the machines that run them.

Papers in this version

Showing 60 of 1,146 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 1540 papers
  2. Version 1 · 2026-10-04 · 1146 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_flops_latency_discrepancy_ee9ed2b9,
  title = {FLOPs-latency discrepancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/flops-latency-discrepancy/rdr_ee9ed2b94eb14d9cb432a13621118891},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_gap_between_flops_and_hardw_ee9ed2b9_v1,
  title = {Bridging the Gap Between FLOPs and Hardware Latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-gap-between-flops-and-hardware-latency/rdr_ee9ed2b94eb14d9cb432a13621118891/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}