When designing deep neural networks for edge devices, microcontrollers, or mobile accelerators, the standard playbook is to minimize the number of parameters and Multiply-Accumulate (MAC) operations. This design philosophy gave rise to highly popular compact architectures that rely heavily on depthwise separable convolutions, $1\times1$ pointwise convolutions, and aggressive grouping strategies.
However, a counterintuitive reality has emerged in real-world deployments: minimizing FLOPs or MACs does not guarantee a faster or more energy-efficient model. As Nandan Kumar Jha et al. demonstrate in The Ramifications of Making Deep Neural Networks Compact, parameter-reduction techniques often drastically lower a model's arithmetic intensity—the ratio of computational operations to memory accesses. When arithmetic intensity drops too low, the network becomes entirely memory-bound. The hardware spends more time fetching weights and intermediate activations from memory than it does performing actual calculations, leading to severe processor underutilization and wasted energy.
Why Specialized Hardware Stalls
To understand why compact layers fail to deliver their theoretical speedups, we must look at how modern AI hardware is built. High-performance accelerators—such as GPUs, Edge TPUs, and systolic arrays—are optimized for dense General Matrix Multiplications (GEMMs). These architectures achieve massive throughput by exploiting data reuse. In a standard convolutional layer, a single weight is reused across many spatial locations of the input feature map, and a single input activation is reused across multiple filters. This high data reuse allows hardware to load a value from slow off-chip memory once and use it hundreds of times across an array of processing elements (PEs).
When executing depthwise convolutions, this assumption completely breaks down. Because each filter channel operates independently on a single input channel, there is zero cross-channel data reuse. As Shreyas Ravishankar et al. observe in DRACO, this creates a fundamental structural mismatch for systolic arrays. The unique data-access patterns of depthwise operations cannot align with standard spatial acceleration units (Jiajun Zhou et al. in DyBit). As a result, the PEs sit idle, starved of data, while the memory bandwidth is fully choked (Angelo Garofalo et al. in DARKSIDE).
This memory-bound overhead is not unique to depthwise layers. Other common network structures suffer from a similar lack of data reuse:
- Fully Connected Layers & Residual Links: These structures place extreme pressure on on-chip networks, requiring massive bandwidth to maintain high PE utilization (Robert Guirado et al. in Understanding the Impact of On-chip Communication on DNN Accelerator Performance).
- Macro-Operations: Reshapes, transposes, and normalization layers consume significant runtime on edge hardware because they are entirely memory-bound, despite requiring zero or few actual floating-point operations (Seokju Yun et al. in SHViT).
On specialized edge hardware like DSPs or TPUs, the energy and time costs of these low-intensity layers become entirely dominated by data movement rather than active computation (Hanxiao Liu et al. in MobileDets; Berkin Akin et al.).
Algorithmic Redesign: Balancing Math and Memory
To solve this bottleneck, researchers are shifting away from the "low-MACs at all costs" paradigm toward hardware-aware neural architecture design. The core insight is to intentionally design layers that maintain enough arithmetic intensity to saturate the hardware's compute capacity.
Reintroducing Dense Convolutions
One straightforward solution is to selectively reintroduce standard convolutions where accelerators expect them. In MobileDets, the authors utilize a hardware-aware Neural Architecture Search (NAS) to place regular convolutions back into mobile vision models, outperforming depthwise-only models on Edge TPUs and DSPs. Similarly, Berkin Akin et al. propose a Group Convolution-based Inverted Bottleneck (GC-IBN) block in Searching for Efficient Neural Architectures for On-Device ML on Edge TPUs, bridging the gap between parameter-efficient depthwise layers and hardware-friendly full convolutions.
Optimizing Group Sizes
Instead of extreme grouping (where the group size equals 1, as in depthwise convolution), architectures can tune the number of channels per group to strike an optimal balance between parameter reduction and data reuse. Nandan Kumar Jha et al. introduce this concept in E2GC, keeping the group size constant across layers to prevent arithmetic intensity from cratering. Algorithmic frameworks like DRACO also strategically scale up channel counts within group convolutions to boost hardware utilization without modifying the underlying accelerator.
On microcontrollers, this effect is so pronounced that standard convolutions often achieve higher energy efficiency and throughput than depthwise convolutions because their superior arithmetic intensity prevents CPU registers from stalling (Alberto Ancilotto et al. in XiNet).
Bypassing Convolutions via Shift Operations
Another creative algorithmic alternative is to replace expensive, low-intensity convolutions entirely. In GhostSR, Ying Nie et al. identify redundant feature maps and generate them using a learnable shift operation rather than memory-heavy depthwise convolutions, successfully translating theoretical FLOP reductions into actual GPU speedups.
Hardware and Dataflow Co-Design
When the neural network architecture cannot be altered, the hardware's data movement patterns must adapt instead. Standard accelerators use weight-stationary or output-stationary dataflows optimized for cross-channel reuse. To accelerate depthwise layers efficiently, custom hardware must implement intra-channel data reuse.
Standard Dataflow (Cross-Channel Reuse):
[Input Channel 0] ---> [Filter 0] ---> [Output Channel 0]
[Input Channel 0] ---> [Filter 1] ---> [Output Channel 1] <-- High Reuse of Input
Depthwise Dataflow (Intra-Channel Spatial Reuse):
[Input Channel 0] ---> [Filter 0] ---> [Output Channel 0] <-- Zero Cross-Channel Reuse
|-> Spatial Shift -> [Filter 0] <-- Must Reuse Within Channel
By tailoring the hardware data path to exploit spatial dependencies within a single channel, custom architectures can bypass the traditional utilization bottleneck. For example, Haoran You et al. implement specialized data paths and parallel-read buffers in EyeCoD to optimize depthwise execution, achieving an order of magnitude better energy efficiency than general-purpose mobile hardware.
Similarly, the authors of e-G2C employ hybrid dataflow strategies—combining column-wise and deep intra-channel reuse—to ensure that multiplication-and-accumulation (MAC) lanes remain continuously active during sparse depthwise processing (Yang Zhao et al.).
Finally, software-level memory management frameworks can be used to mitigate low operational intensity when hardware changes are impossible. Alessio Burrello et al. developed DORY, a framework that uses a constraint-based optimizer and automated double-buffering to aggressively tile and schedule memory transfers. By overlapping data movement with execution, it hides the latency overhead of low-intensity layers, allowing resource-constrained IoT endnodes to execute compact networks near ideal on-chip speeds.
Key Foundational Work
DORY: Lightweight memory hierarchy management for deep NN inference on IoT endnodes: work-in-progress (2019) — contributed 3 concept elements to this direction.
DLAU: A Scalable Deep Learning Accelerator Unit on FPGA (2016) — contributed 1 concept elements to this direction.
Understanding the Impact of On-chip Communication on DNN Accelerator Performance (2019) — contributed 1 concept elements to this direction.