Memory Efficient Direct Convolution for Deep Learning

This direction is split into other directions: memory-intensive data transformation. Its last version stays readable and citable here.

Version 1 of 132 papers34 conceptsfrozen 2026-10-04

The efficiency of modern computer vision rests on a hidden compromise. To achieve the high speeds required for real-time inference, deep learning frameworks typically transform the intuitive "sliding window" of a convolution into a massive matrix multiplication. While this makes the math fast, it creates a significant "memory tax" that limits the deployment of neural networks on edge devices and prevents the scaling of models to higher resolutions.

A growing body of research is focused on breaking this trade-off by developing high-performance direct convolution methods and "on-the-fly" transformations that eliminate the need for redundant data storage.

The im2col Tax: Speed at the Cost of Memory

The standard way to implement a convolution is a technique called im2col (image-to-column). As Sharan Chetlur and colleagues at NVIDIA detailed in their work on cuDNN, this process rearranges input image patches into columns of a large matrix. This allows the operation to use General Matrix Multiplication (GEMM), which is highly optimized for modern hardware.

However, because convolutional kernels overlap, the same pixel values are duplicated many times in the resulting matrix. Jimmy Ren and Li Xu noted in On Vectorization of Deep Convolutional Neural Networks for Vision Tasks that this redundancy creates a "scalability ceiling." The memory overhead of data reorganization eventually outweighs the computational gains, especially as networks grow larger or are deployed on memory-constrained hardware.

Eliminating the Intermediate Buffer

To solve this, researchers are revisiting direct convolution—performing the operation without an intermediate expanded matrix. Historically, direct convolution was considered slow because it didn't map well to the vector units of CPUs and GPUs.

Recent work has proven this conventional wisdom wrong. Jiyuan Zhang, Franz Franchetti, and Tze Meng Low demonstrated in High Performance Zero-Memory Overhead Direct Convolutions that by meticulously reordering loops and using hardware-aware data layouts, direct convolution can actually outperform GEMM-based methods by 10% to 400% while requiring zero additional memory overhead.

On the hardware side, Jordi Fornt and Jaume Abella proposed a Customized NoC Architecture that performs im2col transformations "on-the-fly." By integrating a data feeder that presents pixels to the processing units in the correct order without storing them in a buffer, they cut off-chip memory traffic by 50%.

Exploiting Sparsity and Patterns

Another way to reduce the memory footprint is to avoid processing redundant or empty data altogether. Ahmed H. Salamah and EN-HUI YANG introduced Compressed Pattern Overlap (CPO), which exploits activation map sparsity. Instead of duplicating pixels in overlapping regions, their algorithm uses pattern-based index compression to achieve up to 26x memory compression.

Similarly, for pruned networks where many weights are zero, Xuhao Chen developed Escort, a system that bypasses the im2col lowering step in favor of direct sparse convolution. This prevents the "memory bandwidth waste" that occurs when sparse data is expanded into dense matrices, leading to a 3x speedup on GPUs.

Optimization for the Edge

The push for memory efficiency is most critical at the "edge"—on low-power processors like ARM and RISC-V. Cristián Ramírez and Enrique S. Quintana-Orti have explored how to orchestrate data movement across the software-managed memory of RISC-V processors to perform GEMM-based convolution without exceeding tiny L1 caches.

For even more extreme constraints, Albert Gural and Boris Murmann proposed a Memory-Optimal Direct Convolution using a "herringbone" computation strategy. This method mathematically optimizes the order of pixel processing so that "stale" data is overwritten immediately, allowing high-accuracy CNNs to run on hardware with as little as 2KB of RAM.

A New Paradigm for Convolution

The shift away from standard im2col is not just about saving memory; it's about rethinking the fundamental operator of vision tasks. Whether through im2win, which uses a "window-order" format to reduce GPU memory usage by 30%, or SMM-Conv, which treats convolution as a summation of shifted sub-matrices, the goal is the same: to achieve the throughput of dense linear algebra without the prohibitive cost of data duplication.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_memory_efficient_direct_convolution_for__f269598b,
  title = {Memory Efficient Direct Convolution for Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-efficient-direct-convolution-for-deep-learning/rdr_f269598b403e4348b437c76824c86192},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_memory_efficient_direct_convolution_for__f269598b_v1,
  title = {Memory Efficient Direct Convolution for Deep Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-efficient-direct-convolution-for-deep-learning/rdr_f269598b403e4348b437c76824c86192/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}