To compute a convolutional layer, modern deep learning frameworks rarely execute a naive nested loop over spatial dimensions. Instead, they rely on a process called lowering—most commonly via the im2col (image-to-column) transformation. This operation flattens the input tensor and unrolls overlapping local patches into a dense 2D matrix. Once lowered, the convolution is reduced to a standard General Matrix Multiplication (GEMM), allowing frameworks to leverage highly optimized BLAS libraries.
However, as Jimmy Ren et al. observe in On Vectorization of Deep Convolutional Neural Networks for Vision Tasks, this speed comes at a steep cost. Because convolutional kernels slide across the input with overlapping receptive fields, the same pixel is required by multiple patches. Explicitly unrolling these patches means duplicating data multiple times in memory.
Jiyuan Zhang 0002 et al. point out in High Performance Zero-Memory Overhead Direct Convolutions that this intermediate matrix can inflate memory consumption by two to four times the original input size, while Yangjie Zhou et al. note in Characterizing and Demystifying the Implicit Convolution Algorithm on Commercial Matrix-Multiplication Accelerators that it can reach up to ten times the original footprint. This data inflation creates a critical bottleneck, leading to hardware memory exhaustion on large datasets (Fabian Tschopp et al. in Efficient convolutional neural networks for pixelwise classification on heterogeneous hardware systems) and severely limiting deployment on memory-constrained edge hardware (Adrián Castelló 0001 et al. in Experience-guided, mixed-precision matrix multiplication with apache TVM for ARM processors).
Implicit Lowering and On-the-Fly Address Generation
If the primary bottleneck is the physical creation of a bloated intermediate matrix, the most direct solution is to perform the transformation virtually. This is the core intuition behind implicit im2col. Instead of rearranging data in global memory before computation begins, implicit methods compute the mapping between the original tensor coordinates and the lowered matrix coordinates on the fly.
When a matrix multiplication engine requests a specific element of the lowered input matrix, hardware or software address-generation logic intercepts the request. It calculates exactly where that pixel resides in the original, non-duplicated tensor. For instance, in High Performance and Portable Convolution Operators for Multicore Processors, Pablo San Juan et al. introduce CONVGEMM, an algorithm that embeds the im2col transformation directly into the data-packing phase of the GEMM kernel, completely bypassing the intermediate memory footprint.
Explicit im2col:
[Input Tensor] ---> [Allocate Massive Buffer & Duplicate Pixels] ---> [GEMM Kernel]
Implicit im2col:
[Input Tensor] -----------------------------------------------------> [GEMM Kernel]
^ |
|--- [On-the-Fly Coordinate Mapping] -------------|
This concept extends directly into hardware design. Jordi Fornt et al. propose a custom ASIC accelerator in An Energy-Efficient GeMM-Based Convolution Accelerator With On-the-Fly im2col featuring a dedicated data feeder that performs im2col on-the-fly, cutting off-chip memory traffic in half.
Similarly, for training workloads on systolic arrays, Mei Wen et al. develop BP-Im2col in BP-Im2col: Implicit Im2col Supporting AI Backpropagation on Systolic Arrays. Their approach uses smart hardware address generation to eliminate the massive memory and bandwidth overhead caused by explicit zero-padding and dilation unrolling during backpropagation. By shifting the burden from memory storage to coordinate arithmetic, these approaches eliminate the data-duplication bottleneck entirely.
Exploiting Geometry via Compact Matrix Lowering
Another way to mitigate memory bloat is to redesign the lowering transformation itself to minimize redundancy. Standard im2col treats every patch independently, ignoring the spatial relationships between adjacent rows and columns.
To exploit this geometric structure, Minsik Cho et al. introduce Memory-efficient Convolution (MEC) in MEC: Memory-efficient Convolution for Deep Neural Network. This technique is known as compact matrix lowering. Instead of fully unrolling the input in both spatial dimensions, the authors eliminate vertical redundancy by storing the input in compact strips. By unrolling along only one spatial axis and using clever pointer strides to handle the other, MEC achieves up to a 5.9x reduction in memory consumption while remaining fully compatible with standard, highly optimized BLAS GEMM kernels.
Bypassing GEMM with Direct and Tensor-Based Formulations
Rather than fixing the lowering step, some researchers argue for abandoning the matrix-multiplication paradigm altogether in favor of direct or alternative mathematical formulations.
Direct convolutions execute the sliding window natively without any data reorganization. While historically slower due to the lack of hardware optimization compared to GEMM, modern direct implementations achieve high performance through hardware-aware loop reordering and register tiling. In High Performance Zero-Memory Overhead Direct Convolutions, the authors demonstrate that direct convolutions can outperform BLAS-based implementations while requiring zero additional memory overhead. For specialized layers like depthwise convolutions, Ruochen Hao et al. demonstrate in Optimizing Depthwise Convolutions on ARMv8 Architecture that tailored register tiling can bypass the memory accumulation bottlenecks typical of standard frameworks.
On a more abstract level, Yu-Sheng Lin et al. introduce the Unrolled Memory Inner-Product (UMI) operator in Unrolled Memory Inner-Products: An Abstract GPU Operator for Efficient Vision-Related Computations, a GPU primitive that abstracts complex multidimensional memory access patterns into a unified inner-product form to avoid physical unrolling.
Taking this abstraction further, Felix Dangel reformulates convolutions through the lens of Tensor Networks in Convolutions Through the Lens of Tensor Networks. By representing convolutional layers as network diagrams and executing them via efficient Einstein summation (einsum) operations, this approach eliminates intermediate matrix unfolding entirely. This massive memory reduction makes it feasible to run memory-intensive second-order optimization methods (like KFAC) that were previously blocked by im2col memory bloat.
Finally, avoiding explicit matrix expansion is crucial for preserving the inherent properties of the data. As Chen Zhang et al. note in Dual-side Sparse Tensor Core, traditional lowering often destroys dynamic activation sparsity patterns or causes massive memory expansion. By utilizing hardware-software co-design like outer-product primitives, architectures can process sparse feature maps directly without sacrificing memory efficiency.
Key Foundational Work
On Vectorization of Deep Convolutional Neural Networks for Vision Tasks (2015) — contributed 3 concept elements to this direction.
Efficient convolutional neural networks for pixelwise classification on heterogeneous hardware systems (2016) — contributed 1 concept elements to this direction.