Standard self-attention is a victim of its own success. In the Transformer architecture, every token attends to every other token, creating a global receptive field. However, this comes at a quadratic cost: for an image with $N$ pixels, the computational complexity is $O(N^2)$. As we move toward high-resolution images or large-scale spatial networks, this "quadratic wall" makes standard attention computationally prohibitive.
To scale Transformers to high resolutions, researchers have converged on a shared strategy: Dimensional Decomposition. By factorizing a single 2D attention operation into separate 1D operations along orthogonal axes, models can maintain a global receptive field while reducing complexity to linear or sub-quadratic levels.
The Axial Insight
The core intuition behind dimensional decomposition is that a pixel does not need to look at every other pixel simultaneously to understand the whole image. Instead, it can look at its own row, and then its own column.
Huiyu Wang and colleagues pioneered this with Axial-DeepLab, which factorizes 2D self-attention into two sequential 1D operations. By attending only to pixels on the same height and width axes, the model captures long-range dependencies across the entire image grid with $O(N\sqrt{N})$ complexity. This approach allows for "stand-alone" attention networks that outperform traditional CNNs on dense tasks like panoptic segmentation.
Similarly, Xinggang Wang introduced CCNet, which uses a "criss-cross" attention module. By recurrently applying attention along horizontal and vertical paths, each pixel eventually gains a global receptive field. This mechanism uses 11x less GPU memory than standard non-local blocks, enabling the processing of high-resolution feature maps that were previously unreachable.
Parallel and Cross-Shaped Windows
While sequential axial attention is effective, later researchers found that processing these dimensions in parallel can be even more powerful.
Xiaoyi Dong developed the CSWin Transformer, which utilizes "Cross-Shaped Window" attention. Instead of sequential passes, it splits attention heads into parallel groups: one group processes horizontal stripes and the other processes vertical stripes. This parallelization allows the model to capture a larger receptive field earlier in the network compared to square-window approaches like Swin.
Sitong Wu took this further with the Pale Transformer, introducing "Pale-Shaped" attention. By performing attention within sparse, interlaced rows and columns, the model achieves linear complexity relative to the image area. This efficiency is critical for dense prediction tasks like semantic segmentation, where maintaining high-resolution feature maps is essential for accuracy.
Beyond Attention: Sparse MLPs and Proxies
The principle of dimensional decomposition is not limited to the attention mechanism itself; it has been successfully applied to Multi-Layer Perceptrons (MLPs) and large-scale spatial forecasting.
In sMLPNet, Chuanxin Tang showed that even without self-attention, one can achieve state-of-the-art image recognition by mixing information along horizontal and vertical axes using shared weights. This "Sparse MLP" design matches the accuracy of advanced Transformers while significantly reducing the risk of overfitting and computational overhead.
For non-visual data, such as nationwide air quality forecasting, Jiaming Ma introduced LargeAQ. Instead of computing pairwise attention between thousands of monitoring stations, the model uses "cache-attention" (CachLormer). Nodes attend to a small set of learnable latent representations (caches) that represent coarse-grained regional patterns. This "proxy-based" approach reduces complexity from quadratic to linear, allowing for real-time forecasting across massive spatial networks.
Why Decomposition Works
The success of these methods rests on a fundamental geometric truth: information propagation on a grid is efficient.
In a standard $N \times N$ attention matrix, we assume every pixel's relationship to every other pixel is equally unique and necessary to compute in one step. Dimensional decomposition recognizes that these relationships are often redundant or can be composed. By decomposing the 2D interaction:
- Memory Efficiency: We avoid materializing the $N^2$ similarity matrix. Francesca Babiloni demonstrated in Poly-NL that factorizing interactions using Hadamard products can capture long-range dependencies with $O(N)$ complexity.
- Receptive Field: Two 1D passes (horizontal then vertical) allow any pixel $(x_1, y_1)$ to communicate with any other pixel $(x_2, y_2)$ via an intermediate point $(x_1, y_2)$.
- Inductive Bias: As noted by Ce Zheng in HeatER, dimensional decomposition avoids the "unnatural flattening" of 2D images into 1D sequences, preserving the inherent coordinate structure of the data.
By breaking the 2D problem into 1D components, these architectures bypass the quadratic wall, enabling Transformers and MLPs to process the high-resolution world with linear efficiency.