Factorized Attention for Volumetric and High Dimensional Data

58 concepts38 papers1 proposal

Standard self-attention scales quadratically with the number of input tokens. When processing a 2D image, a resolution of $224 \times 224$ pixels yields roughly 50,000 tokens—a workload well within the limits of modern accelerators. However, when we transition to 3D volumes (such as medical MRI scans or point clouds) or high-dimensional data cubes (such as hyperspectral images or video), this token count explodes. A modest $256 \times 256 \times 256$ MRI volume contains over 16 million voxels. Computing a full dense attention matrix across these voxels requires $O(N^2)$ operations, causing immediate memory exhaustion.

To bypass this computational bottleneck, a powerful geometric paradigm has emerged: decomposing high-dimensional attention into factorized, low-dimensional passes along orthogonal axes. Instead of calculating a massive joint attention matrix across all dimensions simultaneously, these architectures decouple the problem into sequential or parallel operations. This shifts the computational complexity from quadratic to linear relative to the total voxel or pixel count, unlocking the ability to process global context in high-dimensional datasets.


Orthogonal Spatial Decomposition

The most direct way to factorize volumetric attention is to break down a 3D space into its constituent Cartesian or coordinate axes. Rather than allowing every voxel to attend to every other voxel in the entire volume, the network restricts the attention search space to 1D lines or 2D planes, aggregating global context incrementally.

Standard 3D Attention:  [Voxel] <---> All Voxels in Volume (O(N²))

Factorized Attention:   [Voxel] <---> Voxels on X-axis (Pass 1)
                        [Voxel] <---> Voxels on Y-axis (Pass 2)
                        [Voxel] <---> Voxels on Z-axis (Pass 3)

In 3D medical imaging, Kim et al. pioneered this with RSANet, which processes 3D MRI volumes as sequences of slices across three orthogonal axes in a recurrent fashion. By sequentially applying attention along the sagittal, coronal, and axial planes, each voxel incorporates global context from the entire 3D volume. This directional decomposition reduces GPU memory consumption by approximately 28 times compared to standard non-local attention blocks.

Similarly, Abdul Kareem et al. developed DwinFormer, which decomposes 3D attention into separate horizontal, vertical, and depthwise passes. This directional windowing creates a pseudo-global receptive field that learns how anatomical structures extend across different axes without the quadratic tax. Zhu et al. utilized an identical strategy in their cross-dimensional Transformer framework, modeling dimensional correlations independently along the depth, height, and width axes to achieve a strictly linear complexity relative to the total voxel count.

This principle is not bound to Cartesian coordinates. For 3D semantic occupancy prediction in autonomous driving, Du et al. introduced PVP, which applies axial decomposition to a polar coordinate space. By splitting 3D self-attention into independent operations along the radial, azimuth, and height axes, the model maintains a global receptive field capable of capturing continuous structures like roads while avoiding the memory explosion of standard 3D transformers.

Instead of 1D lines, some architectures factorize space into 2D planes or slices:

  • Inter-slice vs. Intra-slice: Yan et al. proposed AFTer-UNet, which decomposes 3D self-attention into separate inter-slice and intra-slice operations, allowing high-resolution volumetric context to be processed on standard consumer hardware.
  • Orthogonal Planes: For 3D point clouds, Duan et al. introduced ConDaFormer, which decomposes a 3D cubic attention window into three orthogonal 2D planes. Restricting the search space for keys to these planes significantly reduces the number of query-key pairs while preserving essential spatial dependencies.

Decoupling Space and Features

The intuition of decomposition extends beyond spatial dimensions to the relationship between spatial positions and feature channels. In dense networks, computing element-wise attention across both spatial positions ($H \times W$) and channels ($C$) creates a massive, combined problem space.

To make this tractable, Li et al. introduced the Harmonious Attention Network (HA-CNN), which factorizes high-dimensional attention maps into separate spatial and channel components. This allows the network to independently learn where to look (spatial attention) and what features to prioritize (channel attention) before merging the information.

Dense Element-Wise Attention Matrix: [ H × W × C ] × [ H × W × C ]  --> Prohibitive Cost

Factorized Spatial-Channel:          [ H × W ] × [ H × W ]  (Where to look)
                                              ×
                                     [   C   ] × [   C   ]  (What to prioritize)

This separation allows for the integration of attention mechanisms throughout deep architectures without significant increases in memory or processing requirements. For instance, Tan et al. applied this to single-image de-raining in their Separable Element-wise Attention (SEA) mechanism, demonstrating that sequential channel and spatial attention modules achieve the benefits of dense element-wise weighting with significantly reduced computational overhead.

In more complex feature spaces, such as the high-dimensional group spaces used to preserve geometric symmetries, Romero et al. demonstrated in Attentive Group Equivariant Convolutional Networks that factorizing attention into distinct spatial and channel dimensions mitigates the quadratic memory scaling typically associated with large symmetry groups.

A parallel challenge exists in hyperspectral imaging, where data cubes possess a massive spectral dimension alongside spatial coordinates. Meng et al. designed TSA-Net to process spatial and spectral dimensions independently, capturing non-local spatial features and channel-wise dependencies without flattening the high-dimensional data. Similarly, the Spatial-Spectral Transformer (SST) by Li et al. alternates between shifted-window spatial attention and global spectral-wise attention to capture multi-dimensional correlations efficiently.


The Mathematical Intuition: Why Decomposition Works

To see why this works, consider a 3D tensor of shape $H \times W \times D$. If we compute standard self-attention, the number of tokens is $N = H \cdot W \cdot D$. The attention matrix size scales as:

$$\text{Complexity}_{\text{Standard}} = O(N^2) = O(H^2 \cdot W^2 \cdot D^2)$$

If $H=W=D=128$, then $N \approx 2.1 \times 10^6$, and $N^2 \approx 4.4 \times 10^{12}$ operations per attention layer.

By decomposing the attention mechanism into three sequential passes along each individual axis, the model computes attention for a given voxel only along its current line or plane. For an axis-wise decomposition, the complexity becomes:

$$\text{Complexity}_{\text{Decomposed}} = O(H \cdot W \cdot D \cdot (H + W + D))$$

For our $128^3$ volume, this reduces the operation count per layer to roughly $8 \times 10^8$—a reduction of several orders of magnitude. The product in the exponent is effectively replaced by a sum, transforming an intractable quadratic problem into a highly scalable linear one relative to the total volume size.

This mathematical approximation is pushed further by multi-branch architectures. Zhou et al. implemented this parallelization in LIDFormer by splitting self-attention into parallel horizontal, vertical, and channel branches. This triple-path approach approximates full spatial attention by aggregating information along orthogonal axes and feature channels simultaneously, allowing lightweight deployment on resource-constrained edge devices.


Alternative Linearization via Projection

While axis decomposition cuts down the search space geometrically, a closely related branch of linear-complexity volumetric attention achieves a similar reduction via mathematical projection. Instead of isolating coordinate axes, these methods project the high-dimensional key and value spaces into lower-dimensional latent subspaces.

For example, Maaz et al. introduced UNETR++, which features an Efficient Paired Attention (EPA) block. Instead of computing voxel-to-voxel matrices, it projects the key and value matrices into a lower-dimensional space before computing attention scores, capturing global context without the memory overhead of standard self-attention.

In the domain of large-scale aerodynamics and computational fluid dynamics, Ren proposed AeroGTO, which projects geometric data into a learned subspace using basis vectors. This projection mechanism allows global context across massive 3D vehicle meshes to be captured with linear complexity instead of quadratic, demonstrating that whether through spatial factorization or subspace projection, breaking down the joint dimensionality of attention is the key to scaling Transformers to the physical world.

Key Foundational Work

RSANet: Recurrent Slice-wise Attention Network for Multiple Sclerosis Lesion Segmentation (2020) — contributed 3 concept elements to this direction.
Attention Driven Person Re-identification (2018) — contributed 2 concept elements to this direction.
Coupled Rain Streak and Background Estimation via Separable Element-Wise Attention (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decomposed_spatial_channel_attention_5d6b37f4,
  title = {decomposed spatial-channel attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposed-spatial-channel-attention/rdr_5d6b37f410144b07b362aa1f6409b733},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decomposed_spatial_channel_attention_5d6b37f4_v1,
  title = {decomposed spatial-channel attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposed-spatial-channel-attention/rdr_5d6b37f410144b07b362aa1f6409b733/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake