When processing three-dimensional data—such as CT scans, MRIs, or geographic point clouds—machine learning practitioners face a fundamental conflict between spatial context and computational complexity.
To understand why this trade-off occurs, consider the geometry of a standard medical volume. A single high-resolution 3D CT scan can easily measure $512 \times 512 \times 512$ voxels, totaling over 134 million data points. Processing this volume with a traditional 3D Convolutional Neural Network (CNN) requires dense 3D kernels (e.g., $3 \times 3 \times 3$). This scales the computational cost and memory consumption cubically ($O(N^3)$). As a result, full 3D models frequently exceed GPU memory limits, run slowly, and risk overfitting when trained on small datasets because of their massive parameter footprints.
The simplest alternative is to slice the volume into independent 2D images. This drops the complexity back to a manageable quadratic scale ($O(N^2)$) and allows models to leverage powerful, pre-trained 2D weights. However, 2D models are "context-blind" along the depth axis. They treat adjacent slices as completely unrelated, destroying the spatial continuity required to trace blood vessels, identify fuzzy tumor boundaries, or preserve structural integrity.
To resolve this dilemma, researchers have developed a spectrum of hybrid architectures. These methods aim to capture the essential 3D volumetric context of an image while maintaining the efficiency and speed of 2D processing.
Compressing the Volume via Multi-Plane Projections
One intuitive way to bypass 3D convolutions is to project the volumetric data onto lower-dimensional 2D planes before feeding it into a network. Instead of forcing a model to learn across a dense 3D grid, the volume is compressed into a set of informative 2D views.
For instance, Jiang et al. introduced an axis-projection strategy in their APAUNet architecture. By squashing 3D medical images into three orthogonal 2D views (axial, sagittal, and coronal), the network can capture structural information from all three dimensions simultaneously without ever computing a 3D convolution. This drastically reduces the ratio of background-to-foreground voxels, making it much easier to detect small targets.
Dense 3D Volume Orthogonal 2D Projections
[ 512 x 512 x 512 ] ---> [Axial] [Sagittal] [Coronal]
(512x512) (512x512) (512x512)
\_______________________________/
Processed via 2D Networks
A similar logic applies to annotation and supervision. Guo et al. demonstrated that 3D vascular segmentation can be effectively supervised using 2D annotations of Maximum Intensity Projection (MIP) images. MIP compresses the volumetric vascular tree into a 2D plane by projecting the highest intensity voxel along a specific viewing axis. This allows a 2D-3D feature fusion network to learn full volumetric structures while saving over 75% of the manual annotation time typically required for voxel-by-voxel 3D labeling.
Injecting Spatial Vision into 2D Networks
Rather than projecting the raw data, another approach keeps the core architecture entirely 2D but introduces lightweight, "plug-and-play" modules that pass information between neighboring slices. This gives 2D networks a "3D perspective" with negligible computational overhead.
Sun et al. designed a Contextual Embedding (CE) block that operates precisely on this principle. The CE block takes a standard 2D network and forces it to match pixel embeddings between adjacent slices. By calculating these inter-slice correlations, the model learns how structures evolve along the z-axis, yielding smooth volumetric segmentations without the memory burden of 3D kernels.
Slice z-1: [ 2D Feature Map ] ---\
+---> [ Context Fusion / Embedding Match ]
Slice z: [ 2D Feature Map ] ---/
Similarly, He proposed Asymmetric 3D Context Fusion (A3D). The author observed that traditional 3D convolutions enforce translation equivariance along the depth dimension, which is computationally expensive and often unnecessary for localized lesion detection. By breaking this symmetry and using an asymmetric fusion operator, 2D-pretrained networks can incorporate multi-slice context while adding almost zero floating-point operations (FLOPs).
Sequential Modeling and 2.5D Slice Stacking
A third paradigm treats the third dimension not as a spatial coordinate, but as a sequence—analogous to video frames or time steps. This leverages the spatial continuity of human anatomy, where a structure visible in slice $z$ is highly likely to continue into slice $z+1$.
Poudel et al. and Bar et al. pioneered this by combining 2D fully convolutional networks with Recurrent Neural Networks (RNNs or LSTMs). The 2D network extracts high-resolution features from individual slices, while the recurrent units propagate a hidden state along the depth axis. This sequential approach captures long-range volumetric dependencies across the entire scan while keeping the GPU memory footprint strictly bounded to the size of a single 2D slice.
For tasks like image denoising, where local context matters more than global sequence tracking, "2.5D" training serves as a middle ground. As demonstrated by Shi et al. in industrial CT imaging, a 2.5D network processes a small, localized stack of adjacent slices rather than a single slice or the entire volume. This provides enough z-axis context to preserve fine structural details and filter out noise, while avoiding the parameter explosion and training delays of full 3D architectures.
Exploiting Geometric and Rotational Symmetries
In many imaging modalities, the computational overhead of 3D processing is exacerbated by the use of standard Cartesian $(x, y, z)$ coordinate systems, which fail to reflect the physical geometry of how the data was acquired.
Yuan et al. bypassed this bottleneck in prostate ultrasound segmentation by treating the 3D volume as a series of rotationally resliced 2D images. By enforcing a mathematically optimized "rotational symmetry" prior across these slices, they guaranteed a smooth 3D surface reconstruction while achieving processing speeds up to 100x faster than traditional 3D volumetric models.
Standard Cartesian Grid (x,y,z) Rotational / Radial Slices
[Prohibitive 3D Convs] [Efficient 2D Convs]
+---------+ \ /
| | \ /
| | *---*
+---------+ / \
/ \
This concept of geometry-aware dimension reduction is also highly effective for artifact removal. Minnema et al. tackled high cone-angle artifacts in circular Cone-Beam CT scans by shifting the deep learning pipeline from Cartesian coordinates to radial slices. Because this transformation directly mirrors the rotational path of the CT scanner, a 2D network can exploit the scanner's inherent geometric symmetry to eliminate complex 3D artifacts, completely avoiding the need for resource-heavy 3D convolutions.
Summary of the Paradigm Shift
The research direction across these works highlights a clear evolution away from monolithic 3D models. Instead, the field has converged on a more nuanced strategy: preserve the efficiency of 2D operations, but engineer targeted pathways—whether through projections, recurrent states, embedding blocks, or geometric transformations—to capture the critical 3D context.
By understanding why 3D convolutions are computationally inefficient (cubic scaling and parameter redundancy) and why 2D models fail (loss of inter-slice continuity), these hybrid approaches successfully bridge the gap, delivering clinical-grade accuracy on standard hardware.
Key Foundational Work
Efficient 3D Endfiring TRUS Prostate Segmentation with Globally Optimized Rotational Symmetry (2013) — contributed 1 concept elements to this direction.
Vessels as 4D Curves: Global Minimal 4D Paths to Extract 3D Tubular Surfaces (2006) — contributed 1 concept elements to this direction.