In computer vision, moving from 2D to 3D is often viewed as a natural progression toward richer context. However, in clinical medical imaging, the "upgrade" to 3D convolutional neural networks (CNNs) frequently leads to worse performance, higher computational costs, and increased sensitivity to noise.
The reason lies in the physical reality of how medical data is acquired. While we think of CT and MRI as volumes, they are often stored as "thick-slice" scans where the resolution within a slice (in-plane) is much higher than the resolution between slices (through-plane). This spatial inconsistency, known as anisotropy, fundamentally changes how neural networks should "look" at the data.
The Anisotropy Problem
In an ideal world, a voxel (a 3D pixel) would be a perfect cube. In reality, clinical scans are often highly anisotropic. For example, a cardiac MRI might have a resolution of 1mm x 1mm within a slice but a 10mm gap between slices.
When a 3D convolutional kernel slides across these gaps, it attempts to extract features from a "blurred" or "smeared" dimension. As Lisa M. Koch and Christian F. Baumgartner demonstrate in An Exploration of 2D and 3D Deep Learning Techniques for Cardiac MR Image Segmentation, 3D context becomes a hindrance rather than a help when inter-slice misalignment and low through-plane resolution are present. The 3D kernel essentially "sees" noise where there should be anatomical structure.
Why 2D Often Wins
To achieve high accuracy on clinical data, researchers frequently revert to 2D architectures that process volumes slice-by-slice. This approach offers several intuitive advantages:
- Preserving High-Resolution Features: By using large 2D spatial patches, models can capture intricate in-plane details that would be lost if forced into smaller 3D kernels. Snehashis Roy and colleagues utilized this strategy in FLEXCONN to achieve human-level accuracy in MS lesion segmentation, prioritizing high-resolution axial features over unreliable depth information.
- Avoiding Interpolation Artifacts: Attempting to "fix" anisotropy by interpolating slices into an isotropic volume often introduces blurring. Jianhua Yao 0001 and Ronald M. Summers found that restricted 2D analysis within the high-resolution acquisition plane avoids the artifacts introduced by reformatting thick-slice data (Detection of Sclerotic Spine Metastases...).
- Signal Dilution: In pathology grading, 3D volumetric regions of interest (ROIs) can actually be less robust than 2D "hotspot" measurements. As Koji Takano and Manabu Kinoshita show, 3D volumes may incorporate non-representative tissue that dilutes the quantitative signal of a lesion (Influence of region‑of‑interest designs...).
The 2.5D Middle Ground
If 3D is too noisy and 2D lacks context, the "2.5D" approach offers a compromise. These architectures leverage 3D spatial context—such as looking at a few adjacent slices—while using efficient 2D-based operations.
Yichi Zhang 0007 and Jicong Zhang argue in Exploring Efficient Volumetric Medical Image Segmentation Using 2.5D Method that these models match 3D performance at a fraction of the cost. Similarly, Fausto Milletarì and Seyed‐Ahmad Ahmadi suggest that the dimensionality of input patches must balance anatomical context against modality-specific artifacts, such as the noise patterns found in ultrasound (Hough-CNN).
Stability and Generalization
In clinical practice, robustness is paramount. Models trained on 2D slices tend to be more stable across different scanners and centers. Enamundram Naga Karthik and Jan Valošek found that simple 2D slice-by-slice models outperformed complex 3D approaches for spinal cord lesion segmentation because they avoided the feature blurring that occurs in the gaps between thick slices (Automatic segmentation of spinal cord lesions in MS).
By prioritizing the high-resolution plane where the data is most reliable, 2D and 2.5D models mitigate the loss of spatial coherence inherent in clinical acquisition. As Jun Ma demonstrates in Cascaded Framework for Automatic Evaluation of Myocardial Infarction, when data is sparse along the longitudinal axis, the spatial context within a single plane is simply more trustworthy than the information between slices.