Much of modern computer vision is built on a flat assumption: that the world can be understood through two-dimensional pixel grids. This assumption works remarkably well for natural images, where legacy architectures like Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have mastered spatial patterns. However, when these 2D models are applied to domains that are inherently three-dimensional—such as medical CT/MRI scans, LiDAR point clouds, or astronomical spectral cubes—they encounter a fundamental dimensionality barrier.
To process 3D data efficiently, practitioners traditionally slice the volume into a stack of independent 2D images. While computationally convenient, this slice-by-slice approach discards the rich, out-of-plane spatial context that defines the true geometry of the object. Overcoming this 2D limitation is not just a matter of adding an extra axis; it requires rethinking how models learn spatial dependencies, how architectures scale computationally, and how algorithms evaluate multi-dimensional structures.
The Illusion of the Third Dimension
When a 3D volume is treated as a sequence of independent 2D slices, the model becomes blind to continuity along the perpendicular axis (the $z$-axis). Imagine trying to segment a blood vessel or a nerve fiber that snakes vertically through a tissue sample. A 2D model only sees a disconnected series of circles appearing and disappearing across slices. It has no mechanism to understand that a pixel at coordinates $(x, y)$ on slice $z$ is physically connected to the pixel at $(x, y)$ on slice $z+1$.
As Amir Gholami et al. point out in their work on medical image segmentation, A Novel Domain Adaptation Framework for Medical Image Segmentation, operating purely on 2D slices prevents networks from leveraging crucial volumetric information that native 3D networks could utilize to resolve structural ambiguities.
This spatial blindness extends beyond segmentation to high-level reasoning. For instance, large Vision-Language Models (VLMs) trained predominantly on 2D images struggle significantly when tasked with interpreting clinical orientations. In Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images, Daniel Wolf demonstrates that evaluating VLMs on isolated 2D slices of CT scans fails to capture the spatial complexities required for accurate radiological analysis, causing models to rely on memorized anatomical priors rather than true visual understanding.
2D Slice-by-Slice Processing (Blind to Z-Axis)
Slice z-1: [ O ] <- Isolated circle
Slice z: [ O ] <- Isolated circle
Slice z+1: [ O ] <- Isolated circle
Native 3D Volumetric Processing (Captures Continuity)
Volume: [===] <- Continuous cylinder/vessel recognized across space
The Computational Bottleneck of Going 3D
If 2D processing is so limiting, why not simply swap 2D operators for 3D ones? The obstacle is the curse of dimensionality.
As an architecture expands from 2D to 3D, its computational complexity scales cubically rather than quadratically. A standard $N \times N$ image requires $O(N^2)$ operations, but an $N \times N \times N$ volume demands $O(N^3)$. For self-attention mechanisms in Transformers, which scale quadratically with the number of tokens, this explosion makes native 3D attention prohibitively expensive for standard hardware.
Consequently, researchers are forced to design clever intermediate representations that capture 3D context without incurring full cubic costs. For example, Abbas Khan et al. developed a highly efficient 2D cardiac segmentation model in CAMS: Convolution and Attention-Free Mamba-based Cardiac Image Segmentation, yet the authors explicitly note that extending such architectures to native 3D processing remains an urgent next step to capture true volumetric physiology.
One prominent strategy for bridging this gap is the tri-plane architecture. Instead of processing a massive 3D voxel grid, the volume is projected onto three orthogonal 2D planes: Axial ($XY$), Sagittal ($YZ$), and Coronal ($XZ$).
In Tri-Plane Mamba: Efficiently Adapting Segment Anything Model for 3D Medical Images, Hualiang Wang et al. utilize this exact mechanism to adapt the 2D Segment Anything Model (SAM) for 3D volumes. By combining multi-scale 3D convolutions with a tri-plane structure, they extract cross-plane spatial dependencies at a fraction of the computational cost of full 3D attention. However, as the authors observe, adapting only the image encoder leaves prompt-based interactive segmentation unaddressed in 3D space, highlighting how deeply entrenched 2D assumptions remain in model interfaces.
Discontinuity in Boundaries and Features
The mathematical formulations of many classical vision algorithms also break down when confined to two dimensions. When active contour models (snakes) or edge-detection filters operate slice-by-slice, they generate fragmented boundaries. An organ boundary might appear sharp and distinct in an axial slice but possess complex, high-curvature structures along the z-axis that the 2D filter cannot smooth or regularize.
In A New Active Contour Method Based on Elastic Interaction, Yang Xiang et al. demonstrate how treating image boundaries as physical dislocations can snap contours to complex shapes. Yet, the authors emphasize that extending these planar mathematical frameworks to 3D stacks requires substantial extension to handle the true topological complexity of three-dimensional organs.
Similarly, deep learning models that use localized attention often fail at boundary delineation when restricted to flat inputs. In FocalUNETR: A Focal Transformer for Boundary-Aware Prostate Segmentation Using CT Images, Chengyin Li et al. introduce a focal self-attention mechanism to resolve ambiguous prostate boundaries. Because their attention mechanism was initially designed for 2D inputs, the authors identify the transition to native 3D volumetric data as the critical path forward to fully exploit spatial context and eliminate inter-slice boundary artifacts.
The Need for Native 3D Datasets and Metrics
The 2D dimensionality constraint is not merely an architectural flaw; it is a systemic issue embedded within our datasets and evaluation metrics.
Many multi-task medical foundation models are trained on massive collections of X-rays because image-centric, multi-annotation data is readily available. However, as Xun Zhu et al. note in Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Data, this heavy reliance on 2D modalities severely limits a model's applicability to volumetric diagnostics like CT or MRI scans. To build a true generalist medical AI, the data pipelines themselves must be extended to handle dense annotations directly within 3D space.
Furthermore, our evaluation metrics frequently suffer from the same dimensional flattening. Standard object detection metrics like mean Average Precision (mAP) rely on 2D bounding boxes, which fail to capture the volumetric overlap, shape fidelity, and orientation of 3D structures.
To address this, Felix Meissen introduced the Robust Detection Outcome (RoDeO) metric in Robust Detection Outcome: A Metric for Pathology Detection in Medical Images. While RoDeO improves on traditional metrics by decomposing performance into localization and shape similarity, the author notes that standard bounding-box frameworks remain fundamentally unequipped for higher-dimensional data types, underscoring the open challenge of developing evaluation paradigms that match the 3D reality of clinical and physical data.
The transition from 2D slice-by-slice processing to native 3D volumetric analysis represents a major paradigm shift. By moving past flat representations, the next generation of vision models can finally perceive the world with the spatial continuity and geometric depth it naturally possesses.