Volumetric datasets in medical imaging and structural biology—such as CT, MRI, Optical Coherence Tomography (OCT), and Cryo-Electron Tomography (cryo-ET)—frequently exhibit a stark physical asymmetry. While the resolution within a single imaging slice (the in-plane or intra-slice resolution) is highly detailed, the resolution across slices (the through-plane or inter-slice resolution) is often sparse, characterized by large slice thicknesses or physical gaps. This geometric imbalance is known as anisotropy.
When designing neural networks to segment or classify these volumes, standard deep learning architectures present a difficult trade-off:
- Pure 2D networks process each slice independently. They preserve fine-grained in-plane details but completely discard the three-dimensional context, leading to structural inconsistencies across the depth axis.
- Pure 3D networks apply isotropic $3 \times 3 \times 3$ convolutional kernels uniformly across all axes. When applied to highly anisotropic data, they suffer from severe spatial distortion, introduce heavy computational and memory overhead, and tend to overfit to inter-slice discontinuities.
As noted by Yanglan Ou et al. in LambdaUNet: 2.5D Stroke Lesion Segmentation of Diffusion-weighted MR Images, forcing isotropic 3D convolutions onto data with large slice gaps can degrade model performance by introducing noise and interpolation artifacts. To resolve this, a prominent research direction has emerged: hybrid dimensional (or 2.5D) architectures that strategically decouple intra-slice feature extraction from inter-slice context aggregation.
The Core Intuition: Matching Network Geometry to Physical Resolution
The fundamental insight behind hybrid architectures is that a network's operations should mirror the physical sampling rate of its input data.
In the early layers of a convolutional neural network (CNN), the model extracts low-level features like edges, textures, and fine boundaries. If a 3D convolutional kernel is applied immediately to anisotropic data, it mixes high-resolution in-plane pixels with low-resolution, distant out-of-plane voxels. This corrupts the fidelity of the early feature maps.
Pure 3D Convolution (Isotropic Kernel):
[Low-Res Slice t-1] --> oxoxox \
[High-Res Slice t ] --> XXXXXX |--> Blurs high-res features with
[Low-Res Slice t+1] --> oxoxox / low-res axial neighbors
Hybrid 2D-3D Approach:
[High-Res Slice t ] --> 2D Conv --> Sharp Intra-Slice Features \
|--> 3D Fusion (Later Stage)
[High-Res Slice t+1] --> 2D Conv --> Sharp Intra-Slice Features /
Kisuk Lee et al. demonstrated this principle in Recursive Training of 2D-3D Convolutional Networks for Neuronal Boundary Prediction, showing that establishing strong intra-slice representations using 2D convolutions before transitioning to 3D operations prevents axial sparsity from degrading early-stage feature maps. By processing high-resolution planes first, the network preserves fine-grained anatomical details while mitigating the memory constraints and artifacts associated with forced image interpolation.
Architectural Paradigms of Hybrid Networks
To implement this 2D-to-3D transition, researchers have developed several distinct structural patterns within encoder-decoder frameworks.
1. 2D Encoders Paired with 3D Decoders
A widely adopted strategy is to utilize a pure 2D convolutional encoder to extract features from individual slices independently, and then hand those features over to a 3D decoder to enforce volumetric consistency.
Dong Wei et al. leveraged this design in Simultaneous Alignment and Surface Regression Using Hybrid 2D-3D Networks for 3D Coherent Layer Segmentation of Retina OCT Images and its subsequent expansion for sparse annotations, Simultaneous Alignment and Surface Regression Using Hybrid 2D-3D Networks for 3D Coherent Layer Segmentation of Retinal OCT Images with Full and Sparse Annotations. Their architecture maintains fine-grained intra-slice details during encoding while utilizing 3D decoding branches to aggregate information across the volume, ensuring smooth, continuous surface reconstructions.
Similarly, Benjamin Lambert et al. confirmed in Anisotropic Hybrid Networks for liver tumor segmentation with uncertainty quantification that this asymmetric combination consistently outperforms standard isotropic 3D networks when processing medical scans with unequal spatial resolutions across axes.
2. Resolution-Triggered Transitions
Rather than restricting 2D and 3D operations strictly to the encoder or decoder, some architectures transition dynamically based on the spatial scale of the feature maps.
As a network goes deeper and undergoes downsampling, the spatial dimensions of the feature maps shrink while the receptive field expands. This downsampling naturally reduces the ratio of anisotropy, bringing the feature maps closer to a near-isotropic state.
Guotai Wang et al. formalized this in Automatic Segmentation of Vestibular Schwannoma from T2-Weighted MRI by Deep Spatial Attention with Hardness-Weighted Loss. Their architecture utilizes 2D layers for high-resolution in-plane features and switches to 3D layers only after the data has been downsampled to a specific resolution threshold. This optimizes the trade-off between local detail and global volumetric context, avoiding the spatial distortion of non-square voxels in early layers. The authors later extended this philosophy to multi-scale contexts in Semi-Supervised Segmentation of Radiation-Induced Pulmonary Fibrosis from Lung CT Scans with Multi-Scale Guided Dense Attention, using 2D convolutions for fine details and reserving 3D convolutions for coarser scales to capture inter-slice relationships without being biased by large inter-slice gaps.
3. 3D Bottlenecks in 2D Networks
A highly computationally efficient variant isolates 3D operations exclusively within the network's lowest-resolution bottleneck, keeping both the main encoder and decoder paths entirely 2D.
This design was successfully deployed by Haochen Mei et al. in Automatic Segmentation of Gross Target Volume of Nasopharynx Cancer using Ensemble of Multiscale Deep Neural Networks with Spatial Attention and by Florence E. van Lieshout et al. in Deep Learning for Ventricular Arrhythmia Prediction Using Fibrosis Segmentations on Cardiac MRI Data. By embedding 3D convolutions solely within the bottleneck, these models capture the essential global volumetric context and mitigate inconsistent inter-plane voxel spacing while completely bypassing the memory overhead and interpolation artifacts that plague full 3D models.
Hybrid 2D-3D Bottleneck Architecture:
Input Volume ---> [2D Encoder Stages] ---> [3D Bottleneck] ---> [2D Decoder Stages] ---> Output
(Extracts sharp (Aggregates global (Restores fine spatial
in-plane details) 3D context at resolution)
coarse scale)
Conversely, architectures can also sandwich 3D layers between 2D boundaries. Zijie Chen et al. developed OrganNet2.5D in A Novel Hybrid Convolutional Neural Network for Accurate Organ Segmentation in 3D Head and Neck CT Images, which integrates 2D convolutional layers at the absolute input and output stages of a 3D network. This design preserves high-resolution features at the data interface while leveraging volumetric processing internally.
Generalization Across Modalities and Representations
The success of hybrid dimensional architectures spans diverse imaging domains and has been validated by automated design frameworks:
- Cryo-Electron Tomography: In Accurate Detection of Proteins in Cryo-Electron Tomograms from Sparse Labels, Qinwen Huang et al. applied a 2D encoder-decoder to individual tomogram slices before fusing them into a 3D representation. This targeted approach mitigates severe imaging artifacts, such as missing wedge distortion, by focusing initial feature extraction on the planes with the highest signal-to-noise ratio.
- Transformers and Discrete Representations: The hybrid paradigm extends naturally to attention-based models. Xiaomeng Li introduced HResFormer in HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation, which first learns fine-grained details from 2D slices and subsequently refines them with a 3D volumetric context through a residual fusion module. Similarly, Ainkaran Santhirasekaram et al. utilized a hybrid 2D/3D design in Robust prostate disease classification using transformers with discrete representations to capture local details alongside volumetric context for robust prostate cancer classification.
- Neural Architecture Search (NAS): Automated searches confirm that uniform processing is sub-optimal for anisotropic data. In Organ at Risk Segmentation for Head and Neck Cancer Using Stratified Learning and Neural Architecture Search, Dazhou Guo et al. demonstrated that networks discovered via NAS inherently favor a mixture of 2D and Pseudo-3D kernels over uniform 3D kernels, providing an optimal balance between depth context and local detail under strict memory constraints.
Early foundational work, such as the pipeline by Patrick Ferdinand Christ et al. in Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields, relied on cascading 2D fully convolutional networks with 3D post-processing (like Conditional Random Fields) to manage anisotropic clinical data. Modern architectures have evolved this concept into unified, end-to-end networks.
As established by baseline evaluations like the one conducted by Jonathan Shapey et al. in Segmentation of vestibular schwannoma from MRI, an open annotated dataset and baseline algorithm, balancing in-plane and through-plane resolutions via 2.5D convolutions remains a highly effective strategy for managing anisotropic clinical data. To achieve maximum robustness, Hao Li et al. demonstrated in Unsupervised Cross-Modality Domain Adaptation for Segmenting Vestibular Schwannoma and Cochlea with Data Augmentation and Model Ensemble that ensembling 2.5D models (which excel at slice-specific resolution sensitivity) with 3D models (which capture global context) yields a fused output that significantly reduces the risk of under-segmentation and fine-detail loss.
Key Foundational Work
Recursive Training of 2D-3D Convolutional Networks for Neuronal Boundary Prediction (2015) — contributed 2 concept elements to this direction.
Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields. (2016) — contributed 1 concept elements to this direction.
Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation (2016) — contributed 1 concept elements to this direction.