Hybrid Convolutional Architectures for Anisotropic 3D Image Segmentation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 225 papers39 conceptsfrozen 2026-10-04

In the ideal world of computer vision, a 3D volume is a uniform grid of voxels—perfect cubes with equal resolution in every direction. In the real world of biomedical imaging, however, we rarely have this luxury. Whether it is a CT scan, an MRI, or electron microscopy (EM), the data is almost always anisotropic: the resolution within a single slice (in-plane) is significantly higher than the resolution between slices (through-plane).

Standard 3D convolutional neural networks (CNNs) struggle with this disparity. By applying symmetric $3 \times 3 \times 3$ kernels, they treat the low-resolution depth axis the same as the high-resolution spatial axes. This leads to two major problems: it wastes computation on sparse information and, more importantly, it blurs fine-grained features by mixing them with low-resolution context too early.

To solve this, a growing body of research advocates for hybrid dimensionality. Instead of choosing between 2D or 3D, these architectures strategically combine both to respect the physical properties of the data.

The Resolution Gap

In clinical head CT scans, for example, the slice thickness might be several millimeters, while the in-plane pixels are sub-millimeter. As noted in The state-of-the-art 3D anisotropic intracranial hemorrhage segmentation on non-contrast head CT: The INSTANCE challenge by Xiangyü Li and colleagues, this anisotropy is a primary hurdle for segmenting complex structures like hemorrhages.

If you use a pure 2D approach, you lose the 3D continuity needed to ensure an organ looks like a coherent object across slices. If you use a pure 3D approach, the network’s receptive field becomes physically distorted—it "sees" a much larger physical area in the depth dimension than in the width or height, leading to structural inconsistencies.

Extracting Before Integrating

The most successful hybrid strategy is to prioritize high-resolution feature extraction before attempting volumetric integration.

Kisuk Lee, H. Sebastian Seung, and their team demonstrated this in Recursive Training of 2D-3D Convolutional Networks for Neuronal Boundary Prediction. Their architecture uses initial 2D layers to establish strong intra-slice representations. Only in the deeper layers, after the features are well-defined, does the network transition to 3D convolutions to capture the axial context.

A similar intuition drives the work of Dong Wei and Hong Liu in Simultaneous Alignment and Surface Regression Using Hybrid 2D-3D Networks for 3D Coherent Layer Segmentation of Retina OCT Images. They utilize 2D encoders to maintain fine-grained details within retinal B-scans, while employing 3D decoders to ensure the resulting surfaces are smooth and continuous across the entire volume.

Strategic Transitions and Resolution Equalization

When exactly should a network switch from 2D to 3D? The most effective models time this transition based on the data's physical scale.

In Synaptic Cleft Segmentation in Non-Isotropic Volume Electron Microscopy of the Complete Drosophila Brain, Jan Funke and Constantin Pape introduced the DTU-2 architecture. This model uses 2D operations in the early stages and only switches to 3D after pooling operations have reduced the in-plane resolution to match the through-plane resolution. By equalizing the representation's scale, they ensure the network's physical field of view remains isotropic, preventing spatial distortion.

Guotai Wang and Shuwei Zhai applied a similar logic in Semi-Supervised Segmentation of Radiation-Induced Pulmonary Fibrosis From Lung CT Scans With Multi-Scale Guided Dense Attention. Their PF-Net transitions to 3D only when the receptive field is balanced, allowing the model to capture volumetric context without sacrificing the high-resolution details of the initial lung scans.

Alternatives to 3D Convolutions

Sometimes, the best way to handle the third dimension isn't a 3D convolution at all. Recurrent layers offer a computationally efficient way to pass information between slices without the heavy memory footprint of 3D kernels.

Jianxu Chen and Lin Yang pioneered this in Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation. By pairing a 2D FCN with a Bi-Directional Convolutional LSTM, they allowed the model to "remember" features from previous and subsequent slices. This respects the anisotropy by treating the depth axis as a sequence, resulting in better structural consistency than standard 3D CNNs.

Automating the Hybrid Balance

Given the variety of medical imaging modalities, the "perfect" mix of 2D and 3D operations varies from task to task. This has led to the use of Neural Architecture Search (NAS) to automate the design.

In Organ at Risk Segmentation for Head and Neck Cancer Using Stratified Learning and Neural Architecture Search, Dazhou Guo and Dakai Jin developed SOARS. Their system searches for an optimal combination of 2D, 3D, and "Pseudo-3D" kernels (which approximate 3D effects using 2D filters). Their findings confirm that uniform 3D kernels are rarely optimal; instead, the most accurate models are those that adapt their dimensionality to the specific resolution and difficulty of the organ being segmented.

By moving away from the "one-size-fits-all" 3D approach, these hybrid architectures allow researchers to build models that are not only more accurate but also more grounded in the physical reality of how medical data is acquired.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 23 papers
  2. Version 1 · 2026-10-04 · 25 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_anisotropy_aware_feature_extraction_846ed63b,
  title = {anisotropy-aware feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anisotropy-aware-feature-extraction/rdr_846ed63ba4694131bc469954355bc687},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_convolutional_architectures_for_a_846ed63b_v1,
  title = {Hybrid Convolutional Architectures for Anisotropic 3D Image Segmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-convolutional-architectures-for-anisotropic-3d-image-segmentation/rdr_846ed63ba4694131bc469954355bc687/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}