Orthogonal Multi-View Fusion for Volumetric Medical Imaging

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2111 papers144 conceptsfrozen 2026-10-04

Medical imaging presents a fundamental computational paradox. While the human body is inherently three-dimensional, the neural networks we use to analyze it often struggle with the "curse of dimensionality." Native 3D convolutional neural networks (CNNs) are notoriously memory-intensive, computationally expensive, and prone to overfitting due to the relative scarcity of annotated volumetric datasets.

To bypass these bottlenecks, a robust research direction has emerged: Orthogonal Multi-View Fusion (often referred to as 2.5D learning). Instead of processing a 3D volume as a single block, these methods decompose the volume into 2D slices along the three cardinal anatomical planes—axial, coronal, and sagittal—and then fuse the results to reconstruct 3D context.

The Intuition: 3D Context from 2D Perspectives

The core insight behind this approach is that a 3D structure can be uniquely and efficiently described by its projections or slices across different orientations. As noted in Deep Learning in Medical Image Analysis by Dinggang Shen et al., capturing volumetric information through orthogonal 2D views provides a computationally efficient alternative to full 3D analysis while maintaining high accuracy.

By training models on 2D slices, researchers gain two immediate advantages:

  1. Data Augmentation: A single 3D volume provides hundreds of 2D training samples, which helps prevent the overfitting common in small medical datasets.
  2. Memory Efficiency: 2D kernels allow for higher input resolution and deeper architectures on standard hardware, whereas 3D kernels often force researchers to downsample images or use small, local patches.

Mechanisms of Fusion

The research field has developed several distinct ways to "stitch" these 2D perspectives back into a coherent 3D understanding.

1. Late Aggregation and Voting

The simplest form of fusion involves running independent 2D models on each plane and combining their predictions at the end. In QuickNAT: Segmenting MRI Neuroanatomy in 20 seconds, Abhijit Guha Roy and colleagues demonstrate that aggregating predictions from three orthogonal views provides sufficient spatial context to segment complex brain structures in seconds. Similarly, Zhoubing Xu and Camilo Bermudez used a union operation and morphological cleaning in SSNet to segment enlarged spleens, finding that this multi-view approach mitigates errors specific to any single anatomical orientation.

2. Multi-Planar Stacking and Ensembles

Some architectures process the views in parallel branches before merging them into a final decision layer. Jakob Wasserthal and Fabian Isensee utilized a Stacked U-Net where separate networks process axial, coronal, and sagittal views, with a final network integrating these concatenated predictions to reconstruct white matter bundles. This "consensus" mechanism allows one view to compensate for artifacts or ambiguities that might be present in another.

3. Feature-Level Fusion

More sophisticated models fuse information at the latent feature level. Anjany Sekuboyina introduced Btrfly Net, which processes sagittal and coronal projections through symmetric branches and fuses them at a high-dimensional bottleneck. This captures the inter-dependency between views, allowing for accurate 3D localization with quadratic rather than cubic computational scaling.

Beyond Segmentation: Generative and Super-Resolution Tasks

The multi-planar strategy has recently expanded into generative modeling and image translation. In the realm of diffusion models, George Webber and Andrew P. King showed that averaging "scores" (gradients) from 2D models trained on orthogonal planes effectively enforces 3D spatial consistency in PET reconstructions.

This approach is particularly powerful for Super-Resolution. Zejun Wu and Jerry L Prince developed AniRes2D, which processes slices independently across planes to reconstruct high-resolution volumes from anisotropic (low-res) scans. By averaging the outputs, they maintain spatial consistency across the volume without the massive overhead of 3D diffusion.

Why It Works: The "Pseudo-3D" Effect

The success of orthogonal fusion lies in its ability to create a "pseudo-3D" context. While a single 2D slice lacks depth, the simultaneous consideration of three perpendicular slices constrains the possible 3D geometry. As Yuyin Zhou and Wei Shen argue in their work on Multi-Planar Co-Training, forcing models to "agree" across different anatomical views acts as a powerful regularizer. This is especially evident in semi-supervised learning, where multi-view consistency allows models to generate high-quality pseudo-labels for unlabeled data, often outperforming fully supervised 3D networks.

In summary, Orthogonal Multi-View Fusion treats 3D volumes not as a single daunting cube of data, but as a collection of intersecting perspectives. By reconciling these views, researchers achieve the spatial fidelity of 3D models with the efficiency and robustness of 2D deep learning.

Papers in this version

Showing 60 of 111 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 76 papers
  2. Version 1 · 2026-10-04 · 111 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_orthogonal_multi_view_fusion_ed7a5827,
  title = {orthogonal multi-view fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-multi-view-fusion/rdr_ed7a5827562544b4ab4d8c61d16f22e1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_orthogonal_multi_view_fusion_for_volumet_ed7a5827_v1,
  title = {Orthogonal Multi-View Fusion for Volumetric Medical Imaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-multi-view-fusion-for-volumetric-medical-imaging/rdr_ed7a5827562544b4ab4d8c61d16f22e1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}