Orthogonal Multi View Fusion for Volumetric Imaging

105 concepts81 papers1 proposal

When dealing with 3D medical images like MRI or CT scans, deep learning practitioners face a fundamental architectural dilemma: the curse of dimensionality. Volumetric data contains crucial spatial context across all three dimensions, but processing it with native 3D convolutional neural networks (CNNs) is incredibly expensive. 3D convolutions scale cubically in memory and computation, severely limiting input resolution. Furthermore, because 3D clinical datasets are often small, these high-parameter models are highly prone to overfitting.

To bypass these limitations, researchers developed orthogonal multi-view fusion (often called a 2.5D representation). Instead of treating a 3D volume as a single massive cube, this paradigm decomposes the volume into slices along its three orthogonal anatomical planes: axial (top-down), sagittal (side-to-side), and coronal (front-to-back). By processing these 2D views independently and fusing their representations, models can approximate full 3D spatial context with a fraction of the computational cost.

The Geometric Intuition: Why 2.5D Suffices

Why can we get away with using 2D networks for 3D data? The answer lies in projective geometry.

If a model looks at a 3D volume through a single 2D plane (e.g., only axial slices), it is blind to out-of-plane continuity, causing thin or complex structures to appear fragmented. However, any point in a 3D volume is uniquely localized by the intersection of three orthogonal planes. By observing a structure simultaneously from the top, side, and front, a network can resolve spatial ambiguities.

As noted by Mohammad Havaei et al. in Deep Learning Trends for Focal Brain Pathology Segmentation in MRI, this multi-planar approach effectively approximates 3D spatial context without the high memory requirements of full 3D volumes. This is particularly powerful for anisotropic medical images where the spatial resolution varies significantly across different axes.

Decomposing volumes into 2D slices also provides two massive practical advantages:

  1. Data Efficiency and Regularization: A single 3D volume yields hundreds of 2D slices, drastically increasing the effective size of the training set. Se Jin Cho et al. demonstrate in Prediction of treatment response after stereotactic radiosurgery of brain metastasis using deep learning and radiomics on longitudinal MRI data that processing orthogonal 2D slices acts as a regularization technique, reducing the input space dimensionality to prevent overfitting when training data is scarce.
  2. Transfer Learning: Native 3D models cannot easily use pre-trained weights because large-scale 3D datasets do not exist. By contrast, 2.5D approaches can exploit powerful models pre-trained on millions of 2D images. Hoo-Chang Shin et al. demonstrated in Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning that "off-the-shelf" 2D vision models pre-trained on ImageNet can be successfully adapted to volumetric medical tasks by aggregating orthogonal views. Taking this further, Ulzee An et al. show in Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models that aggregating features from three orthogonal planes allows frozen 2D foundation models to capture sufficient 3D context for complex tasks without requiring any 3D training at all.

Architectural Paradigms of Fusion

How exactly do we combine information from different planes? The literature generally splits multi-view fusion into two main strategies: Late Fusion (Decision-Level) and Early/Mid Fusion (Feature-Level).

Late Fusion: Consensus and Voting

In a late fusion setup, completely independent 2D networks are trained on the axial, sagittal, and coronal slices. During inference, each network generates its own prediction or probability map, and these maps are aggregated to form the final 3D output.

                  ┌──► Axial Network ───► Axial Prediction ───┐
                  │                                           ▼
3D Volume Slices ─┼──► Sagittal Network ─► Sagittal Prediction ─┼─► Multi-View Aggregation ─► Final 3D Output
                  │                                           ▲
                  └──► Coronal Network ──► Coronal Prediction ──┘

This aggregation can take several forms:

Early and Mid Fusion: Joint Feature Spaces

While late fusion is simple and highly parallelizable, it prevents the model from learning interactions between the planes during feature extraction. Early and mid-fusion architectures solve this by combining multi-view information directly within the network's hidden layers.

One common approach is to stack orthogonal slices or regions of interest directly into the input channels of a standard 2D network, as explored by the authors of M2Fusion: Bayesian-based Multimodal Multi-level Fusion on Colorectal Cancer Microsatellite Instability Prediction. This forces the very first convolutional layers to find correlations across the different planes.

Alternatively, multi-branch networks process each view through its own encoder branch before merging the feature maps at a central bottleneck. For instance, the architecture proposed in Btrfly Net: Vertebrae Labelling with Energy-Based Adversarial Learning of Local Spine Prior routes orthogonal 2D projections through symmetric network branches, fusing them at a high-dimensional bottleneck to capture the strict inter-dependency between views. This allows for precise 3D localization without the cubic scaling of 3D kernels.

Enforcing 3D Consistency

A common critique of 2.5D methods is that because the networks operate on 2D slices, the final reconstructed 3D volume might suffer from "slice-stacking artifacts"—unnatural discontinuities or jagged edges along the boundaries where independent slices meet.

To combat this, modern multi-view fusion frameworks incorporate explicit spatial regularization across planes. For example, Shuwei Zhai et al. show in PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation that enforcing 2D conditional random field (CRF) losses simultaneously across the axial, sagittal, and coronal planes forces the model to maintain 3D spatial consistency, bypassing the need for expensive 3D affinity matrices.

Similarly, in generative modeling, George Webber et al. demonstrate in Likelihood-Scheduled Score-Based Generative Modeling for Fully 3D PET Image Reconstruction that aggregating gradients from multiple 2D generative models trained on different orthogonal planes effectively enforces 3D consistency during volumetric reconstruction, eliminating slice-stacking artifacts entirely while keeping memory usage minimal.

Ultimately, orthogonal multi-view fusion works because it exploits the structured, geometric nature of medical imaging. By recognizing that three intersecting 2D perspectives contain nearly all the information required to reconstruct a 3D relationship, it provides a highly efficient, regularized, and foundation-model-friendly alternative to full 3D architectures.

Key Foundational Work

Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning (2016) — contributed 1 concept elements to this direction.
Deep learning trends for focal brain pathology segmentation in MRI (2016) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 76 papers
  2. Version 1 · 2026-10-04 · 111 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_orthogonal_multi_view_fusion_ed7a5827,
  title = {orthogonal multi-view fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-multi-view-fusion/rdr_ed7a5827562544b4ab4d8c61d16f22e1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_orthogonal_multi_view_fusion_ed7a5827_v2,
  title = {orthogonal multi-view fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-multi-view-fusion/rdr_ed7a5827562544b4ab4d8c61d16f22e1/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake