Native Volumetric Representation for 3D Medical Imaging

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,001 papers1,267 conceptsfrozen 2026-10-04

Medical imaging data—such as CT, MRI, and PET scans—is inherently three-dimensional. However, for years, computer vision models treated these volumes as stacks of independent 2D slices. This "slice-by-slice" approach simplifies computation but discards the rich spatial context and anatomical continuity that define human biology.

A growing body of research is shifting toward native 3D volumetric representation. By processing medical data as a single spatial entity rather than a sequence of planes, researchers are building models that understand the true morphology of organs, the connectivity of vascular networks, and the subtle volumetric patterns of disease.

The Cost of Dimensionality Reduction

When a 3D volume is decomposed into 2D slices, the model loses the "depth" dimension. This leads to several critical failures in clinical tasks:

  1. Anatomical Inconsistency: Independent slice processing can result in "staircase artifacts" or discontinuities where structures do not align across the depth axis.
  2. Loss of Global Context: Complex structures, like the branching of a tumor or the curvature of the mandible, cannot be fully captured within a single 2D plane.
  3. Feature Ambiguity: Pathological patterns, such as ground-glass opacities in COVID-19, are often defined by their three-dimensional extent. Qingqiu Li and colleagues demonstrated that representing scans as cohesive 3D volumes is essential for distinguishing these complex patterns from generic inflammation.

Preserving Spatial Continuity

To solve these issues, researchers have developed architectures that operate directly on voxels (3D pixels). Dong Nie et al. pioneered this in medical image synthesis, using 3D Generative Adversarial Networks (GANs) to ensure that synthesized CT scans from MRI data remained anatomically consistent across all three dimensions. Similarly, Pengchong Qiao showed that 3D CNNs are necessary to accurately represent the complex morphology of kidneys and lesions during surgical planning.

This continuity is not just about visual smoothness; it is about functional accuracy. In orthodontic planning, Tomasz Włodarczyk found that 3D-based architectures significantly outperform 2D methods by maintaining the geometric integrity of the jaw and individual teeth.

Bridging Local and Global Context

The challenge of 3D data is its scale. Processing a full volume requires balancing fine-grained local details (like a small fracture) with global context (the entire organ system).

The 3D Brainformer, developed by Rui Nian and others, addresses this by using a "Fusion-Head" attention mechanism. This allows the model to bridge the gap between long-range spatial dependencies and local anatomical details. For classification tasks, Qikui Zhu introduced cross-dimensional Transformers that model dimensional correlations, proving that maintaining 3D integrity is more effective than flattening volumes into 1D sequences.

Toward Generalist 3D Foundation Models

The field is now moving toward "foundation models"—large-scale systems trained on massive datasets to perform diverse tasks. Models like RadFM, introduced by Chaoyi Wu and Weidi Xie, provide native support for 3D data, allowing a single encoder to bridge the gap between different imaging dimensions and modalities.

Self-supervised learning (SSL) is a key driver here. By pretraining on unlabeled 3D volumes, models learn robust representations of human anatomy. Tony Xu and Sepehr Hosseini demonstrated with 3DINO that maintaining full 3D context during SSL produces features that generalize across different organs and scanners far better than 2D-based features.

Overcoming the 3D Computational Burden

The primary hurdle for native 3D processing is the "curse of dimensionality"—the massive memory and compute requirements. Researchers are tackling this through two main strategies:

  1. Efficient Patching and Embedding: Instead of processing every voxel, models like RAPS-3D by Corentin Dancette use native 3D patch embeddings. This treats the volume as a grid of 3D blocks, capturing spatial dependencies across all axes simultaneously without the overhead of slice-by-slice propagation.
  2. 3D-Aware Compression: Maya Varma and the MedVAE team showed that compressing volumes into 3D latent representations can increase throughput by 70x. Crucially, because the compression is 3D-aware, it preserves the spatial continuity needed for tasks like detecting fractures that span multiple slices.

By treating the third dimension as a first-class citizen, these methods are moving medical AI closer to the holistic, volumetric reasoning used by human radiologists.

Papers in this version

Showing 60 of 1,001 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 891 papers
  2. Version 1 · 2026-10-04 · 1001 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_volumetric_spatial_context_238750ee,
  title = {volumetric spatial context},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/volumetric-spatial-context/rdr_238750ee0b5c4f5099eeada6c189c527},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_native_volumetric_representation_for_3d__238750ee_v1,
  title = {Native Volumetric Representation for 3D Medical Imaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/native-volumetric-representation-for-3d-medical-imaging/rdr_238750ee0b5c4f5099eeada6c189c527/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}