Cross-Plane Self-Supervision for Isotropic Volumetric Reconstruction

32 concepts29 papers1 proposal

When capturing 3D volumes in medical imaging or microscopy, researchers face a fundamental physical limitation: resolution anisotropy. While a scanner or microscope can capture incredibly sharp, high-resolution details within a single horizontal slice (the lateral or in-plane dimensions), the resolution across slices (the axial or through-plane dimension) is often significantly degraded. This degradation happens because of physical constraints like patient movement, acquisition time, or the optical physics of lenses.

The traditional deep learning solution to this problem is supervised super-resolution. However, training these models requires pairs of low-resolution and high-resolution 3D volumes. In clinical or specialized laboratory settings, obtaining external high-resolution ground truth data is often impossible.

To solve this, a powerful paradigm has emerged: internal reference training. Instead of looking outward for external datasets, these methods realize that a single anisotropic volume already contains all the information needed to train its own restoration model.


The Core Insight: The Volume as Its Own Atlas

Why can a single volume act as its own training dataset? The answer lies in the statistical and structural redundancy of physical objects.

Consider a 3D biological structure, such as a neuron branching through tissue or a blood vessel winding through a brain. The physical structure itself does not care about the orientation of the scanner's grid; its textures, shapes, and material properties are statistically similar whether viewed from the top, side, or front.

Because the lateral planes ($XY$) are captured at high resolution, they contain a pristine gallery of the sample's specific anatomical features, noise characteristics, and contrast profiles. Early foundational work by Royer et al. in fluorescence microscopy and Carass et al. in magnetic resonance imaging (MRI) capitalized on this property. They proposed that the high-resolution information present in one plane could serve as an internal ground truth to train models for enhancing the lower-resolution orthogonal planes.

Anisotropic 3D Volume
┌─────────────────────────┐ 
│  High-Res XY Slices     │ ───► Extract & Artificially Degrade ───► Create Pseudo-Pairs (LR/HR)
├─────────────────────────┤                                                    │
│  Low-Res Z-Axis Slices  │ ◄─── Apply Trained Model to Restore Isotropic ◄────┘
└─────────────────────────┘

The Mechanism: Pseudo-Pair Generation

To train a neural network without external labels, the internal reference strategy constructs pseudo-pairs directly from the high-resolution dimensions. The pipeline operates in three distinct phases:

  1. Synthetic Degradation: The user extracts sharp 2D slices from the high-resolution lateral plane. The authors then apply a mathematical degradation operator—such as blurring and downsampling—to these slices. This operator is carefully designed to mimic the exact physical degradation process that occurs along the low-resolution axial axis.
  2. Internal Training: This creates a synthetic training pair where the degraded slice is the input and the original sharp slice is the target. A neural network (such as a CNN or a lightweight transformer, as explored by Möller et al.) is trained to invert this degradation.
  3. Inference (Restoration): Once the model learns to restore the synthetic low-resolution images back to their original high-resolution state, it is frozen and applied directly to the actual, physically degraded axial slices.

By framing the problem this way, Ghahremani et al. and Ning et al. demonstrated that 3D electron and fluorescence microscopy images could achieve uniform, isotropic resolution without requiring any external training data. Similarly, Shi et al. extended this concept to CT imaging, explicitly modeling arbitrary slice overlaps to produce sharp sagittal and coronal views.


Why It Works: Eliminating the Domain Gap

A notorious failure mode of standard deep learning models is the domain gap. A model trained to super-resolve lung CT scans from a specific GE scanner will often produce severe artifacts when applied to a kidney scan from a Siemens scanner due to differences in noise, contrast, and tissue properties.

Internal reference training completely bypasses this issue through patient-specific model adaptation. Because the model trains exclusively on the target volume (or fine-tunes on it at test-time), it adapts perfectly to the unique characteristics of that specific scan.

As Remedios et al. and Peng et al. show, leveraging a scan's own in-plane slices ensures that the model learns the exact noise profile and anatomical features of the individual subject. To accelerate this process, Wang et al. introduced a hybrid approach: pre-training a model on large datasets to learn general spatial priors, followed by rapid, subject-specific fine-tuning on the patient's low-resolution scan. This strategy brings reconstruction fidelity remarkably close to supervised gold standards.


Advanced Variations of Internal Supervision

The concept of exploiting internal self-similarity has evolved beyond simple cross-plane downsampling into several sophisticated architectures:

Spatial and Depth-Based Adaptation

Optical distortions are not always uniform across a volume; in microscopy, light scattering increases the deeper you image into tissue. To correct for this, Guo et al. developed a method that utilizes the near-diffraction-limited shallow regions of a 3D stack as a sample-specific ground truth to train a model that corrects the heavily aberrated deep regions.

Cycle Consistency and Self-Training Loops

When cross-plane data is unavailable or multi-modal alignment is required, networks enforce consistency loops. Yoo et al. and Li et al. (2024) demonstrated that models can be trained by enforcing a downsampling consistency constraint: if you downsample your high-resolution prediction, it must exactly match the original low-resolution input.

This idea is pushed further by Li et al. (2025), who combine cross-plane references with dual cycle-consistent diffusion models to simultaneously denoise and super-resolve microscopy volumes. Similarly, Ni et al. use learnable degradation blocks to ensure that the generated outputs remain strictly faithful to the source structures.

Continuous Scale Upsampling

Traditional super-resolution networks are locked into fixed integer scale factors (e.g., $2\times$ or $4\times$). By combining internal self-supervision with Implicit Neural Representations (INRs), Zhang et al. showed that a model trained on a volume's own high-resolution planes can learn a continuous coordinate-based mapping. This allows the volume to be upsampled at any arbitrary, non-integer scale factor.

Impact on Downstream Science

By bypassing the need for external ground truth, internal reference training has unlocked new capabilities in structural discovery. For instance, Schilling et al. applied these self-supervised super-resolution techniques to ultra-high-resolution 7T brain MRIs. This allowed them to resolve micro-structures beyond the physical limits of the scanner, ultimately revealing complex, orthogonal relationships between white matter tract orientations and vascular geometry that were previously obscured by axial blurring.

Key Foundational Work

Self Super-Resolution for Magnetic Resonance Images using Deep Networks (2018) — contributed 2 concept elements to this direction.
Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks (2017) — contributed 2 concept elements to this direction.
Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_internal_reference_training_17be34cf,
  title = {internal reference training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-reference-training/rdr_17be34cf45ef4cf6ad96f46322d9b312},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_reference_training_17be34cf_v1,
  title = {internal reference training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-reference-training/rdr_17be34cf45ef4cf6ad96f46322d9b312/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake