DA-VSR: Domain Adaptable Volumetric Super-Resolution for Medical Images

Cheng Peng, S. Kevin Zhou, Rama Chellappa

TL;DR

DA-VSR is a new framework for medical image super-resolution that bridges the "domain gap" between training and testing data. By leveraging high-resolution in-plane data as a self-supervision signal at test time, it allows a model trained on one organ (like lungs) to adapt itself to others (like kidneys or livers) without needing new ground-truth labels.

What Problem Does This Paper Address?

In clinical practice, MRI and CT scans are often "anisotropic." To save time and reduce costs, machines capture high-resolution images in one plane (in-plane) but leave large gaps between slices (through-plane). This results in "stretched" pixels that make it difficult for doctors and algorithms to analyze anatomy accurately.

While AI models can "super-resolve" these volumes to fill in the gaps, they face a major hurdle: domain inconsistency. A model trained to sharpen lung scans often performs poorly on liver or kidney scans because the textures and structures are different. Furthermore, different scanning machines produce different image characteristics. Most existing models are "brittle"—they work well on data that looks like their training set but fail in the messy reality of diverse clinical environments.

Methods

The authors propose DA-VSR (Domain Adaptable Volumetric Super-Resolution), which combines the power of supervised learning with the flexibility of self-supervised adaptation.

  1. Unified Architecture: Instead of a heavy 3D network, DA-VSR uses a Unified Feature Extraction (UFE) backbone paired with lightweight "heads" for specific tasks:
  2. TPU (Through-Plane Upsampler): The primary head for increasing resolution between slices.
  3. IPU (In-Plane Upsampler): A head used specifically for the adaptation process.
  4. SRN (Slice Refinement Net): A final stage to clean up artifacts and ensure 3D consistency.
  5. Two-Stage Training:
    • Supervised Stage: The model is first trained on a large, labeled dataset (e.g., lung CTs) to learn general features of medical imaging.
  6. Self-Supervised Adaptation Stage: This is the "secret sauce." When the model encounters a new test volume (e.g., a kidney scan), it uses the high-resolution in-plane data of that specific scan to fine-tune its backbone. It essentially "practices" on the parts of the test data it already knows are high-quality to better predict the parts it doesn't.
  7. Gradient Propagation: During adaptation, the authors freeze the task-specific heads and only allow gradients to update the UFE backbone. This prevents the model from "forgetting" how to do super-resolution while allowing it to "learn" the specific textures of the new domain.

Datasets

  • Source/Training: LIDC-IDRI (890 lung CT scans). This served as the "base knowledge" for the model.
  • Target/Testing (Out-of-Domain): To test adaptability, the authors used datasets of different organs:
  • Liver (120 volumes)
    • Colon (30 volumes)
    • Kidney (59 volumes from the KiTS19 challenge)

Key Insights and Ideas

  • In-Plane as a Proxy: The most clever insight is treating the high-resolution in-plane slices as a "free" ground truth. By artificially downsampling these slices and asking the model to reconstruct them, the model learns the specific "style" of the test data (e.g., the specific noise or contrast of a particular CT machine).
  • Stability through Freezing: The authors found that if they didn't freeze the In-Plane Upsampler (IPU) head during adaptation, the model would overfit. Freezing the head forces the backbone to do the heavy lifting of feature alignment, making the adaptation much more stable.
  • Efficiency: By using a slice-based (2D-ish) approach rather than a full 3D-kernel approach, the model avoids the massive memory bottlenecks that plague other volumetric SR methods, allowing it to process larger volumes more efficiently.

Key Results

DA-VSR consistently outperformed state-of-the-art methods (like SAINT, SMORE, and 3D-RCAN) across all "unseen" organ datasets.

  • Quantitative Gains: On the Kidney dataset (X4 scale), DA-VSR achieved a PSNR of 38.07 dB, significantly higher than the non-adapted version (37.60 dB) and the previous SOTA, SAINT (37.70 dB).
  • Qualitative Improvements: Visual results showed that DA-VSR was much better at reconstructing complex, small-scale structures like the spine or fine vessels. While other self-supervised methods (like SMORE) tended to produce overly smooth or "blurry" results, DA-VSR maintained sharp, anatomically plausible details.
  • Convergence: The refinement stage of DA-VSR converged much faster than previous models because it reused features from the already well-trained backbone.

Limitations and Future Work

  • Anisotropy Assumption: The method relies on the existence of high-resolution in-plane data. If a scan is low-resolution in all directions, the self-supervised adaptation trick won't work.
  • Domain Scope: While the paper proves adaptation across different organs, it doesn't extensively explore "cross-modality" adaptation (e.g., training on CT and adapting to MRI), which the authors suggest as a future research direction.
  • Computation at Test-Time: Because the model performs adaptation at test time (fine-tuning for 5-10 epochs per volume), it is slower than a simple "plug-and-play" inference model, though the authors argue this is a necessary trade-off for clinical accuracy.

Summary updated

Abstract

Medical image super-resolution (SR) is an active research area that has many potential applications, including reducing scan time, bettering visual understanding, increasing robustness in downstream tasks, etc. However, applying deep-learning-based SR approaches for clinical applications often encounters issues of domain inconsistency, as the test data may be acquired by different machines or on different organs. In this work, we present a novel algorithm called domain adaptable volumetric super-resolution (DA-VSR) to better bridge the domain inconsistency gap. DA-VSR uses a unified feature extraction backbone and a series of network heads to improve image quality over different planes. Furthermore, DA-VSR leverages the in-plane and through-plane resolution differences on the test data to achieve a self-learned domain adaptation. As such, DA-VSR combines the advantages of a strong feature generator learned through supervised training and the ability to tune to the idiosyncrasies of the test volumes through unsupervised learning. Through experiments, we demonstrate that DA-VSR significantly improves super-resolution quality across numerous datasets of different domains, thereby taking a further step toward real clinical applications.
BibTeX
@inproceedings{peng2021da,
  author = {Cheng Peng and S. Kevin Zhou and Rama Chellappa},
  title = {{DA-VSR: Domain Adaptable Volumetric Super-Resolution for Medical Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=yOJo7znumN},
}
Figure