Unsupervised Domain Transfer with Conditional Invertible Neural Networks

Kris K. Dreher, Leonardo Ayala, Melanie Schellenberg, Marco Hübner, Jan-Hinrich Nölke, Tim J. Adler, Silvia Seidlitz, Jan Sellner, Alexander Studier-Fischer, Janek Gröhl, Felix Nickel, Ullrich Köthe, Alexander Seitel, Lena Maier-Hein

Introduction

In medical imaging, training robust deep learning models is often hindered by the lack of large-scale, accurately labeled real-world datasets. While physics-based simulations can generate infinite labeled data, a significant "sim-to-real" domain gap persists, where simulated images fail to capture the nuances of real clinical data. Traditional domain adaptation methods like CycleGANs often suffer from training instability, hallucinations, or mode collapse.

Overview of the proposed domain transfer pipeline, bridging physics-based simulation and real-world data for downstream tasks.

This paper introduces a novel unsupervised domain transfer method using Conditional Invertible Neural Networks (cINNs). By leveraging the mathematical property of invertibility, the proposed framework ensures cycle consistency by design and produces synthetic training data that is more realistic and effective for downstream diagnostic tasks.

Methodology: Conditional Invertible Neural Networks

The core of the approach is the cINN architecture, which maps both simulated and real data into a shared latent space. Unlike standard feed-forward networks, cINNs are designed to be fully reversible.

The cINN architecture featuring shared latent space, domain conditioning, and adversarial training components.

Key Architectural Features

  • Inherent Cycle Consistency: The network's invertibility guarantees that mapping from the simulated domain to the real domain and back is perfectly consistent, eliminating the need for complex cycle-consistency loss functions.
  • Shared Latent Space: Domain-invariant features (e.g., anatomy) are captured in a shared latent space, while domain-specific characteristics (e.g., sensor noise) are controlled via a domain label ($D$).
  • Conditioning: The network is conditioned on both the domain ($D$) and a tissue label ($Y$), ensuring that the biological identity of the data is preserved during the transfer process.
  • Hybrid Training: The model is trained using a combination of Maximum Likelihood (for stable distribution matching) and Adversarial Loss (to enhance visual realism).

Experimental Validation

The method was evaluated across two distinct spectral imaging modalities:

  1. Photoacoustic Tomography (PAT): Focused on image-level transfer using 960 real images from human volunteers and ~1,600 simulated images.
  2. Hyperspectral Imaging (HSI): Focused on pixel-level spectral transfer using ~6.4 million real spectra from porcine organs and ~210,000 simulated spectra.

Results and Insights

The cINN approach demonstrated significant improvements over baseline simulations and existing GAN-based methods like UNIT.

  • Distribution Alignment: Principal Component Analysis (PCA) confirmed that cINN-transferred data distributions aligned much more closely with real-world data than raw simulations.
  • Downstream Task Performance:
    • In PAT artery/vein classification, cINN-transferred data yielded a 37% relative improvement in Balanced Accuracy over raw simulations.
  • In HSI organ classification, the method achieved a 21% relative improvement in Balanced Accuracy and a 33% improvement in F1-score.

The primary innovation lies in using invertibility as a hard constraint. This prevents the network from "hallucinating" features, as every transformation must be reversible. Furthermore, maximum likelihood training provides a more stable alternative to GANs, effectively avoiding mode collapse.

Limitations and Future Outlook

While effective, cINNs are computationally intensive due to their volume-preserving nature, which prevents dimensionality reduction within the network. This makes scaling to high-dimensional 3D volumes challenging. Future research will focus on optimizing these architectures and validating them against physical "phantoms" where ground truth is known with absolute certainty.

Summary updated

Abstract

Synthetic medical image generation has evolved as a key technique for neural network training and validation. A core challenge, however, remains in the domain gap between simulations and real data. While deep learning-based domain transfer using Cycle Generative Adversarial Networks and similar architectures has led to substantial progress in the field, there are use cases in which state-of-the-art approaches still fail to generate training images that produce convincing results on relevant downstream tasks. Here, we address this issue with a domain transfer approach based on conditional invertible neural networks (cINNs). As a particular advantage, our method inherently guarantees cycle consistency through its invertible architecture, and network training can efficiently be conducted with maximum likelihood training. To showcase our method's generic applicability, we apply it to two spectral imaging modalities at different scales, namely hyperspectral imaging (pixel-level) and photoacoustic tomography (image-level). According to comprehensive experiments, our method enables the generation of realistic spectral data and outperforms the state of the art on two downstream classification tasks (binary and multi-class). cINN-based domain transfer could thus evolve as an important method for realistic synthetic data generation in the field of spectral imaging and beyond.
BibTeX
@inproceedings{dreher2023unsupervised,
  author = {Kris K. Dreher and Leonardo Ayala and Melanie Schellenberg and Marco Hübner and Jan-Hinrich Nölke and Tim J. Adler and Silvia Seidlitz and Jan Sellner and Alexander Studier-Fischer and Janek Gröhl and Felix Nickel and Ullrich Köthe and Alexander Seitel and Lena Maier-Hein},
  title = {{Unsupervised Domain Transfer with Conditional Invertible Neural Networks}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43907-0\_73},
}
Figure