What can we learn about a generated image corrupting its latent representation?

Agnieszka Tomczak, Aarushi Gupta, Slobodan Ilic, Nassir Navab, Shadi Albarqouni

Abstract

Generative Adversarial Networks (GANs) have become a cornerstone of medical image-to-image translation, yet their tendency to "hallucinate" anatomical features poses a significant risk in clinical settings. This paper proposes a "stress test" for GANs by injecting noise into their latent representations. By measuring output fluctuations—specifically variance and mutual information—researchers can quantify uncertainty and identify unreliable features. Our findings show that this method not only flags corrupted inputs but also improves synthesis quality when integrated into the training process.

Introduction

In the domain of medical imaging, GANs are frequently employed to translate between modalities, such as synthesizing a CT scan from an MRI. However, the black-box nature of these models leads to a critical issue of trust: models may omit real pathology or invent non-existent structures to satisfy learned patterns. Existing uncertainty estimation techniques often require architectural overhauls or computationally expensive optimizations. This research introduces a "plug-and-play" noise injection technique to evaluate the robustness of latent representations, providing a metric for the reliability of generated outputs.

Methodology: The Latent Stress Test

The core of the proposed approach is a noise injection technique targeting the GAN's bottleneck—the latent representation ($h$). The process involves three steps:

  1. Corruption: An input image is encoded into a latent code $h$. Gaussian noise is then added to create $k$ variations of this code.
  2. Generation: These $k$ noisy codes are passed through the decoder, resulting in a set of slightly different output images.
  3. Quantification: Two primary metrics are derived:
  4. Variance ($\gamma$): Measures the pixel-wise difference among the $k$ outputs. High variance indicates the model is "guessing" in specific regions.
  5. Mutual Information ($\delta$): Acts as a confidence score by measuring how much information is preserved between the original output and its noisy counterparts.

Key Insights: Visualizing the Unknown

A primary advantage of this method is the ability to generate "uncertainty heatmaps." By intentionally perturbing the input image (e.g., blacking out a patch), the authors demonstrated that the resulting variance and confidence scores accurately flag these regions as highly uncertain.

Visualizing uncertainty through heatmaps generated by latent corruption on perturbed inputs across various modalities.

In a clinical workflow, these heatmaps could serve as an automated "red flag" system, alerting radiologists to specific areas in a synthetic image that may be hallucinations. Furthermore, the study found that injecting a small amount of noise ($\alpha = 0.5$) during the training phase acts as a regularizer. This forces the encoder to learn more stable and meaningful latent codes, ultimately improving the Fréchet Inception Distance (FID) scores of the generated images.

Key Results and Quantitative Validation

The effectiveness of the metrics was validated using the CHAOS19 and LiTS datasets. In a "sanity check" involving CT to T1 translation, the variance ($\gamma$) increased from 0.0567 for original images to 0.0676 for perturbed images. Simultaneously, the confidence score ($\delta$) showed a marked decrease, confirming that the metrics are sensitive to input errors.

Distribution of confidence scores for original vs. corrupted samples across CT, T1, and T2 modalities, showing a clear separation.

Beyond visual quality, the researchers explored the correlation between latent robustness and downstream task performance. For successful translations, the proposed confidence score ($\delta$) correlated significantly (often $>0.5$) with the DICE coefficient of a subsequent liver segmentation task. This suggests that the confidence score can serve as a proxy for whether a synthetic image is of sufficient quality for further automated analysis.

Conclusion and Future Directions

While noise injection improves overall image quality and provides a robust uncertainty metric, it is not a panacea. The authors noted that increased robustness during training can sometimes lead to a loss of fine-grained detail necessary for precise boundary detection in segmentation tasks. Additionally, the correlation between confidence and accuracy is less reliable when the base image quality is very low. Future research will investigate targeted adversarial attacks to further probe the limits of these latent representations and validate the approach in broader clinical scenarios.

Summary updated

Abstract

Generative adversarial networks (GANs) offer an effective solution to the image-to-image translation problem, thereby allowing for new possibilities in medical imaging. They can translate images from one imaging modality to another at a low cost. For unpaired datasets, they rely mostly on cycle loss. Despite its effectiveness in learning the underlying data distribution, it can lead to a discrepancy between input and output data. The purpose of this work is to investigate the hypothesis that we can predict image quality based on its latent representation in the GANs bottleneck. We achieve this by corrupting the latent representation with noise and generating multiple outputs. The degree of differences between them is interpreted as the strength of the representation: the more robust the latent representation, the fewer changes in the output image the corruption causes. Our results demonstrate that our proposed method has the ability to i) predict uncertain parts of synthesized images, and ii) identify samples that may not be reliable for downstream tasks, e.g., liver segmentation task.
BibTeX
@inproceedings{tomczak2022what,
  author = {Agnieszka Tomczak and Aarushi Gupta and Slobodan Ilic and Nassir Navab and Shadi Albarqouni},
  title = {{What can we learn about a generated image corrupting its latent representation?}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16446-0\_48},
}
Figure