An Asymmetric Cycle-Consistency Loss for Dealing with Many-to-One Mappings in Image Translation: A Study on Thigh MR Scans

Michael Gadermayr, Maximilian Tschuchnig, Laxmi Gupta, Dorit Merhof, Nils Krämer, Daniel Truhn, Burkhard GessarXiv 2020 · arXiv

TL;DR

Standard CycleGAN architectures assume a bijective (one-to-one) mapping between image domains, which fails when translating complex pathologies to healthy states—a classic many-to-one problem. This paper introduces a simple yet highly effective asymmetric cycle-consistency loss that drops the restrictive reverse-reconstruction penalty. Tested on thigh MRIs, this approach significantly improves the generation of "pseudo-healthy" images, enabling more accurate muscle segmentation for neuromuscular disease assessment.

What Problem Does This Paper Address?

Unpaired image-to-image translation frameworks, like CycleGAN, are incredibly valuable in medical imaging because paired datasets (e.g., an MRI of the exact same patient's thigh in both a healthy and diseased state) are virtually impossible to obtain.

However, standard CycleGAN relies on a symmetric bidirectional "cycle-consistency loss." This mathematical constraint assumes that if you translate an image from Domain A to Domain B and back again, you should get your original image back. This implies a perfect one-to-one relationship between domains.

In medical scenarios, this assumption breaks down. Multiple distinct pathological states—varying in the shape, severity, and texture of fat infiltration—all map to a single homogeneous "healthy" muscle state. This creates a many-to-one mapping problem. Under a standard symmetric loss, the network is forced to "hide" details of the pathology within the pseudo-healthy image just so it can successfully reconstruct the diseased state on the reverse cycle. This prevents the model from generating a truly clean, fully healed image.

Fig. 2: T1-weighted MRI slices in axial orientation at the thigh level showing (a) healthy muscle, (b) moderate infil- trations and (c) largely affected muscle areas.

Methods

Instead of redesigning the network architecture or introducing complex latent spaces, the authors propose an elegant mathematical fix: Asymmetric Cycle-Consistency Loss.

In a standard setup, the loss combines two cycles:

  1. Healthy → Pathological → Healthy ($L_{AC}$)
  2. Pathological → Pseudo-Healthy → Pathological ($L_{AC'}$)

Because the mapping from Pathological to Pseudo-Healthy is many-to-one, enforcing the second cycle ($L_{AC'}$) forces the generator to leave behind unwanted artifacts or "remnants" of the disease to facilitate the reverse reconstruction.

The authors simply drop the second cycle entirely. By optimizing only the cycle that starts in the healthy domain ($L_{AC}$), the generator translating pathological images is no longer penalized for completely erasing the disease characteristics. To maintain the balance between the cycle-consistency loss and the GAN loss, the remaining loss term is multiplied by a factor of 2.

Fig. 1: Schematic outline of the problem of one-to-many and many-to-one mappings with the formulation proposed in [1]. The cycle on the left side generates a small loss in case of well performing generators F and G. The cycle on the right side does not necessarily produce a small loss if F(x) reliably removes all information regarding the characteristics of domain X.

Datasets

The method was evaluated on a dataset of T1-weighted MR images of human thighs in axial orientation:

  • Total Patients: 21 'Healthy' controls and 20 'Pathological' subjects exhibiting clear muscular fat-infiltration.
  • 2D Slices: To maximize sample size, 2D slices were extracted, yielding 1,124 Healthy and 1,070 Pathological images (256×256 pixels).
  • Evaluation Subsets: The pathological images were further categorized into 7 'Moderate' and 13 'Severe' cases.
  • Ground Truth: Manual annotations of the muscular compartments were performed on every fourth slice by a student assistant and supervised by a medical expert.

Key Insights and Ideas

  • The "Information Hiding" Trap: Standard CycleGANs end up "cheating" when faced with many-to-one tasks. The network embeds high-frequency, imperceptible noise or leaves visible remnants of the pathology in the target image solely to satisfy the reverse cycle-consistency requirement.
  • Occam's Razor for GANs: While prior works attempted to solve many-to-one mappings using complex multi-modal latent spaces (like Augmented CycleGAN), these methods require heavy hyperparameter tuning and struggle with arbitrary pathology shapes. Dropping half the cycle loss achieves superior results without adding a single parameter to the model.
  • Downstream Task Simplification: Instead of training a complex, data-hungry deep learning segmentation model on highly variable pathological images, the authors use the asymmetric GAN to "cure" the image first. Once the image is pseudo-healthy, simple and robust intensity-based thresholding is sufficient to segment the muscle and calculate the fat-fraction.

Key Results

The performance was measured using the Dice Similarity Coefficient (DSC) after threshold-based muscle segmentation. The proposed asymmetric loss ($L_{AC}$) consistently outperformed both the original un-translated data and the standard CycleGAN ($L_C$).

Table 1: Results obtained with (a) the asymmetric cycle- consistency loss compared to (b) the original formulation [1] and (c) the performance with original data without any im- age translation. The P-value indicates whether improvements using the asymmetric loss are significant (compared to (b)).
  • Severe Cases: The asymmetric approach achieved a significant performance boost, raising the DSC to 0.808 compared to 0.750 on raw data and 0.788 with standard CycleGAN ($p < 0.01$).
  • Qualitative Improvement: Visual inspection confirmed that larger fat infiltrations were "filled in" much more effectively, drastically reducing false negatives during segmentation.
Fig. 3: The left column (a) shows example original images of the ’Severe’ data set including a visualization of the optimum pixel-based separability of muscle tissue using two intensity thresholds. Columns (b) and (c) show the improvements ob- tained with the LC loss [1] and with the proposed LAC for- mulation.

Limitations and Future Work

  • Subcutaneous Fat Leakage: The model occasionally translates subcutaneous fat at the outer boundaries into muscle tissue. This is primarily due to the partial volume effect, where boundary pixels share intensity characteristics with muscle tissue, tricking the discriminator.
  • Data Correlation: Although the dataset contains over 1,000 2D slices, they originate from a relatively small pool of 41 patients. Because consecutive slices from the same patient are highly correlated, the model still exhibits minor visual artifacts.

Future work will focus on incorporating larger, more diverse patient cohorts and developing specialized preprocessing steps to better isolate boundary tissues.

Summary updated

Abstract

Generative adversarial networks using a cycle-consistency loss facilitate unpaired training of image-translation models and thereby exhibit a very high potential in manifold medical applications. However, the fact that images in one domain potentially map to more than one image in another domain (e.g. in case of pathological changes) exhibits a major challenge for training the networks. In this work, we offer a solution to improve the training process in case of many-to-one mappings by modifying the cycle-consistency loss. We show formally and empirically that the proposed method improves the performance significantly without radically changing the architecture and without increasing the overall complexity. We evaluate our method on thigh MRI scans with the final goal of segmenting the muscle in fat-infiltrated patients' data.
BibTeX
@misc{gadermayr2020asymmetric,
  author = {Michael Gadermayr and Maximilian Tschuchnig and Laxmi Gupta and Dorit Merhof and Nils Krämer and Daniel Truhn and Burkhard Gess},
  title = {{An Asymmetric Cycle-Consistency Loss for Dealing with Many-to-One Mappings in Image Translation: A Study on Thigh MR Scans}},
  year = {2020},
  eprint = {2004.11001},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.11001},
}
Figure