Abstract
In surgical computer vision applications, obtaining labeled training data is challenging due to data-privacy concerns and the need for expert annotation. Unpaired image-to-image translation techniques have been explored to automatically generate large annotated datasets by translating synthetic images to the realistic domain. However, preserving the structure and semantic consistency between the input and translated images presents significant challenges, mainly when there is a distributional mismatch in the semantic characteristics of the domains. This study empirically investigates unpaired image translation methods for generating suitable data in surgical applications, explicitly focusing on semantic consistency. We extensively evaluate various state-of-the-art image translation models on two challenging surgical datasets and downstream semantic segmentation tasks. We find that a simple combination of structural-similarity loss and contrastive learning yields the most promising results. Quantitatively, we show that the data generated with this approach yields higher semantic consistency and can be used more effectively as training data.
Introduction
Training robust AI models for surgical applications is often hindered by the scarcity of labeled data. While real-world surgical videos are difficult to acquire and annotate, synthetic simulations offer a scalable alternative with "free" ground-truth labels. However, a significant "reality gap" exists between these clean simulations and the complex environment of an actual operating room. Unpaired Image-to-Image (I2I) translation techniques, such as CycleGAN, are typically used to bridge this gap. Unfortunately, these methods often suffer from semantic inconsistency, where anatomical structures are incorrectly translated (e.g., mapping blood textures onto surgical tools). This research proposes a solution to maintain semantic integrity during the translation process.
Methodology: The ConStructS Framework
The authors propose ConStructS (Contrastive Learning coupled with Structural Similarity), which enhances the Contrastive Unpaired Translation (CUT) framework with a dedicated semantic loss.
The framework relies on three primary components:
- PatchNCE (Patch-based Noise Contrastive Estimation): This core mechanism forces the model to maintain correspondence between specific patches in the input and output, ensuring that a liver patch in the simulation remains a liver patch in the realistic output.
- MS-SSIM (Multi-Scale Structural Similarity) Loss: Acting as the "semantic loss," MS-SSIM penalizes the model if it alters the luminance, contrast, or structure of the original simulation across multiple scales. This serves as a structural anchor, preventing the "hallucination" of textures.
- Adversarial Learning: A standard discriminator pushes the generator to produce images that are indistinguishable from real surgical frames.
Experimental Setup and Datasets
The method was evaluated on two distinct surgical datasets:
- Cholecystectomy (Gallbladder removal): Comprising 20,000 synthetic images and ~26,000 real frames from the Cholec80 dataset.
- Gastrectomy (Stomach removal): Featuring 3,400 synthetic and 4,500 real images, complete with segmentation masks for organs like the liver, pancreas, and stomach.
Results and Discussion
The evaluation focused on both qualitative consistency and downstream task performance. ConStructS demonstrated a clear advantage in preserving anatomical boundaries and preventing "texture bleeding," where textures from one organ might bleed onto another or onto metallic tools.
Quantitatively, ConStructS achieved a Mean Intersection over Union (mIOU) of 0.29 on the Cholecystectomy dataset, outperforming CUT (0.26) and CycleGAN (0.23). Most importantly, when the generated data was used to train a liver segmentation model from scratch, it yielded a 9% improvement in Dice score compared to models trained on baseline synthetic data. These results suggest that structural similarity is a more effective constraint than more complex cycle-consistency or specialized surgical models.
Conclusion
ConStructS effectively addresses the semantic inconsistency problem in surgical I2I translation by combining contrastive learning with multi-scale structural similarity. While some failure cases remain in highly complex scenes, the approach significantly improves the utility of synthetic data for training downstream segmentation models. Future research will explore multi-modal I2I translation to better capture the vast variability in patient anatomy and surgical environments.
Summary updated