TL;DR
This paper introduces SAMScore, a novel metric for evaluating image translation models by focusing on the preservation of content structure. By leveraging the powerful segmentation capabilities of the Segment Anything Model (SAM), SAMScore offers a more robust and accurate assessment of faithfulness compared to traditional metrics, especially in complex translation tasks.
What Problem Does This Paper Address?
Image translation, the process of transforming an image from one domain to another while preserving its core content, has numerous applications, from artistic style transfer to medical imaging. A critical challenge in this field is ensuring that the content structure of the original image is faithfully maintained in the translated output. Existing metrics like L2 norm, PSNR, SSIM, and even LPIPS, often fall short because they focus on pixel-level similarity or general perceptual similarity, failing to capture high-level structural integrity. Metrics like FCNScore, which use segmentation, are hampered by the need for labeled data, domain gaps, and limited granularity. This leaves a significant gap in reliably evaluating the faithfulness of image translation models, hindering progress in developing better techniques.
Methods
The core innovation of SAMScore lies in its use of the Segment Anything Model (SAM) encoder. SAM, a recent breakthrough in image segmentation, possesses remarkable zero-shot generalization capabilities, allowing it to extract rich, high-level structural information from images without task-specific training.
Here's how SAMScore works:
- Content Structure Embedding: Both the source image and the generated (translated) image are passed through the SAM encoder. This process extracts "content structural embeddings" that capture the essential structural information of each image.
- Spatial Cosine Similarity: Instead of comparing pixels directly, SAMScore calculates the cosine similarity between the spatial embeddings of the source and generated images. This measures the similarity of the structural information at each spatial location.
- Averaging for SAMScore: The final SAMScore is the average of these spatial cosine similarities across all locations, providing a single, comprehensive score for content structural similarity.
This approach leverages SAM's ability to understand and represent image content at a structural level, moving beyond simple pixel-wise comparisons.
Datasets
The paper evaluates SAMScore on a diverse set of 19 image translation tasks across 8 datasets. These include:
- Medical Imaging: MICCAI 2020 challenge data (Head CT and MR), and the SynthRAD2023 dataset.
- General Image Translation: Apple to orange, orange to apple, horse to zebra, zebra to horse.
- Domain Transfer: Cityscapes (label to photo, photo to label), facades (label to photo, photo to label).
- Artistic Style Transfer: Monet to photo, photo to Monet, photo to Cezanne, photo to Ukiyoe, photo to Vangogh.
- Geographic/Scene: Aerial photograph to map, map to aerial photograph.
- Seasonal: Yosemite (summer to winter, winter to summer).
The paper does not introduce new datasets but utilizes publicly available ones, often from established benchmarks like CycleGAN's dataset repository.
Key Insights and Ideas
- SAM's Structural Power: The Segment Anything Model (SAM) is not just for segmentation; its encoder provides powerful, generalizable embeddings that are highly effective for capturing structural content.
- Beyond Pixels: Traditional metrics (L2, PSNR, SSIM) are fundamentally limited because they operate at the pixel level and are easily fooled by minor color or texture changes that don't alter the core structure.
- Perceptual vs. Structural: Metrics like LPIPS capture perceptual similarity but don't specifically target structural faithfulness, which is crucial for many image translation tasks.
- Robustness to Noise: SAMScore demonstrates remarkable robustness to Gaussian noise, a common artifact that can significantly degrade image quality without necessarily altering the fundamental structure. This is a key advantage over metrics that are sensitive to pixel value fluctuations.
- Sensitivity to Deformation: Conversely, SAMScore is highly sensitive to geometric deformations, accurately reflecting changes in content structure. This sensitivity is desirable for evaluating how well a translation model preserves structure under transformations.
- Domain Gap Impact: The paper highlights how domain gaps can severely impact metrics like FCNScore, which rely on segmentation models trained on target domain data. SAMScore, with its general-purpose encoder, is less affected by such gaps.
Key Results
- Superior Performance: SAMScore consistently outperformed L2, PSNR, SSIM, and LPIPS across 19 image translation tasks in terms of correlation with structural deformations (Table I).
- Robustness to Noise: SAMScore showed very low correlation with Gaussian noise levels, indicating high robustness, while other metrics showed significant correlations (Table II).
- Outperforming FCNScore: On the cityscapes dataset, SAMScore demonstrated superior accuracy and robustness compared to FCNScore, especially when dealing with domain gaps and varying distortions (Tables IV, V, VI, and Figs. 4, 5, 6).
- Medical Imaging Success: In medical image translation (CT to MR, MR to CT), SAMScore ranked models consistently with visual evaluations and L2 (GT) metrics, unlike other metrics that failed to provide consistent measurements (Table VIII, Fig. 8).
- Color Jitter Resilience: SAMScore embeddings were minimally affected by increasing color jitter, focusing on structure rather than color variations, unlike ViTScore which dropped significantly (Fig. 7, Table VII).
Limitations and Future Work
- Specific Scenarios: The authors acknowledge limitations in specific medical imaging scenarios (e.g., bone MR-to-CT, prostate imaging) where subtle differences in imaging mechanisms or anatomical variations can lead to discrepancies that SAMScore might not perfectly capture. This is due to the assumption of invariant structures before and after translation.
- Future Work:
- Assessing image structural similarity for specified regions of interest.
- Developing SAMScore-guided image translation models.
- Further exploring the potential of SAMScore for more sophisticated evaluations.
Summary updated