SAMScore: A Semantic Structural Similarity Metric for Image Translation Evaluation

Yunxiang Li, Meixu Chen, Wenxuan Yang, Kai Wang, Jun Ma, Alan C. Bovik, You Zhang

Abstract

Image translation has wide applications, such as style transfer and modality conversion, usually aiming to generate images having both high degrees of realism and faithfulness. These problems remain difficult, especially when it is important to preserve semantic structures. Traditional image-level similarity metrics are of limited use, since the semantics of an image are high-level, and not strongly governed by pixel-wise faithfulness to an original image. Towards filling this gap, we introduce SAMScore, a generic semantic structural similarity metric for evaluating the faithfulness of image translation models. SAMScore is based on the recent high-performance Segment Anything Model (SAM), which can perform semantic similarity comparisons with standout accuracy. We applied SAMScore on 19 image translation tasks, and found that it is able to outperform all other competitive metrics on all of the tasks. We envision that SAMScore will prove to be a valuable tool that will help to drive the vibrant field of image translation, by allowing for more precise evaluations of new and evolving translation models.

Introduction

In image-to-image (I2I) translation, the objective is twofold: achieving realism (visual quality) and maintaining faithfulness (structural integrity). While metrics like FID handle realism effectively, measuring faithfulness remains a significant challenge. Traditional pixel-wise metrics such as PSNR or SSIM often fail because they are sensitive to color shifts and noise rather than high-level semantic structure. Perceptual metrics like LPIPS focus more on textures than shapes. Furthermore, existing structural metrics like FCNScore require domain-specific pre-trained segmentation models and ground-truth labels, which are expensive and difficult to scale.

SAMScore addresses this gap by providing a "zero-shot" structural similarity measure that understands objects and structures inherently, without requiring custom-trained models for every new dataset.

Methodology

The core innovation of SAMScore is its utilization of the Segment Anything Model (SAM) Encoder. Because SAM was trained on over 1.1 billion masks, its internal embeddings are exceptionally "structure-aware."

As illustrated in the pipeline below, the process involves:

  1. Feature Extraction: Both the source image and the generated image are passed through the SAM encoder (ViT-L backbone).
  2. Spatial Cosine Similarity: The metric calculates the cosine similarity between the high-dimensional semantic embeddings at every spatial position $(h, w)$.
  3. Aggregation: These local similarities are averaged to produce a final SAMScore between 0 and 1.
The SAMScore architecture: extracting semantic embeddings from source and generated images to compute a spatial-wise cosine similarity matrix.

This approach requires no labels or fine-tuning and works "out of the box" for diverse tasks ranging from artistic style transfer to medical imaging.

Key Results and Insights

The researchers evaluated SAMScore across 19 image translation tasks, including Cityscapes (label-to-photo), artistic styles (Monet, Van Gogh), and medical modalities (MRI-to-CT).

Performance and Robustness

SAMScore consistently outperformed traditional metrics like L2, PSNR, SSIM, and LPIPS. A key advantage is its sensitivity to deformation versus its robustness to noise. While Gaussian noise significantly degrades PSNR and SSIM scores, SAMScore remains stable because the underlying objects in the image have not moved or changed shape. Conversely, when images are warped (piecewise affine deformation), SAMScore is the most sensitive metric, accurately capturing the loss of structural integrity.

Quantitative comparison of SAMScore against traditional metrics across 19 diverse image translation tasks.

In medical imaging tasks, such as MRI-to-CT conversion, SAMScore's ranking of generative models aligned more closely with ground-truth structural measurements than traditional metrics, which often produced misleading results due to pixel-level differences.

Conclusion

SAMScore provides a robust, universal, and label-free solution for evaluating faithfulness in image translation. By focusing on semantic structure rather than pixel-level accuracy, it offers a more precise tool for researchers to evaluate and optimize generative models. Future work will explore using SAMScore as a direct loss function during model training and improving its granularity for fine-grained sub-structures.

Summary updated

Abstract

Image translation has wide applications, such as style transfer and modality conversion, usually aiming to generate images having both high degrees of realism and faithfulness. These problems remain difficult, especially when it is important to preserve semantic structures. Traditional image-level similarity metrics are of limited use, since the semantics of an image are high-level, and not strongly governed by pixel-wise faithfulness to an original image. Towards filling this gap, we introduce SAMScore, a generic semantic structural similarity metric for evaluating the faithfulness of image translation models. SAMScore is based on the recent high-performance Segment Anything Model (SAM), which can perform semantic similarity comparisons with standout accuracy. We applied SAMScore on 19 image translation tasks, and found that it is able to outperform all other competitive metrics on all of the tasks. We envision that SAMScore will prove to be a valuable tool that will help to drive the vibrant field of image translation, by allowing for more precise evaluations of new and evolving translation models. The code is available at https://github.com/Kent0n-Li/SAMScore.
BibTeX
@article{li2025samscore,
  author = {Yunxiang Li and Meixu Chen and Wenxuan Yang and Kai Wang and Jun Ma and Alan C. Bovik and You Zhang},
  title = {{SAMScore: A Semantic Structural Similarity Metric for Image Translation Evaluation}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2025},
  doi = {10.1109/tai.2025.3535456},
}

Loading related content...

Figure