Semantic Similarity Score for Measuring Visual Similarity at Semantic Level

Senran Fan, Zhicheng Bao, Chen Dong, Haotai Liang, Xiaodong Xu, Ping Zhang

TL;DR

Researchers have developed SeSS (Semantic Similarity Score), a novel image evaluation metric that uses Scene Graph Generation and graph matching to measure similarity at the "meaning" level rather than the pixel level. By decomposing images into objects and their relationships, SeSS aligns more closely with human perception, making it a vital tool for evaluating next-generation semantic communication systems.

What Problem Does This Paper Address?

As we move toward 6G, the paradigm of communication is shifting from "symbol-based" (sending every bit perfectly) to "semantic-based" (sending the meaning of the data). In visual semantic communication, a system might reconstruct an image that looks slightly different at the pixel level but preserves the essential "story"—for example, a person riding a bike.

Traditional metrics like PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) fail here. They are "pixel-obsessed"; if an object shifts by a few pixels or the color temperature changes slightly, these metrics report a massive loss in quality, even if a human viewer wouldn't care. While newer deep-learning metrics like LPIPS or ClipScore exist, they often lack interpretability or suffer from "bag-of-words" issues—failing to distinguish between "a dog chasing a man" and "a man chasing a dog."

Methods

The authors propose SeSS, which moves away from global feature vectors and toward a structured, graph-based representation of an image. The pipeline consists of three main stages:

  1. Segmentation & Graph Generation: The system uses the Segment Anything Model (SAM) to identify objects and the Panoptic Scene Graph (PSG) model to infer relationships between them (e.g., "on," "near," "holding"). This transforms an image into an "object-relation graph."
  2. Iterative Graph Matching: To compare two images, SeSS doesn't just compare the graphs' nodes. It uses an iterative algorithm where the similarity of two objects is influenced by the similarity of their neighbors and the relationships connecting them. This mimics human cognition, where we understand an object's importance based on its context.
  3. Human-Centric Tuning: The researchers manually annotated 100,000 image pairs with semantic similarity scores. They used this massive dataset to fine-tune the hyperparameters of the graph matching algorithm, ensuring the final score reflects human judgment.

Datasets

  • COCO2017: Used as the primary source for image sampling and testing.
  • Custom Annotated Dataset: A newly introduced benchmark of 100,000 image pairs, each manually scored for semantic similarity. This dataset includes images from open-source sets and those generated by large-scale AI models with varying levels of noise and transformation.

Key Insights and Ideas

  • Structure over Statistics: The core "aha moment" is that semantic similarity is a topological problem. By treating an image as a graph of entities, the metric becomes robust to "semantic-preserving" changes like rotation, translation, or scaling—things that usually "break" traditional metrics.
  • Contextual Importance: Not all objects are equal. SeSS incorporates a "pixel importance" prediction to weight objects that are central to the scene more heavily than background elements.
  • Solving the "Bag-of-Words" Flaw: Unlike ClipScore, which can be confused by the order of subjects and objects, SeSS’s use of directed graphs ensures that the specific relationship (who is doing what to whom) is preserved in the similarity calculation.

Key Results

The authors put SeSS through a gauntlet of tests against six standard metrics (PSNR, SSIM, MS-SSIM, LPIPS, ViTScore, and ClipScore):

  • Robustness to Transformations: In tests involving small rotations, translations, and color filters, SeSS maintained high similarity scores (often >0.8), while PSNR and SSIM plummeted to near-zero.
  • Semantic Communication Evaluation: When testing the LSCI (a semantic communication system), SeSS correctly identified that LSCI preserves meaning better than traditional JPEG compression at extremely low bitrates (low bpp). Traditional metrics often incorrectly ranked the "noisier" but more meaningful semantic images lower.
  • Generative Noise: When comparing images generated by AI models (like Unclip) with varying noise levels, SeSS showed a smooth, logical degradation in score that matched human perception, whereas pixel-level metrics were too sensitive to the stochastic noise.

Limitations and Future Work

The authors identify a few areas for improvement:

  • The "0.3 Floor": Currently, SeSS scores tend to cluster in the range above 0.3, failing to utilize the full 0-1 scale effectively. This makes it harder to distinguish between very dissimilar images.
  • Object Dependency: The metric’s accuracy is heavily dependent on the underlying segmentation and scene graph models. If the model fails to detect objects in a sparse or abstract image, the similarity score may be less reliable.
  • Computational Complexity: Generating scene graphs and performing iterative graph matching is significantly more computationally expensive than calculating a simple PSNR, which might limit its use in real-time applications for now.

Summary updated

Abstract

With the rapid development of Internet of Things (IoT) technology, more sensors are required to operate in complex channel scenarios and under limited communication resources. Semantic communication, as an emerging paradigm, extracts, transmits, and reconstructs information at the semantic level, offering advantages, such as high compression rates and strong noise resistance. These features are expected to find widespread application across various IoT scenarios. However, widely used image similarity evaluation metrics like peak signal-to-noise ratio and multiscale structural similarity index primarily focus on pixel or structural features, making it challenging to accurately measure the loss of semantic-level information during transmission. This limitation poses challenges for the performance evaluation of visual semantic communication systems and restricts the emergence of more novel and efficient systems. To address this issue, we propose a new semantic evaluation metric-semantic similarity score (SeSS). This metric is based on Scene Graph Generation and graph matching techniques, transforming image similarity scores into graph matching scores. By manually annotating thousands of image pairs, we fine-tuned the hyperparameters within SeSS to align it more closely with human semantic perception. The performance of SeSS has been tested across various image datasets and specific IoT visual tasks. Experimental results demonstrate the effectiveness of SeSS in measuring differences in semantic-level information between images, making it a valuable tool for evaluating visual semantic communication systems. This development is expected to encourage the emergence of more robust systems suited for diverse IoT scenarios. The code of SeSS is openly available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FSR3340/Semantic_Similarty_ScoreGitHub</uri>.
BibTeX
@article{fan2025semantic,
  author = {Senran Fan and Zhicheng Bao and Chen Dong and Haotai Liang and Xiaodong Xu and Ping Zhang},
  title = {{Semantic Similarity Score for Measuring Visual Similarity at Semantic Level}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2024.3518543},
}
Figure