TL;DR
Researchers have developed SeSS (Semantic Similarity Score), a novel image evaluation metric that uses Scene Graph Generation and graph matching to measure similarity at the "meaning" level rather than the pixel level. By decomposing images into objects and their relationships, SeSS aligns more closely with human perception, making it a vital tool for evaluating next-generation semantic communication systems.
What Problem Does This Paper Address?
As we move toward 6G, the paradigm of communication is shifting from "symbol-based" (sending every bit perfectly) to "semantic-based" (sending the meaning of the data). In visual semantic communication, a system might reconstruct an image that looks slightly different at the pixel level but preserves the essential "story"—for example, a person riding a bike.
Traditional metrics like PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) fail here. They are "pixel-obsessed"; if an object shifts by a few pixels or the color temperature changes slightly, these metrics report a massive loss in quality, even if a human viewer wouldn't care. While newer deep-learning metrics like LPIPS or ClipScore exist, they often lack interpretability or suffer from "bag-of-words" issues—failing to distinguish between "a dog chasing a man" and "a man chasing a dog."
Methods
The authors propose SeSS, which moves away from global feature vectors and toward a structured, graph-based representation of an image. The pipeline consists of three main stages:
- Segmentation & Graph Generation: The system uses the Segment Anything Model (SAM) to identify objects and the Panoptic Scene Graph (PSG) model to infer relationships between them (e.g., "on," "near," "holding"). This transforms an image into an "object-relation graph."
- Iterative Graph Matching: To compare two images, SeSS doesn't just compare the graphs' nodes. It uses an iterative algorithm where the similarity of two objects is influenced by the similarity of their neighbors and the relationships connecting them. This mimics human cognition, where we understand an object's importance based on its context.
- Human-Centric Tuning: The researchers manually annotated 100,000 image pairs with semantic similarity scores. They used this massive dataset to fine-tune the hyperparameters of the graph matching algorithm, ensuring the final score reflects human judgment.
Datasets
- COCO2017: Used as the primary source for image sampling and testing.
- Custom Annotated Dataset: A newly introduced benchmark of 100,000 image pairs, each manually scored for semantic similarity. This dataset includes images from open-source sets and those generated by large-scale AI models with varying levels of noise and transformation.
Key Insights and Ideas
- Structure over Statistics: The core "aha moment" is that semantic similarity is a topological problem. By treating an image as a graph of entities, the metric becomes robust to "semantic-preserving" changes like rotation, translation, or scaling—things that usually "break" traditional metrics.
- Contextual Importance: Not all objects are equal. SeSS incorporates a "pixel importance" prediction to weight objects that are central to the scene more heavily than background elements.
- Solving the "Bag-of-Words" Flaw: Unlike ClipScore, which can be confused by the order of subjects and objects, SeSS’s use of directed graphs ensures that the specific relationship (who is doing what to whom) is preserved in the similarity calculation.
Key Results
The authors put SeSS through a gauntlet of tests against six standard metrics (PSNR, SSIM, MS-SSIM, LPIPS, ViTScore, and ClipScore):
- Robustness to Transformations: In tests involving small rotations, translations, and color filters, SeSS maintained high similarity scores (often >0.8), while PSNR and SSIM plummeted to near-zero.
- Semantic Communication Evaluation: When testing the LSCI (a semantic communication system), SeSS correctly identified that LSCI preserves meaning better than traditional JPEG compression at extremely low bitrates (low bpp). Traditional metrics often incorrectly ranked the "noisier" but more meaningful semantic images lower.
- Generative Noise: When comparing images generated by AI models (like Unclip) with varying noise levels, SeSS showed a smooth, logical degradation in score that matched human perception, whereas pixel-level metrics were too sensitive to the stochastic noise.
Limitations and Future Work
The authors identify a few areas for improvement:
- The "0.3 Floor": Currently, SeSS scores tend to cluster in the range above 0.3, failing to utilize the full 0-1 scale effectively. This makes it harder to distinguish between very dissimilar images.
- Object Dependency: The metric’s accuracy is heavily dependent on the underlying segmentation and scene graph models. If the model fails to detect objects in a sparse or abstract image, the similarity score may be less reliable.
- Computational Complexity: Generating scene graphs and performing iterative graph matching is significantly more computationally expensive than calculating a simple PSNR, which might limit its use in real-time applications for now.
Summary updated