Abstract
As communication shifts toward semantic-based paradigms in the 6G era, traditional pixel-based metrics like PSNR and SSIM become inadequate for evaluating image reconstruction. This paper introduces SeSS (Semantic Similarity Score), a novel metric that leverages Scene Graph Generation and iterative graph matching to assess visual similarity at the "meaning" level. By decomposing images into structured object-relation graphs, SeSS aligns more closely with human perception and demonstrates superior robustness to semantic-preserving transformations.
Introduction
The transition toward 6G communication marks a fundamental shift from "symbol-based" transmission—where every bit must be perfectly preserved—to "semantic-based" communication, which prioritizes the transmission of meaning. In visual semantic communication, a system may reconstruct an image that differs significantly at the pixel level but preserves the essential "story" or context.
Traditional evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are "pixel-obsessed." Minor shifts or color temperature changes can result in poor scores, even if the semantic content remains identical to a human observer. While deep-learning metrics like LPIPS or ClipScore offer improvements, they often lack interpretability or fail to capture complex relational structures.
The SeSS Framework
To address these limitations, the researchers proposed SeSS (Semantic Similarity Score). Unlike metrics that rely on global feature vectors, SeSS utilizes a structured, graph-based representation of images. The pipeline involves three critical stages:
- Segmentation & Graph Generation: The framework employs the Segment Anything Model (SAM) to identify discrete objects and the Panoptic Scene Graph (PSG) model to infer relationships between them (e.g., "on," "near," "holding"). This process transforms a raw image into an "object-relation graph."
- Iterative Graph Matching: Similarity is not calculated by simple node comparison. Instead, SeSS uses an iterative algorithm where the similarity of two objects is weighted by the similarity of their neighbors and the relationships connecting them, mimicking human contextual understanding.
- Human-Centric Tuning: The algorithm's hyperparameters were fine-tuned using a benchmark of 100,000 manually annotated image pairs, ensuring the final score reflects human judgment of semantic similarity.
Key Insights
- Structure over Statistics: By treating images as topological graphs of entities, SeSS remains robust to "semantic-preserving" changes such as rotation, translation, or scaling that typically degrade traditional metrics.
- Contextual Importance: SeSS incorporates "pixel importance" predictions to prioritize central scene elements over background noise.
- Relational Integrity: By using directed graphs, SeSS preserves the specific roles of subjects and objects, avoiding the "bag-of-words" errors common in CLIP-based metrics (e.g., distinguishing "a dog chasing a man" from "a man chasing a dog").
Experimental Results
SeSS was evaluated against six standard metrics (PSNR, SSIM, MS-SSIM, LPIPS, ViTScore, and ClipScore) across several benchmarks:
- Robustness to Transformations: SeSS maintained high similarity scores (>0.8) during minor rotations and translations, whereas PSNR and SSIM scores dropped precipitously.
- Semantic Communication Evaluation: In testing semantic communication systems (e.g., LSCI), SeSS correctly identified that these systems preserve meaning better than traditional JPEG compression at extremely low bitrates, a distinction traditional metrics failed to make.
- Generative Noise: When comparing AI-generated images with varying stochastic noise, SeSS showed a smooth, logical degradation in score that aligned with human perception.
Limitations and Future Work
Despite its advantages, SeSS has areas for further development. Currently, scores tend to cluster above 0.3, which can make it difficult to distinguish between highly dissimilar images. Additionally, the metric's accuracy is contingent on the performance of the underlying segmentation and scene graph models. Finally, the computational cost of graph matching is higher than pixel-wise calculations, presenting a challenge for real-time applications.
Summary updated