Abstract
As communication systems shift from transmitting raw bits to conveying "semantic" meaning, traditional evaluation tools require a significant upgrade. This paper introduces ViTScore, a novel metric that utilizes pre-trained Vision Transformers (ViT) to measure the preservation of image meaning during transmission, even under heavy pixel distortion. ViTScore serves as a robust, human-centric alternative to traditional metrics like PSNR or SSIM, particularly in the presence of "semantic noise" or adversarial attacks. By prioritizing global semantic similarity over local pixel accuracy, ViTScore provides a more accurate reflection of communication success in modern semantic systems.
Introduction: The Pixel-Obsessed Evaluation Gap
Traditional image quality metrics are fundamentally "pixel-obsessed." PSNR (Peak Signal-to-Noise Ratio) measures differences at the individual pixel level, while SSIM (Structural Similarity Index) focuses on local luminance and contrast. Even more recent deep-learning metrics like LPIPS (Learned Perceptual Image Patch Similarity) rely on convolutional features that are often strictly tied to specific spatial locations.
In the emerging field of Semantic Communications (SC), the primary objective is not the perfect reconstruction of every pixel, but ensuring the receiver correctly interprets the underlying content (e.g., identifying "a woman playing tennis"). If an image undergoes a slight shift, rotation, or color inversion, traditional metrics often report a total failure, even if a human observer can still easily identify the subject. To address this, we propose ViTScore, a metric that prioritizes global semantic similarity.
Methodology: The ViTScore Framework
Inspired by the success of BERTScore in Natural Language Processing, ViTScore adapts the concept of semantic similarity to the visual domain. The process involves four key stages:
- Feature Extraction: A pre-trained Vision Transformer (ViT) decomposes both the original and reconstructed images into patches. It then extracts high-level semantic feature vectors for each patch.
- Greedy Matching: Unlike traditional metrics that compare patches at identical coordinates, ViTScore employs a "greedy match" strategy. For every patch in the original image, it identifies the most similar patch anywhere in the reconstructed image using cosine similarity. This allows for spatial flexibility.
- Recall and Precision: The system calculates "Semantic Recall" (the extent to which the original meaning is present in the reconstruction) and "Semantic Precision" (the extent to which the reconstruction's content is grounded in the original).
- F1-Score Calculation: The final ViTScore is the harmonic mean of these recall and precision values.
Mathematically, ViTScore satisfies three essential properties for a reliable metric: Symmetry (the score is consistent regardless of which image is the reference), Boundedness (scores are confined between -1 and 1), and Normalization (identical images yield a score of exactly 1).
Experimental Setup and Datasets
ViTScore was rigorously evaluated across several standard computer vision and communication datasets to ensure its universality:
- COCO: Used for image captioning to correlate ViTScore with text-based semantic metrics like BERTScore.
- Kodak: A benchmark for image communication systems.
- CelebA & LSUN: Utilized to test "semantic attacks," such as image inversion using Generative Adversarial Networks (GANs).
- General Universality Testing: Datasets including DIV2K, Set5, Set14, B100, and Urban100 were used to verify performance across diverse image types and resolutions.
Key Insights: Navigating Semantic Noise
A primary insight from this research is the "Semantic Noise" paradox. In scenarios involving adversarial attacks—such as using a GAN to "invert" an image—the pixel distribution changes entirely. Consequently, PSNR and SSIM scores drop to near-zero. However, the semantic meaning often remains intact. ViTScore is uniquely capable of maintaining a high score in these scenarios, aligning closely with human perception.
Furthermore, the self-attention mechanism of the ViT allows features to capture global context. This makes ViTScore "spatially flexible," meaning it does not penalize a system if a feature moves slightly within the frame, provided the concept remains present. Ablation studies confirmed that "Max Pooling" (greedy matching) significantly outperforms "Mean Pooling," suggesting that semantic similarity is driven by the presence of key concepts rather than an average of all pixels.
Results and Performance
ViTScore demonstrates superior performance across several metrics:
- Human Alignment: On the COCO dataset, ViTScore showed a Pearson Correlation Coefficient (PCC) of 0.2198 with BERTScore, significantly higher than LPIPS (-0.1217) and PSNR (0.0515).
- Robustness in Low SNR: In simulated wireless environments (AWGN and Rayleigh fading), ViTScore remained stable at low Signal-to-Noise Ratios where traditional metrics fluctuated.
- Semantic Attack Resilience: In GAN-based inversion tests, ViTScore outperformed existing metrics by a wide margin, proving its ability to "see" the subject despite pixel transformations.
Limitations and Future Outlook
While promising, ViTScore has limitations. The use of a full Vision Transformer introduces higher computational overhead compared to MSE or SSIM. Additionally, the metric's effectiveness is tied to the quality and diversity of the ViT's pre-training data.
Future work will focus on extending this framework to video semantic communications, addressing the added complexity of temporal consistency. We also envision integrating semantic metrics directly into the training loops of communication models to optimize for meaning rather than just bit-accuracy.
Summary updated