How to Evaluate Semantic Communications for Images with ViTScore Metric?

Tingting Zhu, Bo Peng, Jifan Liang, Tingchen Han, Hai Wan, Jingqiao Fu, Junjie Chen

Abstract

As communication systems shift from transmitting raw bits to conveying "semantic" meaning, traditional evaluation tools require a significant upgrade. This paper introduces ViTScore, a novel metric that utilizes pre-trained Vision Transformers (ViT) to measure the preservation of image meaning during transmission, even under heavy pixel distortion. ViTScore serves as a robust, human-centric alternative to traditional metrics like PSNR or SSIM, particularly in the presence of "semantic noise" or adversarial attacks. By prioritizing global semantic similarity over local pixel accuracy, ViTScore provides a more accurate reflection of communication success in modern semantic systems.

Introduction: The Pixel-Obsessed Evaluation Gap

Traditional image quality metrics are fundamentally "pixel-obsessed." PSNR (Peak Signal-to-Noise Ratio) measures differences at the individual pixel level, while SSIM (Structural Similarity Index) focuses on local luminance and contrast. Even more recent deep-learning metrics like LPIPS (Learned Perceptual Image Patch Similarity) rely on convolutional features that are often strictly tied to specific spatial locations.

In the emerging field of Semantic Communications (SC), the primary objective is not the perfect reconstruction of every pixel, but ensuring the receiver correctly interprets the underlying content (e.g., identifying "a woman playing tennis"). If an image undergoes a slight shift, rotation, or color inversion, traditional metrics often report a total failure, even if a human observer can still easily identify the subject. To address this, we propose ViTScore, a metric that prioritizes global semantic similarity.

Methodology: The ViTScore Framework

Inspired by the success of BERTScore in Natural Language Processing, ViTScore adapts the concept of semantic similarity to the visual domain. The process involves four key stages:

  1. Feature Extraction: A pre-trained Vision Transformer (ViT) decomposes both the original and reconstructed images into patches. It then extracts high-level semantic feature vectors for each patch.
  2. Greedy Matching: Unlike traditional metrics that compare patches at identical coordinates, ViTScore employs a "greedy match" strategy. For every patch in the original image, it identifies the most similar patch anywhere in the reconstructed image using cosine similarity. This allows for spatial flexibility.
  3. Recall and Precision: The system calculates "Semantic Recall" (the extent to which the original meaning is present in the reconstruction) and "Semantic Precision" (the extent to which the reconstruction's content is grounded in the original).
  4. F1-Score Calculation: The final ViTScore is the harmonic mean of these recall and precision values.
The Vision Transformer (ViT) architecture provides the foundation for extracting semantic features from image patches.

Mathematically, ViTScore satisfies three essential properties for a reliable metric: Symmetry (the score is consistent regardless of which image is the reference), Boundedness (scores are confined between -1 and 1), and Normalization (identical images yield a score of exactly 1).

Experimental Setup and Datasets

ViTScore was rigorously evaluated across several standard computer vision and communication datasets to ensure its universality:

  • COCO: Used for image captioning to correlate ViTScore with text-based semantic metrics like BERTScore.
  • Kodak: A benchmark for image communication systems.
  • CelebA & LSUN: Utilized to test "semantic attacks," such as image inversion using Generative Adversarial Networks (GANs).
  • General Universality Testing: Datasets including DIV2K, Set5, Set14, B100, and Urban100 were used to verify performance across diverse image types and resolutions.

Key Insights: Navigating Semantic Noise

A primary insight from this research is the "Semantic Noise" paradox. In scenarios involving adversarial attacks—such as using a GAN to "invert" an image—the pixel distribution changes entirely. Consequently, PSNR and SSIM scores drop to near-zero. However, the semantic meaning often remains intact. ViTScore is uniquely capable of maintaining a high score in these scenarios, aligning closely with human perception.

Furthermore, the self-attention mechanism of the ViT allows features to capture global context. This makes ViTScore "spatially flexible," meaning it does not penalize a system if a feature moves slightly within the frame, provided the concept remains present. Ablation studies confirmed that "Max Pooling" (greedy matching) significantly outperforms "Mean Pooling," suggesting that semantic similarity is driven by the presence of key concepts rather than an average of all pixels.

Results and Performance

ViTScore demonstrates superior performance across several metrics:

  • Human Alignment: On the COCO dataset, ViTScore showed a Pearson Correlation Coefficient (PCC) of 0.2198 with BERTScore, significantly higher than LPIPS (-0.1217) and PSNR (0.0515).
  • Robustness in Low SNR: In simulated wireless environments (AWGN and Rayleigh fading), ViTScore remained stable at low Signal-to-Noise Ratios where traditional metrics fluctuated.
  • Semantic Attack Resilience: In GAN-based inversion tests, ViTScore outperformed existing metrics by a wide margin, proving its ability to "see" the subject despite pixel transformations.
Scatter plot showing the weak correlation between LPIPS and BERTScore, highlighting the limitations of current perceptual metrics in capturing semantic meaning.

Limitations and Future Outlook

While promising, ViTScore has limitations. The use of a full Vision Transformer introduces higher computational overhead compared to MSE or SSIM. Additionally, the metric's effectiveness is tied to the quality and diversity of the ViT's pre-training data.

Future work will focus on extending this framework to video semantic communications, addressing the added complexity of temporal consistency. We also envision integrating semantic metrics directly into the training loops of communication models to optimize for meaning rather than just bit-accuracy.

Summary updated

BibTeX
@article{zhu2024how,
  author = {Tingting Zhu and Bo Peng and Jifan Liang and Tingchen Han and Hai Wan and Jingqiao Fu and Junjie Chen},
  title = {{How to Evaluate Semantic Communications for Images with ViTScore Metric?}},
  journal = {IEEE Trans. Cogn. Commun. Netw},
  year = {2024},
  doi = {10.1109/tccn.2024.3392803},
}
Figure