BPCLIP: A Bottom-up Image Quality Assessment from Distortion to Semantics Based on CLIP

Chenyue Song, Hui Chen, Wěi Zhāng, Haiqi Zhu, Shaohui Liu, Hongbin Huang, Feng Jiang2025 · DOI

TL;DR

This paper introduces BPCLIP, a novel bottom-up approach for Image Quality Assessment (IQA) that leverages CLIP's multimodal capabilities. By progressively analyzing how low-level distortions impact high-level semantic content, BPCLIP achieves state-of-the-art performance on various IQA benchmarks while offering improved interpretability and robustness.

What Problem Does This Paper Address?

Evaluating image quality is crucial for many applications, from image processing to content storage. Traditional Image Quality Assessment (IQA) methods often rely on combining features from different image scales. However, these methods frequently use simple linear fusions, which struggle to capture the complex interplay between low-level distortions (like blur or noise) and the high-level semantic meaning of an image. This limitation can lead to inaccurate quality assessments, especially when different distortions result in similar semantic content. Furthermore, existing IQA models often lack a direct connection to human language, making their assessments less intuitive and explainable.

Methods

BPCLIP tackles these challenges with a two-pronged approach:

  • Bottom-up Image Encoder: Instead of traditional top-down feature extraction, BPCLIP employs a bottom-up strategy. It uses a ResNet backbone to extract multiscale features from images. These features are then processed through a novel Multiscale Cross Attention (MSCA) module. This MSCA module is the core innovation, allowing information to flow from lower-level feature maps to higher-level ones. This enables the model to progressively understand how subtle distortions at the pixel level affect the overall semantic understanding of the image. A parallel "weight branch" using self-attention (SA) helps focus on semantically important regions.
  • CLIP Integration for Semantic Understanding: BPCLIP leverages the power of CLIP (Contrastive Language-Image Pre-training). The image encoder's output is mapped into CLIP's shared feature space. Crucially, the authors introduce 40 image quality adjectives across six dimensions, which are fed into CLIP's text encoder. By calculating the similarity between these text descriptions of quality and the image features, BPCLIP bridges the gap between objective image quality and human language perception, enhancing interpretability.

Datasets

The paper evaluates BPCLIP on several widely used IQA datasets:

  • Full-Reference (FR) Datasets: LIVE, CSIQ, TID2013, KADID-10k.
  • No-Reference (NR) Datasets: CLIVE, KonIQ-10k, SPAQ.

The SPAQ dataset images were resized to have a shorter side of 448 pixels for consistency.

Key Insights and Ideas

  • Bottom-up is Better for IQA: The paper convincingly argues that a bottom-up approach, where low-level features inform high-level understanding, is more effective for IQA than traditional top-down methods. This allows for a more nuanced capture of distortion effects.
  • MSCA as the Key Enabler: The Multiscale Cross Attention (MSCA) module is highlighted as the critical component that facilitates the bottom-up information flow, enabling the model to learn the impact of distortions on semantic content.
  • Bridging Vision and Language for Explainability: Integrating CLIP with carefully chosen quality adjectives provides a direct link between image quality scores and human language, making the assessment process more interpretable and potentially more aligned with human judgment.
  • Efficiency and Robustness: BPCLIP achieves strong performance with a relatively efficient ResNet backbone, outperforming many transformer-based methods in terms of computational complexity while also demonstrating excellent cross-dataset generalization.

Key Results

BPCLIP achieves state-of-the-art or competitive performance across numerous public FR and NR IQA benchmarks. For instance, it demonstrates superior results on datasets like LIVE, CSIQ, TID2013, KADID-10k, CLIVE, KonIQ-10k, and SPAQ. The paper also shows that BPCLIP is more efficient than many recent deep learning-based FR methods, with inference times roughly half that of transformer-based approaches. Ablation studies confirm that each component (Information Branch, Weight Branch, MSCA, CLIP Text Encoder) contributes positively to the overall performance, with MSCA showing the most significant impact.

Limitations and Future Work

The paper does not explicitly detail limitations or extensive future work directions. However, the reliance on CLIP's pre-trained text encoder implies that the quality of the generated text descriptors and the CLIP model's inherent biases could influence the results. Future work could explore more sophisticated methods for generating quality-related text prompts or investigate the model's performance on even more diverse and challenging distortion types. Further research into fine-tuning the CLIP text encoder for IQA tasks could also yield improvements.

Summary updated

Abstract

Image Quality Assessment (IQA) aims to evaluate the perceptual quality of images based on human subjective perception. Existing methods generally combine multiscale features to achieve high performance, but most rely on straightforward linear fusion of these features, which may not adequately capture the impact of distortions on semantic content. To address this, we propose a bottom-up image quality assessment approach based on the Contrastive Language-Image Pre-training (CLIP, a recently proposed model that aligns images and text in a shared feature space), named BPCLIP, which progressively extracts the impact of low-level distortions on high-level semantics. Specifically, we utilize an encoder to extract multiscale features from the input image and introduce a bottom-up multiscale cross attention module designed to capture the relationships between shallow and deep features. In addition, by incorporating 40 image quality adjectives across six distinct dimensions, we enable the pre-trained CLIP text encoder to generate representations of the intrinsic quality of the image, thereby strengthening the connection between image quality perception and human language. Our method achieves superior results on most public Full-Reference (FR) and No-Reference (NR) IQA benchmarks, while demonstrating greater robustness.
BibTeX
@misc{song2025bpclip,
  author = {Chenyue Song and Hui Chen and Wěi Zhāng and Haiqi Zhu and Shaohui Liu and Hongbin Huang and Feng Jiang},
  title = {{BPCLIP: A Bottom-up Image Quality Assessment from Distortion to Semantics Based on CLIP}},
  year = {2025},
  doi = {10.1109/icme59968.2025.11209992},
}

Loading related content...

Figure