Task-Specific Normalization for Continual Learning of Blind Image Quality Models

Weixia Zhang, Kede Ma, Guangtao Zhai, Xiaokang Yang

TL;DR

This paper introduces TSN-IQA, a novel continual learning method for blind image quality assessment (BIQA) that addresses catastrophic forgetting by freezing the feature extractor and learning task-specific normalization parameters. This approach achieves a better trade-off between plasticity (learning new tasks) and stability (retaining old knowledge) with minimal computational overhead, outperforming existing methods.

What Problem Does This Paper Address?

Image quality assessment (IQA) models are crucial for evaluating visual quality, especially in the era of diverse image processing techniques and evolving visual distortions. However, existing IQA models, particularly those based on deep neural networks (DNNs), struggle with "catastrophic forgetting" when trained sequentially on new datasets or tasks. This means that learning a new task can erase knowledge of previously learned tasks, hindering their ability to adapt to the ever-changing landscape of image distortions in real-world applications. The paper aims to develop a continual learning framework for BIQA that can effectively learn from new data without forgetting past knowledge, while also being robust to variations in task order and length.

Methods

The core innovation of TSN-IQA lies in its parameter decomposition strategy. Instead of fine-tuning the entire DNN, the authors freeze all convolutional filters of a pre-trained feature extractor. This ensures stability by preserving learned general image features. For plasticity, they introduce task-specific Batch Normalization (BN) parameters. Each new task (dataset) gets its own set of BN parameters, allowing the model to adapt to new distortions without altering the core feature extraction.

When a new task is encountered, a new prediction head (a simple fully connected layer) is appended. The final quality score is then computed by a weighted summation of predictions from all learned prediction heads. This weighting is determined by a lightweight K-means gating (KG) module. This module learns to assign weights to each prediction head based on the perceptual relevance of the input image to different tasks, effectively selecting the most appropriate "experts" for a given image. This KG module is made more efficient by leveraging features from multiple stages of the network.

Datasets

The authors evaluate their method on six widely used IQA datasets:

  • LIVE [5]
  • CSIQ [56]
  • BID [24]
  • CLIVE [25]
  • KonIQ-10K [26]
  • KADID-10K [57]

These datasets represent a mix of synthetic and realistic distortions, and the authors organize them chronologically for their main experiments to simulate a realistic continual learning scenario. They also use TID2013, SPAQ, and AGIQA-3K for cross-dataset evaluation and generalization analysis.

Key Insights and Ideas

  • Task-specific normalization as a plasticity-stability lever: The paper demonstrates that by isolating adaptation to normalization layers, significant plasticity can be achieved without sacrificing the stability of the core feature extractor. This is a more targeted approach than global fine-tuning or regularization-based methods.
  • The power of a learned perceptual scale: The method's success hinges on learning a common perceptual scale across diverse datasets. By using binary labels derived from relative quality comparisons, they simplify the learning objective and achieve better generalization.
  • K-means gating for efficient ensemble prediction: The KG module provides an elegant way to combine predictions from multiple task-specific heads. Its efficiency, especially when leveraging multi-stage features, is crucial for keeping the model lightweight.
  • Robustness to task order and length: TSN-IQA shows remarkable resilience to the order and number of tasks encountered, a critical requirement for real-world deployment where task arrival is unpredictable.

Key Results

  • Superior performance: TSN-IQA consistently outperforms existing continual learning methods like LwF-KG, SI-KG, and MAS-KG in terms of mSRCC (mean Spearman's rank correlation coefficient) and mPSI (mean plasticity-stability index) across various backbone architectures.
  • Improved plasticity-stability trade-off: The method achieves a better balance between learning new tasks and retaining old knowledge, as evidenced by its mPI and mSI scores.
  • Task-order and length robustness: TSN-IQA demonstrates significantly better performance than LwF-KG when tasks are presented in different orders and for varying task sequence lengths.
  • Generalizability: The model shows good generalization capabilities to unseen datasets, with the KG module effectively assigning weights to prediction heads based on dataset similarities.
  • Efficiency: TSN-IQA introduces a negligible increase in model size and computational complexity, making it practical for deployment.

Limitations and Future Work

  • Computational complexity for inference: While training is efficient, inference requires loading task-specific BN parameters for each task, leading to a computational complexity that scales linearly with the number of tasks. The authors mention a "hard assignment" variant that reduces this but at a slight performance cost.
  • AGIQA-3K handling: The paper notes that TSN-IQA exhibits some uncertainty in handling artificially generated images like those in AGIQA-3K, suggesting room for improvement in this area.
  • Future work: The authors suggest exploring more advanced backbone networks and further optimizing the KG module for even greater efficiency and accuracy. They also highlight the potential for online continual learning, where tasks are not clearly delineated, which aligns with the proposed method's ability to learn a common perceptual scale.

Summary updated

Abstract

The computational vision community has recently paid attention to continual learning for blind image quality assessment (BIQA). The primary challenge is to combat catastrophic forgetting of previously-seen IQA datasets (i.e., tasks). In this paper, we present a simple yet effective continual learning method for BIQA with improved quality prediction accuracy, plasticity-stability trade-off, and task-order/length robustness. The key step in our approach is to freeze all convolution filters of a pre-trained deep neural network (DNN) for an explicit promise of stability, and learn task-specific normalization parameters for plasticity. We assign each new task a prediction head, and load the corresponding normalization parameters to produce a quality score. The final quality estimate is computed by feature fusion and adaptive weighting using hierarchical representations, without leveraging the test-time oracle. Extensive experiments on six IQA datasets demonstrate the advantages of the proposed method in comparison to previous training techniques for BIQA.
BibTeX
@article{zhang2021task,
  author = {Weixia Zhang and Kede Ma and Guangtao Zhai and Xiaokang Yang},
  title = {{Task-Specific Normalization for Continual Learning of Blind Image Quality Models}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  doi = {10.1109/tip.2024.3371349},
}
Figure