FreqAlign: Excavating Perception-Oriented Transferability for Blind Image Quality Assessment From a Frequency Perspective

Xin Li, Yiting Lu, Zhibo Chen

Abstract

Blind Image Quality Assessment (BIQA) is susceptible to poor transferability when the distribution shift occurs, e.g., from synthesis degradation to authentic degradation. To mitigate this, some studies have attempted to design unsupervised domain adaptation (UDA) based schemes for BIQA, which intends to eliminate the domain shift through adversarial-based feature alignment. However, the feature alignment is usually taken at the low-frequency space of features since the global average pooling operation. This ignores the transferable perception knowledge in other frequency components and causes the sub-optimal solution for the UDA of BIQA. To overcome this, from a novel frequency perspective, we propose an effective alignment strategy, i.e., Frequency Alignment (dubbed FreqAlign), to excavate the perception-oriented transferability of BIQA in the frequency space. Concretely, we study what frequency components of features are more proper for perception-oriented alignment. Based on this, we propose to improve the perception-oriented transferability of BIQA by performing feature frequency decomposition and selecting the frequency components that contained the most transferable perception knowledge for alignment. To achieve a stable and effective frequency selection, we further propose the frequency movement with a sliding window to find the optimal frequencies for alignment, which is composed of three strategies, i.e., warm up with pre-training, frequency movement-based selection, and perturbation-based finetuning. Extensive experiments under different domain adaptation settings of BIQA have validated the effectiveness of our proposed method.

Introduction

Blind Image Quality Assessment (BIQA) aims to predict how humans perceive image quality without a reference "clean" image. While deep learning has made significant strides, these models often suffer from a major "domain gap." A model trained on synthetic distortions (like Gaussian blur or JPEG noise) often fails when tested on authentic, real-world distortions (like low-light noise or camera shake).

To address this, researchers use Unsupervised Domain Adaptation (UDA) to align features between a labeled "source" domain and an unlabeled "target" domain. However, most existing methods align features after a Global Average Pooling (GAP) operation. In signal processing terms, GAP acts as a low-pass filter, meaning current models only align low-frequency components. This ignores high-frequency perceptual cues—like fine textures and sharp edges—that are critical for judging image quality. FreqAlign is a novel UDA framework that shifts the alignment process to the frequency domain, using a sliding window to identify "perception-oriented" frequency bands to boost performance across datasets.

The FreqAlign Framework

FreqAlign moves beyond standard spatial alignment by decomposing features into their frequency components to find the most transferable information.

  1. Frequency Decomposition: Instead of simple pooling, the model applies a 2D Discrete Cosine Transform (DCT) to the feature maps. This decomposes the features into an $8 \times 8$ grid of frequency components, ranging from low (top-left) to high (bottom-right).
  2. Frequency Movement with a Sliding Window: Since not all frequencies are equally useful for transfer, the authors use a sliding window to group continuous frequencies into "bands." They move this window along a trajectory to find the band that offers the best "perception-oriented transferability."
  3. Three-Stage Training Strategy:
    • Warm-up: Pre-train the backbone and discriminator on the lowest frequency band to ensure a stable starting point.
  4. Frequency Movement-based Selection: Iterate through frequency bands to find the one with the highest transferability (measured via Maximum Mean Discrepancy).
  5. Perturbation-based Finetuning: Fine-tune the model around the selected optimal band to provide tolerance against unstable frequency selection.
  6. Adversarial Alignment: Once the optimal band is found, a discriminator is used to make the source and target features in that specific frequency range indistinguishable.

Experimental Results

The authors evaluated FreqAlign across several challenging cross-domain scenarios involving synthetic datasets (KADID10k, CSIQ) and authentic datasets (KonIQ-10k, CLIVE, BID).

FreqAlign consistently outperforms both the Baseline (no adaptation) and DANN (standard spatial adversarial alignment) across various domain adaptation settings.

FreqAlign consistently outperformed both the "Baseline" (no adaptation) and "DANN" (standard spatial adversarial alignment). On the KADID10k $\rightarrow$ KonIQ-10k task, FreqAlign achieved an SROCC of 0.7478, a significant jump over the Baseline (0.6346) and DANN (0.6382). Even when moving from complex real-world data to simpler synthetic data, FreqAlign showed superior stability. The method also proved versatile, improving aesthetic quality assessment models on the AVA and TAD66k datasets.

Key Insights

Analysis of the frequency components reveals several critical findings for BIQA transferability:

  • Perception is Multi-Frequency: The most transferable knowledge isn't always in the low frequencies. For some domain shifts, high-frequency components provide better cues for alignment.
  • Scenario-Dependent Alignment: The "optimal" frequency for alignment is not fixed; it changes depending on the specific source and target domains. This justifies the automated sliding window selection process.
  • The "Clustering" Effect: Frequencies with similar perceptual performance tend to cluster together in the DCT space, making a sliding window approach more effective than selecting isolated frequencies.
Heatmaps of SROCC values for different frequency components across four domain adaptation scenarios, illustrating that the optimal alignment frequency is scenario-dependent.

Conclusion

FreqAlign demonstrates that excavating perception-oriented transferability from a frequency perspective is highly effective for BIQA. By moving beyond low-frequency spatial alignment, the model captures critical perceptual cues that are otherwise lost. While the current sliding window uses a simple trajectory, future work could explore more complex paths or differentiable frequency selection to further optimize performance and reduce training overhead.

Summary updated

Abstract

Blind Image Quality Assessment (BIQA) is susceptible to poor transferability when the distribution shift occurs, e.g., from synthesis degradation to authentic degradation. To mitigate this, some studies have attempted to design unsupervised domain adaptation (UDA) based schemes for BIQA, which intends to eliminate the domain shift through adversarial-based feature alignment. However, the feature alignment is usually taken at the low-frequency space of features since the global average pooling operation. This ignores the transferable perception knowledge in other frequency components and causes the sub-optimal solution for the UDA of BIQA. To overcome this, from a novel frequency perspective, we propose an effective alignment strategy, i.e., Frequency Alignment (dubbed FreqAlign), to excavate the perception-oriented transferability of BIQA in the frequency space. Concretely, we study what frequency components of features are more proper for perception-oriented alignment. Based on this, we propose to improve the perception-oriented transferability of BIQA by performing feature frequency decomposition and selecting the frequency components that contained the most transferable perception knowledge for alignment. To achieve a stable and effective frequency selection, we further propose the frequency movement with a sliding window to find the optimal frequencies for alignment, which is composed of three strategies, i.e., warm up with pre-training, frequency movement-based selection, and perturbation-based finetuning. Extensive experiments under different domain adaptation settings of BIQA have validated the effectiveness of our proposed method.
BibTeX
@article{li2023freqalign,
  author = {Xin Li and Yiting Lu and Zhibo Chen},
  title = {{FreqAlign: Excavating Perception-Oriented Transferability for Blind Image Quality Assessment From a Frequency Perspective}},
  journal = {IEEE Transactions on Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2023.3325755},
}
Figure