JNDMix: JND-Based Data Augmentation for No-reference Image Quality Assessment

Jiamu Sheng, Jiayuan Fan, Peng Ye, Jianjian Cao

Abstract

No-reference image quality assessment (NR-IQA) models often face performance bottlenecks due to over-fitting on limited datasets. While data augmentation is a standard solution in computer vision, traditional methods frequently alter image quality, creating a mismatch between the augmented image and its original quality label. We propose JNDMix, a novel data augmentation technique based on Just Noticeable Difference (JND) noise mixing. By injecting imperceptible noise into training images, JNDMix enriches dataset diversity without compromising label integrity. Our experiments demonstrate that JNDMix significantly enhances the performance, data efficiency, and generalization of state-of-the-art NR-IQA models, including MANIQA, ResNet, and HyperIQA, across major benchmarks like LIVEC and KonIQ-10k.

Introduction

No-reference image quality assessment (NR-IQA) aims to predict human-perceived image quality without a clean reference image. This task is critical for various applications, from social media optimization to medical imaging. However, training robust NR-IQA models is challenging because human subjective scoring—the "ground truth"—is expensive and difficult to scale. Consequently, existing datasets are relatively small, leading to significant over-fitting.

In general computer vision tasks, data augmentation (e.g., cropping, flipping, or adding Gaussian noise) is used to improve model robustness. However, in the context of IQA, these techniques are often counterproductive. Adding visible noise or blurring an image fundamentally changes its quality score. If a high-quality image is augmented with visible distortion but retains its "high quality" label, the model receives contradictory signals, leading to poor convergence and performance degradation.

The JNDMix Methodology

To address the label mismatch problem, we introduce JNDMix. This method leverages the Human Visual System's (HVS) limitations, specifically the Just Noticeable Difference (JND). JND represents the maximum amount of pixel-level change that can occur before a human observer perceives a difference.

The JNDMix workflow consists of four primary steps:

  1. JND Map Generation: For each training image, a JND map is calculated using a pre-existing model (such as RGB-JND). This map identifies the visibility threshold for every pixel based on local luminance and texture masking.
  2. Random Noise Scaling: To ensure the noise remains truly imperceptible, the JND map is multiplied by a random scaling factor $\lambda$, where $0 < \lambda \leq 1$.
  3. Bipolar Injection: A random matrix $r$ (containing values of $1$ or $-1$) is generated. The final noise is the element-wise product of the scaled JND map and this random matrix. This noise is then added to the original image.
  4. Label Preservation: Because the added noise is mathematically present but visually imperceptible, the original subjective quality score ($y$) remains a valid label for the augmented image ($\tilde{y} = y$).

This approach allows the model to encounter a unique version of the image in every training epoch, effectively regularizing the network without the destructive effects of visible noise.

Experimental Evaluation

We evaluated JNDMix on two prominent NR-IQA benchmarks:

  • LIVEC: A dataset featuring 1,162 real-world images with diverse, authentic distortions.
  • KonIQ-10k: A larger-scale dataset containing 10,073 images curated from public multimedia sources.

The method was integrated into several architectures, including standard CNNs (ResNet34, ResNet50), specialized IQA networks (DBCNN, HyperIQA), and the Transformer-based MANIQA.

Key Results and Insights

State-of-the-Art Performance

JNDMix consistently improved the performance of all tested models. Notably, when applied to MANIQA, the method achieved state-of-the-art results on both benchmarks, reaching a Spearman Rank Correlation Coefficient (SRCC) of 0.897 on LIVEC and 0.927 on KonIQ-10k.

Boosting Vision Transformers

Vision Transformers (ViTs) are known to be "data-hungry," often underperforming on small datasets. JNDMix proved particularly effective for ViT-based models. For a standard ViT, the SRCC on the LIVEC dataset jumped from 0.776 to 0.815. This suggests that JNDMix provides the necessary diversity to help Transformers learn robust features even when labeled data is scarce.

Data Efficiency and Generalization

In "low-data" experiments—where models were trained on only 10% of the available data—JNDMix provided a significant performance buffer. For instance, HyperIQA saw an SRCC increase of +0.040 on LIVEC compared to the baseline. Furthermore, models trained with JNDMix showed superior generalization in cross-dataset evaluations, indicating that the noise injection helps the model focus on intrinsic quality features rather than dataset-specific artifacts.

Comparison with Standard Noise

A critical finding of our study was the comparison between JNDMix and standard Gaussian noise. While JNDMix improved model accuracy, the addition of Gaussian noise (which is often visible) actually degraded performance. This confirms that preserving the integrity of the quality label is paramount in IQA augmentation.

Conclusion

JNDMix offers a simple yet powerful solution to the data scarcity and label mismatch problems in NR-IQA. By staying "under the radar" of the human visual system, it provides effective regularization that enhances model robustness and data efficiency. As NR-IQA models continue to evolve toward more complex architectures like Transformers, perceptually-aware augmentation techniques like JNDMix will be essential for reaching new levels of performance and generalization.

Summary updated

Abstract

Despite substantial progress in no-reference image quality assessment (NR-IQA), previous training models often suffer from over-fitting due to the limited scale of used datasets, resulting in model performance bottlenecks. To tackle this challenge, we explore the potential of leveraging data augmentation to improve data efficiency and enhance model robustness. However, most existing data augmentation methods incur a serious issue, namely that it alters the image quality and leads to training images mismatching with their original labels. Additionally, although only a few data augmentation methods are available for NR-IQA task, their ability to enrich dataset diversity is still insufficient. To address these issues, we propose a effective and general data augmentation based on just noticeable difference (JND) noise mixing for NR-IQA task, named JNDMix. In detail, we randomly inject the JND noise, imperceptible to the human visual system (HVS), into the training image without any adjustment to its label. Extensive experiments demonstrate that JNDMix significantly improves the performance and data efficiency of various state-of-the-art NR-IQA models and the commonly used baseline models, as well as the generalization ability. More importantly, JNDMix facilitates MANIQA to achieve the state-of-the-art performance on LIVEC and KonIQ-10k.
BibTeX
@inproceedings{sheng2023jndmix,
  author = {Jiamu Sheng and Jiayuan Fan and Peng Ye and Jianjian Cao},
  title = {{JNDMix: JND-Based Data Augmentation for No-reference Image Quality Assessment}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=6Vp1ash4Vh},
}
Figure