JNDMix: Jnd-Based Data Augmentation for No-Reference Image Quality Assessment

Jiamu Sheng, Jiayuan Fan, Peng Ye, Jianjian Cao

TL;DR

JNDMix is a novel data augmentation technique designed specifically for No-Reference Image Quality Assessment (NR-IQA). By injecting "Just Noticeable Difference" (JND) noise—distortions that are mathematically present but imperceptible to the human eye—the method enriches training datasets without invalidating their original quality labels, leading to state-of-the-art performance and significantly better data efficiency.

What Problem Does This Paper Address?

In the world of No-Reference Image Quality Assessment (NR-IQA), the goal is to train models to predict how a human would score an image's quality without having a "perfect" reference image for comparison. The bottleneck for deep learning in this field is the lack of massive labeled datasets; human subjective scoring is expensive and time-consuming.

Standard data augmentation techniques used in computer vision (like Gaussian noise, heavy cropping, or resizing) are problematic for NR-IQA. Why? Because these methods fundamentally alter the image quality. If you take a high-quality image labeled "9/10" and add visible Gaussian noise, that "9/10" label is no longer accurate. This "label mismatch" confuses models and limits the effectiveness of traditional augmentation. Consequently, NR-IQA models often suffer from over-fitting on small, specialized datasets.

Methods

The authors introduce JNDMix, which leverages the characteristics of the Human Visual System (HVS). The core concept is the Just Noticeable Difference (JND)—the maximum amount of change a pixel can undergo before a human observer notices a difference.

The JNDMix workflow functions as follows:

  1. JND Map Generation: For a given training image, a JND map is calculated (using an existing model like RGB-JND) to determine the visibility threshold for every pixel.
  2. Random Noise Scaling: The JND map is multiplied by a random ratio $\lambda$ (between 0 and 1), ensuring the noise stays below the visibility threshold.
  3. Bipolar Injection: A random matrix $r$ consisting of $1$ and $-1$ is used to randomly add or subtract this noise from the original pixels.
  4. Label Preservation: Because the noise is imperceptible, the authors assume the subjective quality score remains unchanged ($\tilde{y} = y$).

This allows the model to see a "new" version of the image in every training iteration while keeping the ground-truth label perfectly valid.

Datasets

The researchers evaluated JNDMix using two standard NR-IQA benchmarks:

  • LIVEC: Contains 1,162 real-world images with complex, composite distortions.
  • KonIQ-10k: A larger-scale dataset with 10,073 images selected from public multimedia databases.

They tested the method across various architectures, including traditional CNNs (ResNet34, ResNet50), specialized IQA models (DBCNN, HyperIQA), and the state-of-the-art Transformer-based model, MANIQA.

Key Insights and Ideas

  • The "Invisible" Advantage: The primary "aha" moment is that data augmentation for quality assessment doesn't need to be visible to be effective. By staying under the perceptual radar, JNDMix provides the regularization benefits of noise injection without the destructive side effects on label integrity.
  • Overcoming the Transformer Data Hunger: Vision Transformers (ViTs) typically require massive amounts of data to perform well. The paper shows that JNDMix is particularly effective for ViT-based models, helping them reach high performance even on the relatively small LIVEC dataset.
  • Generalization is Key: Models trained with JNDMix showed better performance in cross-dataset evaluations (e.g., training on KonIQ-10k and testing on LIVEC). This suggests the model is learning more robust features rather than just memorizing dataset-specific noise patterns.

Key Results

  • State-of-the-Art: Integrating JNDMix into the MANIQA model allowed it to achieve SOTA results on both LIVEC (SRCC of 0.897) and KonIQ-10k (SRCC of 0.927).
  • Massive Boost for ViT: For a standard Vision Transformer, JNDMix improved the Spearman Rank Correlation Coefficient (SRCC) on LIVEC from 0.776 to 0.815—a significant jump for a simple augmentation tweak.
  • Superior Data Efficiency: In "low-data" regimes (using only 10% of the training data), JNDMix provided a substantial performance buffer. For example, HyperIQA with 10% data and JNDMix saw an SRCC increase of +0.040 on LIVEC compared to the baseline.
  • Ablation Success: The authors compared JNDMix against standard Gaussian noise. While JNDMix improved performance, Gaussian noise actually degraded it, proving that label-preserving noise is crucial for this task.

Limitations and Future Work

  • Dependency on JND Models: The effectiveness of JNDMix is inherently tied to the accuracy of the underlying JND estimation model. If the JND map is inaccurate, the noise might become visible, re-introducing the label mismatch problem.
  • Computational Cost: While the augmentation itself is simple, generating JND maps for every image adds a pre-processing step to the pipeline.
  • Future Directions: The paper suggests that exploring more sophisticated Human Visual System models or adaptive JND thresholds could further refine the augmentation process. There is also potential to combine JNDMix with other label-preserving geometric augmentations like horizontal flipping.

Summary updated

Abstract

Despite substantial progress in no-reference image quality assessment (NR-IQA), previous training models often suffer from over-fitting due to the limited scale of used datasets, resulting in model performance bottlenecks. To tackle this challenge, we explore the potential of leveraging data augmentation to improve data efficiency and enhance model robustness. However, most existing data augmentation methods incur a serious issue, namely that it alters the image quality and leads to training images mismatching with their original labels. Additionally, although only a few data augmentation methods are available for NR-IQA task, their ability to enrich dataset diversity is still insufficient. To address these issues, we propose a effective and general data augmentation based on just noticeable difference (JND) noise mixing for NR-IQA task, named JNDMix. In detail, we randomly inject the JND noise, imperceptible to the human visual system (HVS), into the training image without any adjustment to its label. Extensive experiments demonstrate that JNDMix significantly improves the performance and data efficiency of various state-of-the-art NR-IQA models and the commonly used baseline models, as well as the generalization ability. More importantly, JNDMix facilitates MANIQA to achieve the state-of-the-art performance on LIVEC and KonIQ-10k.
BibTeX
@inproceedings{sheng2023jndmix,
  author = {Jiamu Sheng and Jiayuan Fan and Peng Ye and Jianjian Cao},
  title = {{JNDMix: Jnd-Based Data Augmentation for No-Reference Image Quality Assessment}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=6Vp1ash4Vh},
}

Loading related content...

Figure