In most computer vision tasks, performance scales with the size of the training set. Models for object detection or image classification benefit from datasets like ImageNet, which contain millions of labeled examples. However, in the field of Image Quality Assessment (IQA) and Video Quality Assessment (VQA), researchers face a fundamental "data bottleneck": the ground truth for "quality" is human perception, which is notoriously difficult and expensive to collect at scale.
The High Cost of Human Subjectivity
The gold standard for quality assessment is the Mean Opinion Score (MOS), obtained by asking dozens of human subjects to rate images or videos in controlled environments. As Zhou et al. note, the high cost of this subjective testing creates a significant data bottleneck, making it difficult to train robust, end-to-end models.
While a classification model might have millions of samples, Shi et al. point out that subjective quality datasets are often limited to just a few thousand samples. This scarcity is even more pronounced in specialized domains. For instance, Yu 0005 et al. explain that Virtual Reality (VR) datasets are significantly smaller than general computer vision datasets, which limits the effectiveness of data-hungry deep learning approaches. Similarly, Gnanasambandam et al. highlight that the lack of dedicated datasets for Quanta Image Sensors (QIS) forces researchers to rely on deterministic methods rather than state-of-the-art deep learning.
The Generalization Gap
When models are trained on small, homogeneous datasets, they struggle to generalize to the "in-the-wild" distortions found in the real world. Gu et al. argue that metrics trained on only a few hundred samples cannot learn the complex, multi-dimensional features required to evaluate quality without a reference image.
This lack of diversity manifests in several ways:
- Content Bias: Models may perform well on landscapes but fail on human faces. Su et al. observe that face image quality is underserved by current datasets, which contain too few faces to train accurate predictive models.
- Resolution and Technology Gaps: As imaging technology evolves, datasets fall behind. Hosu et al. note a scarcity of expertly annotated data for high-resolution, authentic photographs, while Sun et al. describe how the lack of High Dynamic Range (HDR) video data makes it difficult to develop models that generalize to HDR content.
- Artifact Diversity: Modern compression techniques create new types of visual artifacts. Smirnov et al. argue that current VQA datasets lack the scale and variety of artifacts produced by modern codecs.
Strategies for Mitigating Scarcity
To overcome the limited availability of subject-rated data, researchers have turned to several innovative training strategies:
1. Transfer Learning and Pre-training If we cannot get enough quality-specific labels, we can leverage features learned from other tasks. Kwon et al. propose using Vision-Language Models (like CLIP) to create generalizable feature extractors that focus on image attributes rather than just a single quality score. Liu et al. also demonstrate that transfer learning from deep CNNs can significantly improve performance on AI-generated content where data is scarce.
2. Intelligent Data Augmentation Since we cannot easily collect more human ratings, we can make existing data work harder. Shi et al. introduced DSMix, a strategy that uses "Distortion-Induced Sensitivity Maps" to weight labels during image mixing, helping the model understand where and how much an image is degraded.
3. Large-Scale Benchmark Construction The most direct solution is the creation of massive, diverse datasets to set new baselines. Recent efforts include LS-PCQA for point clouds (Liu et al.), GFIQA-20k for faces (Su et al.), and UHD-IQA for high-resolution photos (Hosu et al.).
The Path Forward
The fundamental challenge remains the "Bayesian" nature of the problem. As Duanmu et al. explain, the scarcity and inconsistency of subject-rated data are not just quantitative problems but qualitative ones, as human ratings themselves vary. Moving forward, the field is shifting from simply "chasing" scores on small datasets to developing models that are robust to acquisition variations (Drobnjak et al.) and capable of zero-shot or "agnostic" recommendation without human-defined rules (Podo et al.).
Key Foundational Work
Entropy Based Data Expansion Method for Blind Image Quality Assessment (2020) — contributed 2 concept elements to this direction.
Quality Assessment on Authentically Distorted Images by Expanding Proxy Labels (2020) — contributed 1 concept elements to this direction.
Learning a No-Reference Quality Assessment Model of Enhanced Images With Big Data (2019) — contributed 1 concept elements to this direction.