Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment

Yongxu Liu, Yinghui Quan, Guoyao Xiao, Aobo Li, Jinjian Wu

Assessing the quality of an image or video (IQA/VQA) requires a model to act as a polymath, identifying minute local distortions like compression noise while maintaining an understanding of the global semantic context. Historically, this has presented a significant challenge in computer vision: resizing an image destroys local details, while cropping it loses the broader scene.

The Local-Global Paradox

The central challenge in quality assessment is balancing high-resolution local details with global semantics. High resolution is necessary to detect graininess or blur, but the entire scene is required to determine if a lack of detail is a natural feature (like a plain wall) or a rendering artifact.

Traditional sampling often forces a choice between high-resolution local fragments and low-resolution global views.

To address this, current state-of-the-art models often employ complex multi-branch architectures that process different scales in parallel. While effective, these models are computationally expensive. The SAMA (Scaling and Masking) paradigm proposes a data-centric solution: packing multi-resolution information into a single, standard-sized input.

The SAMA Methodology

SAMA introduces a three-step preprocessing workflow designed to create a compact representation of multi-scale data:

  1. Pyramid Scaling: The original high-resolution input is downsampled into a pyramid of varying resolutions.
  2. Grid-based Fragment Sampling: Instead of a single large crop, the model divides each scale into a grid and samples small fragments from each.
  3. Masking and Interlacing: Inspired by the Bayer filter in camera sensors, SAMA "masks" these fragments together. For images, a spatial checkerboard mask alternates between raw-resolution and downsampled fragments. For videos, temporal masking is used, where different frames represent different scales.
The SAMA pipeline: fragments from different scales are sampled and interlaced using a checkerboard mask to create a single multi-scale input.

By leveraging the inherent spatial and temporal redundancy in visual data, SAMA allows a standard single-branch model (such as a SwinTransformer) to perceive multiple scales simultaneously without increasing the input size or model complexity.

Key Results and Insights

The researchers validated SAMA across major benchmarks, including LSVQ for video and KonIQ-10k for images. The results demonstrate that intelligent data formatting can outperform architectural complexity:

  • Efficiency: SAMA maintains a standard 224x224 input size, providing performance gains with negligible extra computational cost.
  • Performance: On the LSVQ dataset, SAMA improved the Spearman Rank Correlation Coefficient (SRCC) from 0.870 to 0.883, surpassing complex multi-branch models like DOVER. In IQA tasks on the KonIQ dataset, it achieved a state-of-the-art SRCC of 0.930.
  • Model Agnostic: As a preprocessing step, SAMA can be integrated into almost any existing transformer-based pipeline.

Conclusion

SAMA proves that redundancy in visual data is an asset rather than a burden. By strategically interlacing data from different scales, researchers can "trick" standard models into capturing multi-scale features. Future work will explore SAMA's applicability to other resolution-sensitive tasks, such as image restoration and super-resolution, and its performance across a wider variety of backbone architectures.

Source Paper: Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment
Code: GitHub - Sissuire/SAMA

Summary updated

BibTeX
@inproceedings{liu2024scaling,
  author = {Yongxu Liu and Yinghui Quan and Guoyao Xiao and Aobo Li and Jinjian Wu},
  title = {{Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28170},
}
Figure