Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment

Yongxu Liu, Yinghui Quan, Guoyao Xiao, Aobo Li, Jinjian Wu

Assessing the quality of an image or video (IQA/VQA) requires a model to act as a polymath, identifying minute local distortions like compression noise while maintaining an understanding of the global semantic context. Historically, this has presented a significant challenge in computer vision: resizing an image destroys local details, while cropping it loses the broader scene.

The Local-Global Paradox

The central challenge in quality assessment is balancing high-resolution local details with global semantics. High resolution is necessary to detect graininess or blur, but the entire scene is required to determine if a lack of detail is a natural feature (like a plain wall) or a rendering artifact.

Traditional sampling often forces a choice between high-resolution local fragments and low-resolution global views.

To address this, current state-of-the-art models often employ complex multi-branch architectures that process different scales in parallel. While effective, these models are computationally expensive. The SAMA (Scaling and Masking) paradigm proposes a data-centric solution: packing multi-resolution information into a single, standard-sized input.

The SAMA Methodology

SAMA introduces a three-step preprocessing workflow designed to create a compact representation of multi-scale data:

  1. Pyramid Scaling: The original high-resolution input is downsampled into a pyramid of varying resolutions.
  2. Grid-based Fragment Sampling: Instead of a single large crop, the model divides each scale into a grid and samples small fragments from each.
  3. Masking and Interlacing: Inspired by the Bayer filter in camera sensors, SAMA "masks" these fragments together. For images, a spatial checkerboard mask alternates between raw-resolution and downsampled fragments. For videos, temporal masking is used, where different frames represent different scales.
The SAMA pipeline: fragments from different scales are sampled and interlaced using a checkerboard mask to create a single multi-scale input.

By leveraging the inherent spatial and temporal redundancy in visual data, SAMA allows a standard single-branch model (such as a SwinTransformer) to perceive multiple scales simultaneously without increasing the input size or model complexity.

Key Results and Insights

The researchers validated SAMA across major benchmarks, including LSVQ for video and KonIQ-10k for images. The results demonstrate that intelligent data formatting can outperform architectural complexity:

  • Efficiency: SAMA maintains a standard 224x224 input size, providing performance gains with negligible extra computational cost.
  • Performance: On the LSVQ dataset, SAMA improved the Spearman Rank Correlation Coefficient (SRCC) from 0.870 to 0.883, surpassing complex multi-branch models like DOVER. In IQA tasks on the KonIQ dataset, it achieved a state-of-the-art SRCC of 0.930.
  • Model Agnostic: As a preprocessing step, SAMA can be integrated into almost any existing transformer-based pipeline.

Conclusion

SAMA proves that redundancy in visual data is an asset rather than a burden. By strategically interlacing data from different scales, researchers can "trick" standard models into capturing multi-scale features. Future work will explore SAMA's applicability to other resolution-sensitive tasks, such as image restoration and super-resolution, and its performance across a wider variety of backbone architectures.

Source Paper: Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment
Code: GitHub - Sissuire/SAMA

Summary updated

Abstract

Quality assessment of images and videos emphasizes both local details and global semantics, whereas general data sampling methods (e.g., resizing, cropping or grid-based fragment) fail to catch them simultaneously. To address the deficiency, current approaches have to adopt multi-branch models and take as input the multi-resolution data, which burdens the model complexity. In this work, instead of stacking up models, a more elegant data sampling method (named as SAMA, scaling and masking) is explored, which compacts both the local and global content in a regular input size. The basic idea is to scale the data into a pyramid first, and reduce the pyramid into a regular data dimension with a masking strategy. Benefiting from the spatial and temporal redundancy in images and videos, the processed data maintains the multi-scale characteristics with a regular input size, thus can be processed by a single-branch model. We verify the sampling method in image and video quality assessment. Experiments show that our sampling method can improve the performance of current single-branch models significantly, and achieves competitive performance to the multi-branch models without extra model complexity. The source code will be available at https://github.com/Sissuire/SAMA.
BibTeX
@inproceedings{liu2024scaling,
  author = {Yongxu Liu and Yinghui Quan and Guoyao Xiao and Aobo Li and Jinjian Wu},
  title = {{Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28170},
}
Figure