Assessing the quality of an image or video (IQA/VQA) requires a model to act as a polymath, identifying minute local distortions like compression noise while maintaining an understanding of the global semantic context. Historically, this has presented a significant challenge in computer vision: resizing an image destroys local details, while cropping it loses the broader scene.
The Local-Global Paradox
The central challenge in quality assessment is balancing high-resolution local details with global semantics. High resolution is necessary to detect graininess or blur, but the entire scene is required to determine if a lack of detail is a natural feature (like a plain wall) or a rendering artifact.
To address this, current state-of-the-art models often employ complex multi-branch architectures that process different scales in parallel. While effective, these models are computationally expensive. The SAMA (Scaling and Masking) paradigm proposes a data-centric solution: packing multi-resolution information into a single, standard-sized input.
The SAMA Methodology
SAMA introduces a three-step preprocessing workflow designed to create a compact representation of multi-scale data:
- Pyramid Scaling: The original high-resolution input is downsampled into a pyramid of varying resolutions.
- Grid-based Fragment Sampling: Instead of a single large crop, the model divides each scale into a grid and samples small fragments from each.
- Masking and Interlacing: Inspired by the Bayer filter in camera sensors, SAMA "masks" these fragments together. For images, a spatial checkerboard mask alternates between raw-resolution and downsampled fragments. For videos, temporal masking is used, where different frames represent different scales.
By leveraging the inherent spatial and temporal redundancy in visual data, SAMA allows a standard single-branch model (such as a SwinTransformer) to perceive multiple scales simultaneously without increasing the input size or model complexity.
Key Results and Insights
The researchers validated SAMA across major benchmarks, including LSVQ for video and KonIQ-10k for images. The results demonstrate that intelligent data formatting can outperform architectural complexity:
- Efficiency: SAMA maintains a standard 224x224 input size, providing performance gains with negligible extra computational cost.
- Performance: On the LSVQ dataset, SAMA improved the Spearman Rank Correlation Coefficient (SRCC) from 0.870 to 0.883, surpassing complex multi-branch models like DOVER. In IQA tasks on the KonIQ dataset, it achieved a state-of-the-art SRCC of 0.930.
- Model Agnostic: As a preprocessing step, SAMA can be integrated into almost any existing transformer-based pipeline.
Conclusion
SAMA proves that redundancy in visual data is an asset rather than a burden. By strategically interlacing data from different scales, researchers can "trick" standard models into capturing multi-scale features. Future work will explore SAMA's applicability to other resolution-sensitive tasks, such as image restoration and super-resolution, and its performance across a wider variety of backbone architectures.
Source Paper: Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment
Code: GitHub - Sissuire/SAMA
Summary updated