Introduction
Assessing the quality of User-Generated Content (UGC) is a significant challenge in computer vision. Unlike professional content, UGC often suffers from complex, overlapping distortions like camera shake, poor lighting, and compression artifacts. Traditional Video Quality Assessment (VQA) models typically treat this as a simple regression task, failing to account for the active nature of human perception.
Humans do not merely observe a frame; we mentally "repair" perceived distortions based on the "free-energy" principle and actively scan scenes using specific gaze dynamics. EyeSim-VQA is a novel framework designed to mimic these biological processes, achieving state-of-the-art performance by simulating "self-repair" and eye-scanning mechanisms.
Methods
The EyeSim-VQA framework employs a dual-branch architecture to process video content through two distinct lenses:
- View Decomposition: The input video is split into an Aesthetic Branch, which uses resized full-frames to capture global composition, and a Technical Branch, which uses grid-based mini-patches to preserve local textures and fine-grained distortions.
- Free-Energy-Guided Self-Repair: To simulate the brain's denoising capability, each branch incorporates a visual enhancement module ($E_A$ and $E_T$). The Aesthetic branch utilizes a lightweight super-resolution model (BasicVSR-mini), while the Technical branch uses iterative residual blocks (CleanNet).
- The EyeSim-VQA Head: This module performs the final quality regression using two biologically inspired sub-branches:
- Gaze Branch: Employs Dynamic Transformers (DyT) to model fixational behavior on salient regions.
- Scan Branch: Utilizes a Mamba-based architecture to simulate the horizontal "sweeping" motion of the human eye across a scene.
Key Insights
A key innovation is the use of the enhanced video as a "pseudo-reference." By concatenating original features with enhanced ones, the model can quantify the degree of "repair" required, which serves as a direct proxy for the original video's quality. Furthermore, repurposing Mamba's linear sequence modeling for eye-scanning simulation provides a computationally efficient way to capture temporal gaze dynamics without the overhead of large language models.
Results and Performance
EyeSim-VQA was evaluated across five major VQA benchmarks, including KVQ, KoNViD-1k, and YouTube-UGC. As shown in the results table, the model consistently achieves state-of-the-art (SOTA) or highly competitive performance across diverse datasets.
On the KoNViD-1k dataset, EyeSim-VQA reached an SROCC of 0.919, outperforming complex CLIP-based models. Its strong performance on YouTube-UGC (0.902 SROCC) and in cross-dataset testing demonstrates robust generalization capabilities. While the current model requires a two-stage training process for enhancement and regression, its success highlights the value of integrating biological priors into automated quality assessment systems.
Summary updated