TL;DR
EyeSim-VQA is a novel video quality assessment framework that mimics the human visual system's ability to "self-repair" perceived distortions and scan scenes via gaze dynamics. By combining dual-branch visual enhancement with a biologically inspired prediction head using Mamba and Dynamic Transformers (DyT), it achieves state-of-the-art performance across major benchmarks without the heavy computational overhead of large language models.
What Problem Does This Paper Address?
Assessing the quality of User-Generated Content (UGC) is notoriously difficult. Unlike professional broadcasts, UGC is plagued by diverse, overlapping distortions—shaky cameras, poor lighting, and aggressive compression.
Traditional Video Quality Assessment (VQA) models often treat the problem as a straightforward regression task. However, human perception is more active: our brains use a "free-energy" principle to mentally "repair" or denoise what we see, and our eyes don't just stare at a frame—they sweep across it (scanning) and fixate on salient details (gaze). The authors identify a gap in existing VQA research: models don't sufficiently account for these biological "self-repair" and scanning mechanisms, leading to predictions that don't always align with human subjective judgments.
Methods
The authors propose a dual-branch architecture designed to simulate different aspects of human vision:
- View Decomposition: The input video is split into an Aesthetic Branch (resized full-frames for global context) and a Technical Branch (patch-based fragments to preserve fine-grained local details).
- Free-Energy-Guided Self-Repair: To simulate the brain's "denoising" process, each branch includes a visual enhancement module.
- The Aesthetic branch uses BasicVSR-mini (a lightweight super-resolution variant) to refine global appearance.
- The Technical branch uses CleanNet (iterative residual blocks) to restore local structural integrity.
- The EyeSim-VQA Head: This is the core innovation for quality regression. It features two sub-modules:
- Gaze Branch: Uses Dynamic Transformers (DyT) to model fixational behavior, focusing on salient regions.
- Scan Branch: Uses a Mamba-based architecture. Mamba’s linear sequence modeling is used here to mimic the horizontal "sweeping" motion of the human eye as it scans a scene.
- Feature Integration: Instead of replacing the original video features with enhanced ones, the model concatenates them. This allows the network to compare the "distorted" reality with the "repaired" ideal to judge quality.
Datasets
The model was evaluated on five major public VQA benchmarks:
- KVQ (Kwai Video Quality)
- KoNViD-1k
- YouTube-UGC
- LIVE-VQC
- LSVQ (including the 1080p subset)
To train the "self-repair" enhancement modules, the authors also created a synthetic distortion dataset based on KoNViD-1k, applying noise, blur, and JPEG compression.
Key Insights and Ideas
- Enhancement as a Reference: One of the most interesting findings is that the model performs best when it sees both the original and the enhanced video. The enhanced version acts as a "pseudo-reference," helping the model quantify how much "repair" was necessary—a direct proxy for quality.
- Mamba for Eye Movements: While Mamba is usually discussed in the context of long-sequence modeling (like LLMs), the authors cleverly repurpose its scanning mechanism to simulate the physical act of human eye movement across a frame.
- Simplicity in Fusion: Through ablation studies, the authors found that "Simple Concatenation" of features outperformed more complex adapter-based integration. This suggests that standard backbones are robust enough to handle the extra "guidance" signal without needing architectural surgery.
- Branch Specialization: Visualization of the attention maps (Fig. 6) shows that the Aesthetic and Technical branches naturally learn to look at different things—the former at global composition and the latter at specific textures—validating the dual-pathway approach.
Key Results
EyeSim-VQA achieved State-of-the-Art (SOTA) or competitive results across all tested datasets.
- On KoNViD-1k, it reached an SROCC of 0.919, outperforming recent CLIP-based and LLM-based models like KSVQE.
- On YouTube-UGC, it achieved an SROCC of 0.902, showing strong generalization to diverse content.
- In Cross-Dataset tests (training on one dataset and testing on another), EyeSim-VQA showed superior robustness, suggesting that its biologically inspired design captures more universal quality cues than purely data-driven models.
Limitations and Future Work
- Pre-training Requirements: The current framework requires a two-stage process: pre-training the enhancement modules on synthetic data before training the VQA head. A fully end-to-end, single-stage training pipeline would be more efficient.
- Semantic Gap: While the model excels at perceptual and technical quality, it lacks the high-level semantic reasoning of Large Language Models (LLMs). The authors suggest that integrating LLM-based knowledge for "content-aware" assessment is a promising future direction.
- Backbone Constraints: The enhancement modules are currently built on relatively "classical" architectures (BasicVSR). Exploring more modern, task-specific backbones could further boost the "self-repair" simulation.
Summary updated