EyeSim-VQA: A Free-Energy-Guided Eye Simulation Framework for Video Quality Assessment

Zhaoyang Wang, Wen Lu, Jie Li, Lihuo He, Maoguo Gong, Xinbo Gao

Introduction

Assessing the quality of User-Generated Content (UGC) is a significant challenge in computer vision. Unlike professional content, UGC often suffers from complex, overlapping distortions like camera shake, poor lighting, and compression artifacts. Traditional Video Quality Assessment (VQA) models typically treat this as a simple regression task, failing to account for the active nature of human perception.

Humans do not merely observe a frame; we mentally "repair" perceived distortions based on the "free-energy" principle and actively scan scenes using specific gaze dynamics. EyeSim-VQA is a novel framework designed to mimic these biological processes, achieving state-of-the-art performance by simulating "self-repair" and eye-scanning mechanisms.

Methods

The EyeSim-VQA framework employs a dual-branch architecture to process video content through two distinct lenses:

  1. View Decomposition: The input video is split into an Aesthetic Branch, which uses resized full-frames to capture global composition, and a Technical Branch, which uses grid-based mini-patches to preserve local textures and fine-grained distortions.
  2. Free-Energy-Guided Self-Repair: To simulate the brain's denoising capability, each branch incorporates a visual enhancement module ($E_A$ and $E_T$). The Aesthetic branch utilizes a lightweight super-resolution model (BasicVSR-mini), while the Technical branch uses iterative residual blocks (CleanNet).
  3. The EyeSim-VQA Head: This module performs the final quality regression using two biologically inspired sub-branches:
  4. Gaze Branch: Employs Dynamic Transformers (DyT) to model fixational behavior on salient regions.
  5. Scan Branch: Utilizes a Mamba-based architecture to simulate the horizontal "sweeping" motion of the human eye across a scene.
The EyeSim-VQA framework architecture, featuring dual-branch view decomposition, visual enhancement for self-repair simulation, and the biologically inspired prediction head.

Key Insights

A key innovation is the use of the enhanced video as a "pseudo-reference." By concatenating original features with enhanced ones, the model can quantify the degree of "repair" required, which serves as a direct proxy for the original video's quality. Furthermore, repurposing Mamba's linear sequence modeling for eye-scanning simulation provides a computationally efficient way to capture temporal gaze dynamics without the overhead of large language models.

Results and Performance

EyeSim-VQA was evaluated across five major VQA benchmarks, including KVQ, KoNViD-1k, and YouTube-UGC. As shown in the results table, the model consistently achieves state-of-the-art (SOTA) or highly competitive performance across diverse datasets.

Performance comparison of EyeSim-VQA against classical, deep learning, and CLIP/LLM-based approaches across multiple benchmarks.

On the KoNViD-1k dataset, EyeSim-VQA reached an SROCC of 0.919, outperforming complex CLIP-based models. Its strong performance on YouTube-UGC (0.902 SROCC) and in cross-dataset testing demonstrates robust generalization capabilities. While the current model requires a two-stage training process for enhancement and regression, its success highlights the value of integrating biological priors into automated quality assessment systems.

Summary updated

BibTeX
@article{wang2026eyesim,
  author = {Zhaoyang Wang and Wen Lu and Jie Li and Lihuo He and Maoguo Gong and Xinbo Gao},
  title = {{EyeSim-VQA: A Free-Energy-Guided Eye Simulation Framework for Video Quality Assessment}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2026},
  doi = {10.1109/tcsvt.2026.3651537},
}
Figure