Human Memory Models for Video Quality Assessment

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 285 papers181 conceptsfrozen 2026-10-04

Traditional video quality metrics often rely on a simple assumption: the total quality of a video is the arithmetic average of the quality of its individual frames. However, human perception does not work like a calculator. We do not perceive video as a sequence of independent snapshots; we experience it as a continuous flow where our memory of the past and expectations for the future fundamentally alter our current judgment.

Researchers have shifted toward temporal aggregation models that mimic the Human Visual System (HVS) by accounting for psychological phenomena like the recency effect, primacy effect, and temporal hysteresis. These models explain why a single half-second glitch can ruin a ten-minute experience and why we are more forgiving of a blurry start than a blurry finish.

The Fallacy of Arithmetic Averaging

If a video has 90% high-quality frames and 10% low-quality frames, a simple average suggests the video is "mostly good." Yet, as Zhengzhong Tu and colleagues note in RAPIQUE, simple pooling fails to capture the impact of localized quality drops. Human observers are disproportionately influenced by brief periods of significant degradation—a phenomenon often called the "negative peak effect."

In Deep Video Quality Assessor, Woojae Kim and his team address this by using a Convolutional Neural Aggregation Network (CNAN). Instead of treating every frame equally, the network learns to assign higher weights to sudden drops in quality, effectively "remembering" the glitches that a simple average would smooth away.

Temporal Hysteresis: The Lingering Ghost of Poor Quality

One of the most critical concepts in modern VQA is temporal hysteresis. This refers to the asymmetric way humans react to quality changes: we react sharply to quality drops but are slow to acknowledge improvements. When a video's resolution suddenly plunges, our dissatisfaction is immediate. When the resolution returns to 4K, we remain annoyed for several seconds, waiting to see if it will drop again.

Bolin Chen and Shiqi Wang modeled this in their FAVOR metric by incorporating facial recognition priors and memory models. They found that a single low-quality frame negatively biases the evaluation of subsequent frames. To capture this, models like HVS-5M use Gated Recurrent Units (GRUs) to simulate this memory effect, ensuring that the "worst parts" of a sequence are weighted more heavily in the final score.

Recency, Primacy, and the Peak-End Rule

The timing of a distortion is often as important as its severity. Research into "in-the-wild" content and VR environments has highlighted two competing biases:

  1. The Primacy Effect: In Light-VQA+, Xiaohong Liu demonstrates that the initial segments of a clip set a "quality anchor." If a video starts poorly, the viewer forms a negative impression that is hard to shake.
  2. The Recency Effect: Conversely, for longer sessions, the end of the experience dominates. Yuming Fang and Kede Ma found in their study of User-Generated VR Videos that the final segments of a viewing session have a disproportionately high influence on the total score.

This mirrors the "Peak-End Rule" from psychology: we judge an experience largely based on how it felt at its peak (the worst glitch) and how it ended. Nagabhushan Eswara and the creators of LSTM-QoE leverage Long Short-Term Memory (LSTM) networks to track these time-varying perceptions, allowing the model to integrate moment-to-moment frustrations into a single retrospective judgment.

Modeling Expectations and Saturation

Human judgment is also filtered through expectations. In Video Quality Assessment Based on Short-Term Memory, researchers argue that our satisfaction depends on how current quality deviates from the baseline established in the previous few minutes. If a stream has been perfect for a while, a small glitch feels like a major betrayal.

Interestingly, this memory effect has a limit. The influence of short-term memory on quality perception appears to reach a saturation point after approximately 45 seconds. Beyond this, a viewer’s expectation of quality stabilizes, suggesting that while we have a "long" memory for glitches, our psychological "window" for active comparison is finite.

Why It Works: Mimicking the Biological Observer

By moving from static frame analysis to recurrent, memory-aware architectures, these models bridge the gap between signal processing and psychology. They work because they acknowledge that video quality is not a property of the pixels alone, but a property of the interaction between those pixels and the human brain's temporal processing constraints.

Whether it is Assessor360 modeling the transitions between viewports in VR or SatQA predicting satisfaction in satellite streaming, the core intuition remains the same: to understand how a human rates a video, you must understand how a human remembers it.

Papers in this version

Showing 60 of 85 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 50 papers
  2. Version 1 · 2026-10-04 · 85 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_perceptual_hysteresis_3455767b,
  title = {perceptual hysteresis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perceptual-hysteresis/rdr_3455767b582b442bb67415b74d844397},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_human_memory_models_for_video_quality_as_3455767b_v1,
  title = {Human Memory Models for Video Quality Assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/human-memory-models-for-video-quality-assessment/rdr_3455767b582b442bb67415b74d844397/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}