When evaluating the quality of a video or an audio stream, automated systems typically begin by calculating a score for every individual frame or short segment. The core challenge then becomes aggregation: how do we collapse a sequence of thousands of frame-level scores into a single number that accurately reflects human judgment?
The most straightforward approach is to compute the arithmetic mean across all frames. However, a large body of research demonstrates that simple mathematical averaging fails to capture how human memory processes quality over time (Moorthy et al.). Humans do not experience time uniformly; instead, our retrospective evaluation of an experience is heavily distorted by psychological biases. To bridge the gap between algorithmic scores and subjective human ratings, modern quality assessment frameworks must explicitly model the temporal mechanics of human memory.
The Failure of the Mean
To understand why simple averaging fails, consider two video streams. The first video has a continuous, barely noticeable blur throughout its entire duration. The second video is perfectly crisp and high-definition for nine seconds, but suffers a severe, jarring one-second drop in quality due to a network glitch.
Mathematically, these two videos might yield the exact same average score. Perceptually, however, human viewers will rate the second video significantly lower. This is because human perception is disproportionately sensitive to brief periods of significant quality degradation—a phenomenon known as the negative peak effect (Kim et al.; Yuan et al.). These brief glitches act as cognitive bottlenecks, dominating a viewer's memory of the entire experience.
This non-linear sensitivity is not unique to vision. In audio and speech quality assessment, standard arithmetic averaging similarly fails because a single moment of high-intensity distortion or leakage will permanently lower a user's subjective rating, even if the rest of the signal is pristine (Xu et al.; Ivry et al.).
The Dynamics of Perceptual Hysteresis
The primary mechanism used to model this cognitive asymmetry is perceptual hysteresis. In physics, hysteresis describes systems whose state depends on their immediate history. In human perception, hysteresis manifests as an asymmetric response to quality fluctuations: our satisfaction drops instantly when quality degrades, but recovers remarkably slowly when quality improves (Li et al.; Jiang et al.).
Quality Score
^
| /---\
| / \ Technical Quality (Instantaneous)
|----+ +--------
| | |
| | \ Perceived Quality (Hysteresis Lag)
|----+ \-------
+-------------------------> Time
When a video stream undergoes a sudden drop in resolution or a rebuffering stall, human satisfaction plummets immediately. However, once the technical issue is resolved and the high-resolution stream returns, the viewer's perceived quality does not track the technical recovery. Instead, it enters a prolonged recovery phase where satisfaction remains suppressed (Eswara et al.). Viewers retain the memory of poor-quality frames much longer than high-quality ones (Min et al.). Because human judgment is filtered through expectations established by recent history (Fang et al.), a single past negative event continues to penalize the user's Quality of Experience (QoE) long after the technical stream has cleared up (Eswara et al.).
The Recency Effect
A second critical bias governing human memory is the recency effect: the tendency to weight the final stages of an experience more heavily than the beginning or middle (Azevedo et al.).
The timing of a network impairment dictates its perceptual cost. Disruptions occurring near the end of a playback session have an outsized, negative impact on the final user rating compared to identical events occurring earlier (Chen et al.; Ahmad et al.). If a video plays smoothly for several minutes but freezes right before the conclusion, the user's final retrospective score will be severely penalized.
This recency bias is so pronounced that it can distort laboratory subjective testing. Felix Mercer Moss et al. demonstrated that shortening evaluation clips from 10 seconds to 5 seconds actually improves score reliability by keeping the entire stimulus within the active working memory window, thereby mitigating the outsized influence of the final frames. The recency effect also extends to spatial exploration; when browsing immersive 360-degree environments, a user's overall quality impression is disproportionately shaped by the regions of the scene they looked at most recently (Wu et al.; Rao et al.).
Algorithmic Implementations of Memory
To build models that see videos the way humans do, researchers have developed several pooling strategies that move beyond simple averaging. Yilin Wang notes that complex recurrent architectures often struggle to outperform simple temporal averaging unless they are explicitly designed to capture these precise short-term versus long-term memory dynamics.
Handcrafted Hysteresis Pooling
One approach is to replace standard pooling layers with specialized, non-linear mathematical operations that explicitly prioritize lower-quality frames. For instance, models can apply an asymmetric recursive filter to the frame scores:
$$\text{Perceived}_t = \alpha \cdot \text{Instantaneous}_t + (1 - \alpha) \cdot \min(\text{Perceived}_{t-1}, \text{Instantaneous}_t)$$
By forcing the model to track the minimum local quality and recover slowly, this mechanism mimics the human tendency to weigh negative experiences more heavily (Li et al.; Li et al.). Similarly, the popular VMAF framework achieves higher accuracy by integrating temporal features that weight recent quality drops more heavily than stable periods (Bampis et al.).
State-Space and Recurrent Modeling
Instead of static pooling, some frameworks treat quality perception as a continuous, evolving state. Eswara et al. introduced a Nonlinear State Space (NLSS) model where the current perceptual state is formulated as a function of the previous state and the current frame input. This allows the system to mathematically track the gradual, suppressed recovery phase following a rebuffering event. Other models leverage Long Short-Term Memory (LSTM) networks or subjectively-inspired weighting to capture how past resolution drops continue to suppress user satisfaction over time (Eswara et al.; Sun et al.).
Hybrid Attention Mechanisms
While continuous memory accumulation works well for smooth transitions, it can fail during abrupt visual changes like scene cuts, which act as "perceptual resets" that clear a viewer's immediate short-term memory baseline (Fang et al.). To handle this, modern architectures combine global pooling with temporal attention mechanisms (Kossi et al.). The attention mechanism acts as a dynamic weighting function, automatically identifying and highlighting the specific, highly memorable frames that dominate human memory while maintaining overall baseline stability.
Practical Implications
Understanding that human perception is driven by the depth and timing of quality drops—rather than the mathematical average—fundamentally changes how engineering systems are optimized.
In video streaming networks, traditional rate-adaptation algorithms attempt to maximize the overall average bitrate. However, by shifting the optimization goal to satisfy subjective temporal constraints, systems can tolerate minor, uniform quality reductions if they prevent deep, sudden quality drops or late-session stalls. As shown by Chen et al., designing network transmission policies around these human memory thresholds can reduce network resource consumption by up to 40% without any perceived loss in user satisfaction. Ultimately, aligning algorithmic pooling with human memory allows engineers to optimize for the viewer's actual experience rather than a sterile mathematical average.
Key Foundational Work
Efficient motion weighted spatio-temporal video SSIM index (2010) — contributed 2 concept elements to this direction.