Implicit Analysis of Perceptual Multimedia Experience Based on Physiological Response: A Review

Seong-Eun Moon, Jong-Seok Lee

TL;DR

This review paper explores the shift from traditional surveys to "implicit" analysis—using physiological signals like brainwaves and heart rates—to measure how users truly experience multimedia. By bypassing self-reporting, researchers can gain real-time, objective insights into Quality of Experience (QoE), emotional states, and even physical fatigue, paving the way for hyper-personalized and adaptive content delivery.

What Problem Does This Paper Address?

Traditionally, understanding a user's "Perceptual Multimedia Experience" (PME) relied on explicit methods: asking users to fill out questionnaires or rate content. While accurate, this approach has major drawbacks: it’s offline, interrupts the experience, and is prone to "recall bias," where users struggle to verbalize or accurately remember their feelings.

The authors tackle the challenge of implicit analysis, which seeks to monitor natural, involuntary responses (physiological and behavioral cues) to multimedia stimuli. The goal is to create systems that can "read" a user's state—such as whether they are bored, frustrated by low video quality, or experiencing 3D-induced eye strain—without the user having to say a word.

Methods

The paper categorizes the physiological signals used for implicit analysis into two main groups:

  1. Cerebral Signals (The Brain):
    • EEG (Electroencephalography): The most popular due to high temporal resolution and increasing portability. Features include Event-Related Potentials (ERPs like the P300 peak) and power-based features (alpha, beta, gamma bands).
  2. fMRI & fNIRS: Used for high spatial resolution to see exactly which brain regions (like the amygdala for emotion) activate, though fMRI is bulky and expensive.
  3. Peripheral Signals (The Body):
    • GSR (Galvanic Skin Response): Measures sweat gland activity, a key indicator of arousal.
  4. ECG/Heart Rate & Respiration: Indicators of stress, relaxation, or negative emotions.
  5. EOG & EMG: Measuring eye blinks and facial muscle tension.

The Analysis Pipeline: The standard workflow involves signal acquisition via sensors, pre-processing to remove noise (like muscle twitches), feature extraction, and finally, feeding these features into Machine Learning models (like SVMs or Neural Networks) to classify the user's state.

Datasets

While the paper is a review and doesn't introduce a new dataset, it provides a vital summary of publicly available benchmarks used in the field (Table IV):

  • DEAP: A massive dataset for emotion analysis using music videos (EEG + peripheral signals).
  • MAHNOB-HCI: Multi-modal data (eye gaze, EEG, etc.) for emotion and implicit tagging.
  • SEED: Focused on EEG responses to emotional film clips.
  • DECAF: Uses MEG (Magnetoencephalography) for movie-induced emotion.
  • Yonsei-tHDRv: A specialized dataset for QoE in High Dynamic Range (HDR) videos.

Key Insights and Ideas

  • Cerebral vs. Peripheral: Brain signals are generally more "informative" because they capture the cognitive process of perception. Peripheral signals, however, are often easier to capture and reflect the "final output" or physical reaction to that perception.
  • Connectivity is Key: Newer research is moving beyond looking at single signal "power" to looking at brain connectivity—how different regions of the brain synchronize. This has proven more effective for distinguishing complex emotional states.
  • The Inverted U-Curve of Preference: Human preference isn't linear; we tend to dislike stimuli that are either too simple or too complex, preferring a "sweet spot" in the middle.
  • 3D Fatigue is Measurable: The paper highlights that 3D-induced fatigue (caused by the conflict between where the eye focuses and where the image appears) shows up clearly in increased beta-band brain activity and unstable heart rates.

Key Results

  • Emotion Recognition: Subject-dependent models (trained on a specific person) can reach accuracies as high as 85-90% for classifying basic emotions like joy or anger using EEG.
  • Quality Detection: EEG can detect specific video artifacts (like blur or "ghosting") with roughly 85% accuracy, showing that our brains register quality drops even if we don't consciously report them.
  • Aesthetics: Neuroscientific studies confirmed that "beautiful" images consistently activate the medial orbitofrontal cortex, providing a biological basis for aesthetic judgment.

Limitations and Future Work

  • The "Subjectivity" Barrier: A major hurdle is individual difference. What one person finds exciting, another finds stressful. Creating "subject-independent" models that work for everyone remains a significant challenge.
  • Environmental Noise: Most studies happen in controlled labs. In the "wild" (e.g., watching a video on a bumpy bus), motion artifacts and environmental noise can drown out physiological signals.
  • Hardware Comfort: While wearable EEG (like "Ear-EEG") is improving, the need for conductive gels and wires still makes many sensors too intrusive for daily use.
  • Deep Learning Potential: The authors point to Deep Learning (CNNs, DBNs) as the future for automatic feature extraction, but note that we need much larger, more diverse open databases to train these models effectively.

Summary updated

Abstract

The exponential growth of popularity of multimedia has led to needs for user-centric adaptive applications that manage multimedia content more effectively. Implicit analysis, which examines users' perceptual experience of multimedia by monitoring physiological or behavioral cues, has potential to satisfy such demands. Particularly, physiological signals categorized into cerebral physiological signals (electroencephalography, functional magnetic resonance imaging, and functional near-infrared spectro-scopy) and peripheral physiological signals (heart rate, respiration, skin temperature, etc.) have recently received attention along with notable development of wearable physiological sensors. In this paper, we review existing studies on physiological signal analysis exploring perceptual experience of multimedia. Furthermore, we discuss current trends and challenges.
BibTeX
@article{moon2017implicit,
  author = {Seong-Eun Moon and Jong-Seok Lee},
  title = {{Implicit Analysis of Perceptual Multimedia Experience Based on Physiological Response: A Review}},
  journal = {IEEE Trans. Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=MdUoDNGZ8YT},
}

Loading related content...

Figure