Implicit Analysis of Perceptual Multimedia Experience Based on Physiological Response: A Review

Seong-Eun Moon, Jong-Seok Lee

Abstract

The exponential growth of popularity of multimedia has led to needs for user-centric adaptive applications that manage multimedia content more effectively. Implicit analysis, which examines users' perceptual experience of multimedia by monitoring physiological or behavioral cues, has potential to satisfy such demands. Particularly, physiological signals categorized into cerebral physiological signals (electroencephalography, functional magnetic resonance imaging, and functional near-infrared spectroscopy) and peripheral physiological signals (heart rate, respiration, skin temperature, etc.) have recently received attention along with notable development of wearable physiological sensors. In this paper, we review existing studies on physiological signal analysis exploring perceptual experience of multimedia. Furthermore, we discuss current trends and challenges.

Introduction

Understanding how users perceive and interact with multimedia is crucial for developing adaptive content delivery systems. Traditionally, this "Perceptual Multimedia Experience" (PME) was measured through explicit methods, such as questionnaires or rating scales. However, these methods are often intrusive, subjective, and prone to "recall bias," where users struggle to accurately verbalize their feelings after the experience has ended.

Overview of implicit perceptual experience analysis and its diverse applications.

Implicit analysis offers a solution by monitoring involuntary physiological and behavioral responses. This approach allows for real-time, objective assessment of Quality of Experience (QoE), emotional states, and physical fatigue, enabling applications ranging from content recommendation and filtering to assistive technologies for disabled individuals.

Physiological Signal Modalities

Research in this field typically categorizes physiological signals into two primary groups:

  1. Cerebral Signals (The Brain): These monitor brain activity directly. EEG is the most popular modality due to its high temporal resolution and increasing portability. Features often include Event-Related Potentials (ERPs) and power-band analysis (alpha, beta, gamma). Other methods like fMRI and fNIRS provide superior spatial resolution, identifying specific brain regions involved in emotional or aesthetic processing.
  2. Peripheral Signals (The Body): These capture the body's physical reactions to stimuli. Key signals include GSR (Galvanic Skin Response) for arousal, ECG/Heart Rate and Respiration for stress or relaxation, and EMG for measuring facial muscle tension related to mental stress.

The Analysis Pipeline

The transition from raw physiological data to actionable insights follows a structured computational workflow.

Standard pipeline for implicit analysis of multimedia experience using physiological signals.

As illustrated in the pipeline, raw data is first pre-processed to remove artifacts such as muscle twitches or environmental noise. Relevant features are then extracted—ranging from spectral power in EEG to heart rate variability. Finally, machine learning models (such as Support Vector Machines or Neural Networks) are trained on labeled datasets to classify user states like preference, emotion, QoE, or fatigue.

Key Insights and Results

  • High Accuracy: Subject-dependent models can achieve accuracies as high as 85-90% for basic emotion classification and roughly 85% for detecting video quality artifacts like blur or ghosting.
  • Brain Connectivity: Modern research emphasizes brain connectivity—how different regions synchronize—as a more robust indicator of complex emotional states than single-region activity.
  • Measuring Fatigue: Physiological signals can objectively quantify 3D-induced fatigue, which manifests as increased beta-band brain activity and unstable autonomic states.
  • Biological Basis for Aesthetics: Neuroscientific studies have linked aesthetic judgments to specific activations in the medial orbitofrontal cortex, providing a biological foundation for subjective beauty.

Challenges and Future Directions

Despite its promise, several hurdles remain. Subjectivity is a major challenge, as physiological responses vary significantly between individuals, making "subject-independent" models difficult to build. Additionally, environmental noise in non-laboratory settings can degrade signal quality. Future work is expected to leverage Deep Learning for automated feature extraction and focus on developing more comfortable, non-intrusive wearable sensors to enable widespread, daily adoption.

Summary updated

Abstract

The exponential growth of popularity of multimedia has led to needs for user-centric adaptive applications that manage multimedia content more effectively. Implicit analysis, which examines users' perceptual experience of multimedia by monitoring physiological or behavioral cues, has potential to satisfy such demands. Particularly, physiological signals categorized into cerebral physiological signals (electroencephalography, functional magnetic resonance imaging, and functional near-infrared spectroscopy) and peripheral physiological signals (heart rate, respiration, skin temperature, etc.) have recently received attention along with notable development of wearable physiological sensors. In this paper, we review existing studies on physiological signal analysis exploring perceptual experience of multimedia. Furthermore, we discuss current trends and challenges.
BibTeX
@article{moon2017implicit,
  author = {Seong-Eun Moon and Jong-Seok Lee},
  title = {{Implicit Analysis of Perceptual Multimedia Experience Based on Physiological Response: A Review}},
  journal = {IEEE Trans. Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=MdUoDNGZ8YT},
}
Figure