Intelligent Video Highlights Generation with Front-Camera Emotion Sensing

Hugo Meyer, Peter Wei, Xiaofan Jiang2021 · DOI

TL;DR

This paper introduces HOMER, a novel cloud-based system for automatically generating video highlights by fusing internal video content with the user's emotional responses captured via a front-facing camera. This approach significantly improves highlight detection accuracy by incorporating subjective user experience, outperforming existing methods and enabling personalized highlight generation without requiring external hardware.

What Problem Does This Paper Address?

The proliferation of digital storage and social media has led to an explosion of video content. Manually creating video highlights or montages is time-consuming and requires specialized editing skills, limiting its accessibility. While existing video summarization techniques exist, they often rely solely on internal video features (audio/visual) and fail to capture the subjective, user-centric nature of what constitutes a "highlight." This leads to generic summaries that may not resonate with individual viewers. Furthermore, many affective computing approaches for video analysis require obtrusive external hardware, hindering practical application. This paper addresses the need for an automated, personalized, and accessible video highlight generation system that leverages user emotions as a key indicator of subjective importance.

Methods

HOMER employs a hybrid approach, combining internal video analysis with external user emotion sensing. The core components include:

  • Multi-modal Feature Extraction:
    • Emotion Sensing: Utilizes a front-facing camera to capture the user's facial expressions. Face detection (Haar-cascade) is performed, followed by emotion recognition using a pre-trained deep learning model (achieving 69% accuracy on AffectNet). The system focuses on positive emotions (happiness and surprise) and extracts "peak" features from these emotion signals.
  • Video Analysis: * Frame Similarity: Computes frame similarity between consecutive frames to detect dynamic scene changes (e.g., camera pans, scene transitions), using a method by Zhang et al. This helps in identifying significant shifts in visual content.
  • Audio Analysis: Detects and segments sound events using an unsupervised bottom-up approach involving wavelet transform and adaptive thresholding. These events are then classified into three categories: speech, laughter, and miscellaneous, with speech and laughter being considered particularly relevant for highlights.
  • Events Timeline (ET): A novel matrix representation that synchronizes extracted features from emotion, frame similarity, and audio over time. Each feature is projected onto the ET as a score or class, creating a unified representation for highlight detection.
  • Highlight Extraction: Two approaches are implemented:
  • Machine Learning: A supervised approach using two independent classifiers for highlight start and end times. This method, however, showed poor performance due to the subjective nature of highlight labeling.
  • Hand-Designed Algorithm: This approach, which proved more effective, computes "start" and "end" score functions based on a linear combination of the fused features. These functions are designed to capture the interplay between emotional responses, sound events, and visual scene changes, with specific penalties to control highlight duration.

Datasets

  • Microsoft's Video Titles in the Wild (VTW) dataset: Used for evaluating the system's performance on home videos.
  • Dual Camera Setup Videos: Videos filmed using a dual camera setup (front and back) on an HTC One M8 smartphone were used in experiments.

Key Insights and Ideas

  • Emotion as a Primary Highlight Indicator: The paper strongly advocates for the use of user emotions as a crucial, subjective signal for identifying highlights, arguing that affective responses are often more indicative of personal relevance than purely objective content analysis.
  • Hybrid Approach Superiority: Combining internal video features with external emotion sensing significantly outperforms methods relying on a single modality or only internal video content.
  • Events Timeline for Synchronization: The ET is a key innovation for coherently fusing and synchronizing disparate multi-modal features, enabling more robust highlight detection.
  • Hand-Designed Algorithm's Robustness: Despite the rise of deep learning, a carefully hand-designed algorithm proved more effective for this specific task, likely due to the inherent subjectivity and complexity of defining "ground truth" for highlights.
  • No External Hardware Needed: The system's ability to leverage a smartphone's front-facing camera for emotion sensing makes it highly accessible and practical for widespread adoption.

Key Results

  • HOMER demonstrated an improvement of up to 38% in F₁-score compared to baseline methods on the VTW dataset.
  • The system achieved better performance on "funny" and "exciting" videos (higher arousal) compared to "other" videos (lower arousal, higher valence), suggesting that emotionally charged content is more reliably captured.
  • User satisfaction ratings indicated that a majority of users were satisfied or partially satisfied with the generated highlights (75.14% combined).
  • The hand-designed algorithm outperformed machine learning approaches for highlight detection in their experiments.

Limitations and Future Work

  • Subjectivity of Highlights: The authors acknowledge that highlight labeling is inherently subjective, which posed a challenge for supervised machine learning approaches.
  • Machine Learning Performance: The supervised ML classifiers showed poor performance, suggesting a need for more training data or improved feature engineering for ML-based approaches.
  • Future Work:
    • Further exploration of machine learning methods for improved performance, especially with more training data.
  • Developing more sophisticated emotion recognition models.
  • Expanding applications to areas like live streaming (Twitch) and video conferencing, where real-time highlight generation could be highly beneficial.
  • Investigating pre-processing of sensitive data on the mobile device to enhance privacy.

Summary updated

BibTeX
@misc{meyer2021intelligent,
  author = {Hugo Meyer and Peter Wei and Xiaofan Jiang},
  title = {{Intelligent Video Highlights Generation with Front-Camera Emotion Sensing}},
  year = {2021},
  doi = {10.3390/s21041035},
}
Figure