Fast Forwarding Egocentric Videos by Listening and Watching

Vinicius Signori Furlan, Ruzena Bajcsy, Erickson R. NascimentoWorkshops @ CVPR 2018

TL;DR

This paper proposes a novel method for fast-forwarding egocentric videos by integrating psychoacoustic metrics from the audio soundtrack with visual features. By analyzing sound "annoyance" (e.g., roughness, loudness), the system can intelligently skip tedious or unpleasant audio segments, leading to more engaging and efficient video compression.

What Problem Does This Paper Address?

The proliferation of wearable cameras has led to an explosion of egocentric videos, often filled with long, uneventful, or even tedious segments. Traditional methods for compressing these videos, like video summarization and semantic-aware hyperlapse, primarily rely on visual information. While these techniques can highlight important visual moments, they often overlook the rich contextual and emotional information embedded in the audio. This can result in summaries that, while visually concise, might still contain annoying sounds or miss opportunities to leverage audio cues for better content selection. This paper aims to bridge this gap by incorporating audio analysis, specifically psychoacoustic metrics, to create more effective and pleasant fast-forwarded egocentric videos.

Methods

The proposed methodology consists of three main steps:

  1. Data Extraction: The audio soundtrack is extracted from the input video and segmented into short slices (e.g., 3 seconds). Simultaneously, visual features such as instability, appearance (color histogram differences), and velocity are extracted from corresponding video frames.
  2. Psychoacoustic Annoyance Estimation: This is the core novelty. The system computes psychoacoustic metrics from the audio slices, focusing on factors that contribute to perceived annoyance. Specifically, it uses Zwicker's metric, which considers:
  3. Fluctuation and Roughness: Measures of how much sound intensity varies over time, with higher values indicating more unpleasantness.
  4. Loudness and Sharpness: Measures related to perceived volume and the spectral composition of the sound, respectively. Sharper sounds are often more annoying. These metrics are combined to calculate a Psychoacoustic Annoyance (PA) value for each audio segment. A high PA value signifies an annoying segment.

  5. Video Compositing: The PA values are then used to generate a "semantic score" for each frame. This score is combined with the extracted visual features (instability, appearance, velocity) in a linear combination. This combined score guides a video compositing process, similar to semantic-aware hyperlapse. The system uses the Multi-Importance Fast-Forward (MIFF) method, which builds a directed graph of frames and finds the shortest path to create a compressed video. Segments with low PA (pleasant sound) are preserved more, while segments with high PA (annoying sound) are sped up more aggressively. The final output is a fast-forwarded video that aims to be both shorter and more pleasant to watch.

Datasets

The authors evaluated their method using the Dataset of Multimodal Semantic Egocentric Videos (DoM-SEV). This dataset comprises 80 hours of egocentric videos, providing not only video and sound but also IMU measurements, GPS, and depth data. For their experiments, they manually selected 11 diverse segments from this dataset, each featuring a rich variety of sound and visual events.

Key Insights and Ideas

  • Sound as a Primary Annoyance Indicator: The paper highlights that audio can be a significant source of annoyance in egocentric videos, often more so than visual tedium. Ignoring sound means missing a crucial dimension for effective video compression.
  • Psychoacoustic Metrics for Semantic Scoring: Leveraging established psychoacoustic metrics (like roughness and loudness) provides a quantifiable way to measure the "unpleasantness" of sound, which can then be directly translated into a semantic score for video segment selection.
  • Complementary to Visual Methods: The proposed approach doesn't replace visual analysis but complements it. By combining audio-based annoyance with visual features, the system can achieve a more holistic understanding of segment relevance and desirability.
  • Balancing Speed-up and Stability: The method aims to achieve a desired speed-up factor while maintaining a reasonable level of visual stability, acknowledging that aggressive speed-up can sometimes lead to jerky or disorienting results.

Key Results

  • Speed-up: The method achieved a target 10x speed-up factor in most of the 11 tested video segments, often performing better than the baseline MIFF method in this regard.
  • Annoyance Reduction: The fast-forwarded videos produced by their method exhibited a lower average PA value compared to the MIFF method, indicating that annoying sound segments were more effectively reduced or skipped.
  • Qualitative Analysis: Visual results showed that segments with high PA (e.g., crowds, noisy environments) received higher speed-up, while segments with low PA (e.g., quiet moments, pleasant background music) were preserved more, leading to a more subjectively pleasant viewing experience.
  • Instability: While the method generally performed well, it was noted that MIFF performed slightly better in terms of instability. The authors attribute this to the potentially larger number of speed-up thresholds created by their fine-grained semantic information, which might lead to greater frame discrepancies.

Limitations and Future Work

  • Instability Trade-off: The paper acknowledges that their method might introduce slightly more instability compared to purely visual methods like MIFF, especially when aggressive speed-up is applied due to high PA values.
  • Parameter Tuning: The effectiveness of the PA metric is influenced by parameters like 'λ' (used in the shape function f(PA) = λe⁻ᴾᴬ/λ), which was set to 5 in their experiments. Further exploration of optimal parameter settings could be beneficial.
  • Future Work: The authors suggest that combining their audio-based approach with more advanced visual features and exploring different psychoacoustic models could lead to further improvements. They also implicitly suggest that refining the video compositing step to further mitigate instability while maintaining annoyance reduction is a promising direction.

Summary updated

BibTeX
@inproceedings{furlan2018fast,
  author = {Vinicius Signori Furlan and Ruzena Bajcsy and Erickson R. Nascimento},
  title = {{Fast Forwarding Egocentric Videos by Listening and Watching}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
}
Figure