Personalizing Fast-Forward Videos Based on Visual and Textual Features from Social Network

Washington Ramos, Michel Silva, Edson Araujo, Alan Neves, Erickson NascimentoWACV 2020

TL;DR

This paper introduces a novel method for creating personalized fast-forward videos (hyperlapses) from first-person videos (FPVs) by leveraging users' social media activity. By analyzing text from social networks, the system infers user interests and uses this to intelligently select frames, resulting in hyperlapses that highlight content relevant to the individual viewer.

What Problem Does This Paper Address?

First-person videos (FPVs), often captured by wearable cameras, generate vast amounts of footage. While these videos can be meaningful to the recorder and their followers, their length makes them unlikely to be watched. Existing video summarization techniques often create disjointed clips, and traditional hyperlapse methods accelerate the entire video, making important moments imperceptible. Semantic fast-forwarding methods aim to address this by emphasizing relevant segments, but they often rely on predefined object sets that don't account for individual user preferences. This paper tackles the challenge of creating personalized hyperlapses that are both temporally coherent and visually engaging, catering to the unique interests of each viewer.

Methods

The core of the proposed method lies in its two-step approach: Frame Scoring and Hyperlapse Composition.

  1. Frame Scoring:
    • Representation Space: A shared representation space is built where both video frames and user preferences are encoded. This space is dimensioned by "topics of interest," derived from clustering word embeddings of visual concepts. The word embeddings are trained on a combination of the Visual Genome dataset and a large corpus of tweets and general text to capture visual similarity and social network vocabulary.
  2. User Interests (xu): User interests are inferred from their social network posts (specifically Twitter, but extensible to other platforms). Nouns from positive posts are extracted as concepts, and these concepts are used to form a "Bag of Topics" (BoT) representation for the user.
  3. Frame Topics (xf): For each frame, concepts are extracted using the DenseCap algorithm. These concepts are then weighted based on:
  4. Attention: Viewer attention, derived from temporal and motion cues.
  5. Confidence: The score assigned by DenseCap, indicating the reliability of the detected concept.
  6. Uniqueness: A log-normalized TF-IDF score, highlighting concepts that are important within the specific frame's context relative to the entire video. These weighted concepts form the frame's BoT representation.

  7. Interestingness Score (Iuf): The cosine similarity between the user's interest vector (xu) and the frame's topic vector (xf) is calculated to determine how interesting that frame is to the specific user.

  8. Hyperlapse Composition:
    • Segmentation: The video is segmented based on the per-frame interestingness scores, with a semantic threshold classifying segments as "relevant" or "non-relevant" to the user.
  9. Speed-up Rate Optimization: Different speed-up rates are assigned to relevant (lower speed-up) and non-relevant (higher speed-up) segments to emphasize important content while maintaining overall speed. An optimization problem is solved to balance speed-up rates, temporal continuity, and visual smoothness.
  10. Speed-up Refinement: A process is used to reduce abrupt speed-up changes between segments, iteratively refining the speed-up rates and semantic threshold.
  11. Transition Cost Minimization: A graph-based approach is used to select frames within each segment, minimizing a weighted sum of costs related to relevance drop, instability, motion speed, and appearance dissimilarity between consecutive frames.
  12. Stabilization: The final hyperlapse video is post-processed with a 2D stabilization technique.

Datasets

  • Training:
    • Visual Genome dataset (for initial word embedding training).
  • A corpus of 198 million tweets and 6.7 billion words from general data (Wikipedia, Google News) (for initializing word embeddings).
  • Evaluation:
    • UT Egocentric (UTE) Dataset: 4 first-person videos with human-annotated concepts.
  • Semantic Dataset: 11 first-person videos with activities like biking, driving, and walking.
  • EgoSequences: 9 first-person videos depicting indoor and outdoor activities.
  • Simulated Users: Generated using a character-based LSTM network trained on tweets related to specific concepts (CAR, CHAIR, COMPUTER, PEOPLE, TREE).
  • Real Twitter Users: Collected from users who explicitly indicated topics of interest in their bios.
  • GTEA Gaze+ dataset: Used to create placebo videos for user studies.

Key Insights and Ideas

  • Leveraging Social Text for Personalization: The most significant insight is the effective use of readily available social media text (tweets) to infer nuanced user interests, going beyond simple object detection. This allows for a truly personalized hyperlapse experience.
  • Generalizing Concepts: The method's ability to generalize concepts (e.g., "birds" to "nature") through the learned representation space is crucial for matching user interests to visual content, even when explicit keywords aren't present in a frame.
  • Balancing Multiple Constraints: The hyperlapse composition effectively balances semantic relevance, visual smoothness, temporal continuity, and desired speed-up rates, which is a complex but necessary trade-off for creating watchable videos.
  • Uniqueness Score's Importance: The "Uniqueness" weight in frame scoring is highlighted as important for capturing singular, potentially highly relevant moments within a video, even if the concepts are common.

Key Results

  • Personalization Performance: The proposed method achieved an average F₁ score up to 12.8 percentage points higher than the best competitors (MIFF) across various datasets and simulated users.
  • Speed-up Accuracy: The method demonstrated significantly better speed-up rate accuracy, with an average deviation of 0.2 compared to 0.9 for MSH and 1.4 for MIFF.
  • Visual Quality: User studies indicated that the proposed method did not compromise perceived stability, with a higher percentage of non-negative answers for visual quality compared to MIFF.
  • User Study Validation: In a user study, evaluators correctly identified the emphasized concepts in personalized videos, with the method performing particularly well when personalizing for "People" (73.8% selection).

Limitations and Future Work

  • Concept Cluster Overlap: The method might struggle if semantically related concepts are in distinct clusters, potentially causing under-emphasis of relevant video segments.
  • Low Saliency/TF-IDF Issues: Objects with low visual saliency or low TF-IDF (due to recurrence) might be assigned lower scores, even if they match user interests.
  • Future Work:
    • Explore multi-modal approaches by incorporating other media types (images, videos, sound) shared by users on social networks.
  • Investigate modeling user behavior and refining semantics further.

Summary updated

BibTeX
@inproceedings{ramos2020personalizing,
  author = {Washington Ramos and Michel Silva and Edson Araujo and Alan Neves and Erickson Nascimento},
  title = {{Personalizing Fast-Forward Videos Based on Visual and Textual Features from Social Network}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure