TL;DR
This paper proposes a novel method for automatically composing coherent video stories from a collection of video clips. By leveraging a two-stream Recurrent Neural Network (RNN) to learn semantic and motion coherence between clips, and then employing a submodular ranking optimization to align with storyline structures, the approach generates more engaging and smoothly transitioning video narratives than existing methods.
What Problem Does This Paper Address?
In today's world, people capture vast amounts of personal experiences through videos. However, manually editing these raw clips into a concise and meaningful "story" is time-consuming and labor-intensive. While video summarization aims to select keyframes, this paper tackles the more complex problem of "video-story composition," which involves not only selecting but also ordering and smoothly transitioning video clips to form a coherent narrative that follows a typical storyline structure (exposition, rising action, climax, resolution). Existing methods often struggle with ambiguous scene transitions, incoherent results from clips taken at different times, and reliance on hand-crafted features.
Methods
The proposed method consists of two main stages:
- Learning Video Coherence via RNNs:
- Two-Stream RNN: A recurrent neural network (RNN) is employed to model the temporal coherence between video clips. This RNN has two streams:
- Semantic Stream: Utilizes C3D features, a 4096-dimensional representation extracted from a pre-trained 3D convolutional neural network, to capture spatial-temporal semantics.
- Motion Stream: Employs SPP-HOOF (Spatial Pyramid Pooling - Histograms of Oriented Optical Flow) features to capture motion dynamics within clips.
- Unsupervised Learning: The RNN is trained in an unsupervised manner using the natural temporal order of clips within real videos. The objective is to predict the next clip in a sequence given the preceding ones, maximizing the log-likelihood.
- Coherence Score Fusion: The probabilities from the semantic and motion streams are jointly fused to generate a coherence score between clips.
- Submodular Ranking for Storyline Alignment:
- Graph Construction: The ordered clips from the RNN are treated as nodes in a fully connected graph.
- Submodular Objective Function: A submodular objective function is designed to ensure two criteria:
- High Coherence: Measured by a facility location (FL) term that considers the pairwise coherence between clips (using the learned coherence scores).
- Rising Activity Dynamics: Captured by an activity dynamics (AD) term that encourages increasing motion intensity throughout the story.
- Greedy Optimization: The submodular ranking problem is solved using a greedy algorithm, starting with the clip having the lowest dynamics score as the initial clip and iteratively adding clips that maximize the objective function. This process rearranges the clips to better fit a storyline structure.
Datasets
- Video Composition Dataset [6]: Used for training and evaluation. This dataset consists of 23 video sets, each containing 8-12 video clips (2-3 seconds each), totaling 236 clips. The content is diverse, covering activities like sightseeing, skateboarding, walking, and shopping, in various scenes.
- SumMe [9, 10] and TV-Sum [4] datasets: Used for training the models, covering activities like holidays, events, and sports.
Key Insights and Ideas
- Two-Stream RNN for Holistic Coherence: Combining semantic and motion features in a two-stream RNN allows for a more robust understanding of clip relationships, leading to smoother transitions than using either feature alone. The semantic RNN, in particular, emphasizes holistic scene coherence rather than just local feature matching.
- Unsupervised Learning for Practicality: Training the RNN in an unsupervised manner bypasses the subjective and difficult labeling of ground truths for video composition, making the approach more flexible and scalable.
- Submodular Ranking for Narrative Structure: Explicitly optimizing for both clip coherence and rising activity dynamics through a submodular function effectively aligns the composed video with a desired storyline structure, improving overall narrative flow.
- Handling Ambiguity: The learned coherence is shown to be robust, even when an unrelated clip is present, as the accumulated information in the RNN helps maintain story consistency.
Key Results
- User Study: A user study on Amazon Mechanical Turk showed that the proposed method (two-stream RNN + submodular ranking) achieved a higher average pairwise preference score (1.14) compared to the baseline (two-stream RNN, 0.94) and the state-of-the-art Plot Analysis method (0.91).
- Bradley-Terry Scores: The proposed algorithm achieved a Bradley-Terry score of 1.22, outperforming the baseline (0.88) and Plot Analysis (0.85).
- ROC Curves: The proposed method achieved higher ROC curves and AUC scores compared to various baseline methods, including semantic RNN, motion RNN, two-stream RNN, and Plot Analysis, indicating superior component-wise coherence.
- Qualitative Examples: Visual examples demonstrate that the proposed method produces video stories with more consistent scenes and activities, and better rising dynamics compared to baselines and the Plot Analysis method.
Limitations and Future Work
- Failure Cases: The paper acknowledges that the method shows less effectiveness in cases with uneventful or irrelevant scenes and activities (e.g., "walk and museum"), where clip contents might not significantly impact the overall story.
- Future Work: While not explicitly detailed, the success in improving coherence and storyline alignment suggests potential for further exploration in more complex narrative structures and diverse video content. The authors also mention making their MATLAB codes available, which could facilitate future research.
Summary updated