What's in the Flow? Exploiting Temporal Motion Cues for Unsupervised Generic Event Boundary Detection

Sourabh Vasant Gothe, Vibhav Agarwal, Sourav Ghosh, Jayesh Rajkumar Vachhani, Pranay Kashyap, Barath Raj Kandur RajaWACV 2024

TL;DR

This paper introduces FlowGEBD, a novel non-parametric, unsupervised method for detecting generic event boundaries in videos. By leveraging optical flow, FlowGEBD achieves state-of-the-art performance among unsupervised methods on challenging benchmarks, demonstrating that motion cues alone can be highly effective for this task.

What Problem Does This Paper Address?

Understanding the structure of long videos is crucial for applications like summarization, editing, and efficient browsing. A key aspect of this understanding is identifying "generic event boundaries" – moments where a video naturally breaks into distinct, meaningful events, regardless of specific action categories. Current methods often rely on complex, supervised deep neural networks (DNNs) that require vast amounts of labeled data and significant computational resources. This poses a challenge for real-time applications and domain generalization. The authors aim to address this by exploring two fundamental questions: can simpler, non-parametric algorithms outperform unsupervised DNNs, and is motion information sufficient for high-performance event boundary detection?

Methods

FlowGEBD is a non-parametric, unsupervised approach that primarily utilizes optical flow. It comprises two main algorithms:

  • Pixel Tracking (PT): This method uses sparse optical flow to identify event boundaries. It works by tracking a set of key pixels (identified through random sampling or corner detection) frame-by-frame. A significant drop in the number of successfully tracked pixels between frames signals a potential event boundary. This approach is inherently causal, relying only on past frames.
  • Flow Normalization (FN): This algorithm leverages dense optical flow. It divides frames into a grid of patches and computes the dense optical flow between corresponding patches in consecutive frames. The "PatchFlow" (maximum displacement within a patch) is then normalized over time. Peaks in this normalized PatchFlow are hypothesized to indicate event changes.

Both PT and FN can operate in either a "framewise" mode (treating the entire frame as a unit) or a "patchwise" mode (dividing the frame into smaller patches). The patchwise approach, especially with the addition of "centroidal patches" (which capture events at the intersections of base patches), offers finer granularity and improved performance.

Finally, an ensemble of PT and FN is proposed. Instead of temporal refinement for each individual method, their predicted boundary sets are merged, and then a unified temporal refinement step is applied. This refinement process identifies and consolidates "popular" (dense clusters of boundaries) and "rare" (isolated boundaries) events to produce the final set of refined event boundaries.

Datasets

The primary dataset used for evaluation is Kinetics-GEBD, a large-scale benchmark for generic event boundary detection. The authors also evaluate on the TAPOS dataset, which contains Olympic sports videos. TAPOS was adapted for the GEBD task by treating sub-action instances within actions as individual event segments.

Key Insights and Ideas

  • Motion is Sufficient: The core insight is that optical flow, representing motion, alone is powerful enough to detect generic event boundaries, obviating the need for complex DNNs and explicit semantic understanding.
  • Non-Parametric Advantage: The paper demonstrates that non-parametric, algorithmic approaches can achieve state-of-the-art results, challenging the dominance of parametric DNNs in this domain.
  • Patchwise Granularity: Dividing frames into patches, especially with centroidal patches, significantly improves the ability to capture localized event changes that might be missed by a global frame-level analysis.
  • Ensemble Power: Combining the complementary strengths of sparse (PT) and dense (FN) optical flow methods through ensembling leads to superior performance.
  • Threshold Robustness: The proposed methods exhibit remarkable robustness to variations in their key thresholds (01, 02, 03), indicating good generalization capabilities.

Key Results

  • State-of-the-Art Unsupervised Performance: FlowGEBD (Ensembled) achieves the best [email protected] score of 0.713 on the Kinetics-GEBD validation set, outperforming previous unsupervised methods by a significant margin.
  • Competitive with Supervised Methods: On Kinetics-GEBD, FlowGEBD achieves an 8.8% absolute gain over the supervised baseline PC [30] and a 6.7% absolute gain over PC + Optical Flow [20].
  • Strong TAPOS Performance: The ensembled method achieves an average F1 score of 0.623 on the TAPOS validation dataset, a 8% absolute improvement over the unsupervised baseline PA [30].
  • Efficiency: The PT and FN algorithms are significantly faster than many DNN-based approaches, with inference times of 2.26 ms and 6.42 ms per frame, respectively, on a mobile device.

Limitations and Future Work

  • Lack of Spatial Semantics: The authors acknowledge that FlowGEBD does not incorporate high-level spatial semantic features. This makes it more suitable for generic event boundary detection rather than specific action or event localization.
  • Future Directions: The authors suggest exploring bi-directional processing of frames to further improve performance, potentially capturing temporal dependencies more comprehensively.

Summary updated

BibTeX
@inproceedings{gothe2024what,
  author = {Sourabh Vasant Gothe and Vibhav Agarwal and Sourav Ghosh and Jayesh Rajkumar Vachhani and Pranay Kashyap and Barath Raj Kandur Raja},
  title = {{What's in the Flow? Exploiting Temporal Motion Cues for Unsupervised Generic Event Boundary Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}

Loading related content...

Figure