Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection

Taichi Nishimura, Shota Nakada, Hokuto Munakata, Tatsuya KomatsuEMNLP 2024

TL;DR

This paper introduces Lighthouse, a user-friendly and reproducible library for video moment retrieval (MR) and highlight detection (HD). It addresses the community's challenges of fragmented experimental setups and difficult-to-use tools by providing a unified codebase, an inference API, and a web demo, making advanced MR-HD research more accessible.

What Problem Does This Paper Address?

The increasing volume of online video content necessitates efficient ways to find specific moments and identify highlights within videos, often guided by natural language queries. This task, known as Moment Retrieval and Highlight Detection (MR-HD), has seen growing research interest. However, the field faces two significant hurdles:

  1. Lack of Reproducibility: Existing research often uses disparate experimental setups, libraries, and datasets, making it difficult to compare methods fairly or reproduce results. Researchers typically need to build custom environments and code for each new experiment.
  2. User-Unfriendliness: Many MR-HD approaches are released with only training codes, requiring users to implement complex inference pipelines from scratch. This includes video processing, feature extraction, and model deployment, posing a barrier for those without deep expertise in video-text processing.

Lighthouse aims to solve these problems by creating a unified, reproducible, and user-friendly platform for MR-HD research.

Methods

Lighthouse is built around a unified codebase that integrates multiple MR-HD methods, video-text feature extractors, and datasets. Key aspects of its methodology include:

  • Unified Codebase: It supports six recent MR-HD models (e.g., Moment DETR, QD-DETR, TR-DETR) and integrates three types of video-text feature extractors: CLIP, CLIP+Slowfast, and ResNet152+GloVe. This unification allows for comprehensive experimentation across various configurations.
  • YAML Configuration: Experiments are configured using YAML files, simplifying the process of setting up and running training and evaluation for different combinations of models, features, and datasets.
  • Inference API: A user-friendly API is provided, abstracting away the complexities of video processing and feature extraction. Users can initialize a model, encode a video, and then predict moments and highlights with a simple query.
  • Web Demo: A web-based demonstration, built on the inference API, allows users to visualize model outputs interactively, making it easier to understand and evaluate results.
  • Reproducible Artifacts: The library releases pre-processed video-text features, trained weights, and logs to facilitate direct reproduction of experiments.

Datasets

Lighthouse supports five commonly used datasets for MR-HD tasks:

  • QVHighlights: This is the primary MR-HD dataset, containing videos, queries, and annotations for both moments (start/end timestamps) and highlights (saliency scores per frame).
  • ActivityNet Captions: Primarily an MR dataset, providing moment annotations.
  • Charades-STA: Another MR dataset with moment annotations.
  • TaCoS: Also an MR dataset with moment annotations.
  • TVSum: An HD dataset, providing highlight annotations for videos.

The paper notes that original videos are not released due to copyright, but pre-processed features are provided.

Key Insights and Ideas

  • Unified Framework is Crucial: The paper strongly advocates for a unified approach to MR-HD research, highlighting how Lighthouse's integrated codebase significantly reduces the effort required for experimentation and comparison.
  • CLIP+Slowfast Advantage: The results suggest that incorporating sequential motion information (via Slowfast) alongside visual and textual features (via CLIP) generally leads to improved performance in MR-HD tasks compared to using CLIP alone.
  • No One-Size-Fits-All: The experiments reveal that newer MR-HD methods do not consistently outperform older ones across all datasets and feature combinations. This underscores the need for researchers to test multiple methods and features for optimal performance on custom datasets.
  • CLIP's Dominance: CLIP-based features consistently outperform traditional ResNet152+GloVe features, reinforcing CLIP's effectiveness in capturing relevant visual-language representations for MR-HD.

Key Results

Lighthouse successfully reproduces the reported scores for the six MR-HD models in the referenced papers, validating its reproducibility claims.

  • MR-HD Performance: Models like TR-DETR, UVCOM, and CG-DETR, proposed in 2024, show competitive performance on the QVHighlights dataset.
  • Feature Comparison: CLIP+Slowfast generally achieves higher performance than CLIP alone, and CLIP-based features outperform ResNet152+GloVe.
  • MR Results: On datasets like Charades-STA, QD-DETR with CLIP+Slowfast and ResNet152+GloVe showed better performance than CG-DETR and UVCOM, illustrating the variability in method performance.
  • HD Results: Experiments on the TVSum dataset also demonstrate Lighthouse's ability to reproduce reported scores, with Moment DETR showing strong performance when using CLIP features.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Usability Study: A formal usability study to assess how the provided demos assist end-users has not been conducted.
  • Model Diversity: The current implementation primarily focuses on DETR-based models. The authors plan to extend Lighthouse to support other model architectures, such as autoregressive approaches (e.g., BLIP2-style) which have shown promise but require significantly more computational resources.

Future work will focus on addressing these limitations, including incorporating new model types and potentially conducting user studies to refine the library's usability.

Summary updated

BibTeX
@inproceedings{nishimura2024lighthouse,
  author = {Taichi Nishimura and Shota Nakada and Hokuto Munakata and Tatsuya Komatsu},
  title = {{Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure