Detecting Moments and Highlights in Videos via Natural Language Queries

Jie Lei, Tamara Lee Berg, Mohit BansalNeurIPS 2021 · OpenReview

As internet video content grows at an unprecedented rate, the ability to find exactly what you're looking for—and the most exciting parts of it—becomes a major engineering challenge. While we've made great strides in searching for whole videos, locating specific moments or highlights within them based on a natural language query remains a difficult, under-studied problem.

A paper from researchers at UNC Chapel Hill introduces a unified dataset and a novel transformer-based model to tackle this challenge head-on.

The Disconnect in Video Understanding

Existing research in video understanding is often split into two distinct silos:

  1. Moment Retrieval: Finding a specific segment (e.g., "a person cooking pasta") in a long video.
  2. Highlight Detection: Identifying the most "salient" or interesting parts of a video.

Current datasets for these tasks have significant flaws. Many moment retrieval datasets suffer from "temporal bias," where relevant moments are disproportionately located at the start of videos. Furthermore, they typically only annotate a single moment per query, even if multiple relevant segments exist. Conversely, highlight detection datasets are often "query-agnostic"—they identify generic highlights regardless of what the user actually asked for.

The authors argue that these tasks are two sides of the same coin and require a unified, high-quality dataset to drive progress.

Introducing QVHighlights

To bridge this gap, the authors present QVHighlights, a comprehensive dataset consisting of 10,148 YouTube videos (spanning Daily Vlogs, Travel Vlogs, and News) paired with 10,310 human-written queries.

QVHIGHLIGHTS provides multiple disjoint moments and 5-point saliency scores for each query.

What makes QVHighlights stand out from previous benchmarks?

  • Multiple Moments: It allows for multiple disjoint segments to be relevant to a single query.
  • Saliency Scores: It includes a 5-point Likert scale (from "Very Good" to "Very Bad") for every 2-second clip within the relevant moments, providing a much richer signal for highlight detection.
  • Reduced Bias: Moments are distributed almost evenly across the video duration, mitigating the temporal bias found in older datasets.

Moment-DETR: A Unified Transformer Approach

Alongside the dataset, the authors propose Moment-DETR, a model inspired by the DETR (Detection Transformer) architecture used in object detection.

Unlike traditional methods that rely on complex, hand-crafted pipelines—like generating hundreds of candidate "proposals" and then filtering them with Non-Maximum Suppression (NMS)—Moment-DETR is elegantly simple. It views moment retrieval as a direct set prediction problem.

The Moment-DETR architecture uses a transformer encoder-decoder to predict moment spans and saliency scores directly from video and text features.

Key components of the architecture include:

  • Feature Extraction: It uses SlowFast and CLIP (ViT-B/32) to extract visual features every 2 seconds, alongside a CLIP text encoder for the user query.
  • Transformer Encoder-Decoder: The model processes these features to understand global temporal relations.
  • Moment Queries: A set of trainable positional embeddings (moment queries) are fed into the decoder. Each query "learns" to look for moments at different temporal locations and scales.
  • Joint Prediction Heads: The model simultaneously predicts span coordinates (center and width), classification (relevant foreground vs. background), and saliency scores (how "highlight-worthy" the segment is).

Key Insights and Results

The research yielded several compelling insights into video understanding:

  • Joint Learning is Better: Training the model to detect saliency actually improves its ability to retrieve moments. The saliency signal helps the model better align text queries with visual content.
  • The Power of ASR Pretraining: One of the biggest performance boosts came from "weakly-supervised pretraining." The authors used 236,000 caption-timestamp pairs from YouTube's Automatic Speech Recognition (ASR) to pretrain the model before fine-tuning it on the high-quality human annotations.

Moment-DETR sets a strong new baseline for the field. On the new QVHighlights dataset, the pretrained version achieves 59.78% [email protected] (Recall at IoU 0.5) and 36.14% average mAP, significantly outperforming well-engineered architectures. For highlight detection specifically, it achieves 37.43% mAP, proving that query-dependent highlights are a viable and useful task. Furthermore, when tested on the standard Charades-STA dataset, Moment-DETR achieved a new state-of-the-art (SOTA) with 55.65% [email protected], demonstrating excellent generalization.

Limitations and Future Work

While Moment-DETR is powerful, the authors note it is a "data-hungry" model. Because it doesn't use human-designed priors (like fixed-length proposals), it requires substantial training data to learn effectively.

Future work could involve developing more sophisticated pretraining tasks that better align with the goal of highlight detection, exploring how to make the model more efficient for hours-long videos, and addressing potential biases in the collected dataset to ensure the system works fairly across diverse types of content.

Summary updated

Abstract

Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHighlights) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, Moment-DETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr.
BibTeX
@inproceedings{lei2021detecting,
  author = {Jie Lei and Tamara Lee Berg and Mohit Bansal},
  title = {{Detecting Moments and Highlights in Videos via Natural Language Queries}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=tfBBt_q4nHT},
}
Figure