As internet video content grows at an unprecedented rate, the ability to find exactly what you're looking for—and the most exciting parts of it—becomes a major engineering challenge. While we've made great strides in searching for whole videos, locating specific moments or highlights within them based on a natural language query remains a difficult, under-studied problem.
A paper from researchers at UNC Chapel Hill introduces a unified dataset and a novel transformer-based model to tackle this challenge head-on.
The Disconnect in Video Understanding
Existing research in video understanding is often split into two distinct silos:
- Moment Retrieval: Finding a specific segment (e.g., "a person cooking pasta") in a long video.
- Highlight Detection: Identifying the most "salient" or interesting parts of a video.
Current datasets for these tasks have significant flaws. Many moment retrieval datasets suffer from "temporal bias," where relevant moments are disproportionately located at the start of videos. Furthermore, they typically only annotate a single moment per query, even if multiple relevant segments exist. Conversely, highlight detection datasets are often "query-agnostic"—they identify generic highlights regardless of what the user actually asked for.
The authors argue that these tasks are two sides of the same coin and require a unified, high-quality dataset to drive progress.
Introducing QVHighlights
To bridge this gap, the authors present QVHighlights, a comprehensive dataset consisting of 10,148 YouTube videos (spanning Daily Vlogs, Travel Vlogs, and News) paired with 10,310 human-written queries.
What makes QVHighlights stand out from previous benchmarks?
- Multiple Moments: It allows for multiple disjoint segments to be relevant to a single query.
- Saliency Scores: It includes a 5-point Likert scale (from "Very Good" to "Very Bad") for every 2-second clip within the relevant moments, providing a much richer signal for highlight detection.
- Reduced Bias: Moments are distributed almost evenly across the video duration, mitigating the temporal bias found in older datasets.
Moment-DETR: A Unified Transformer Approach
Alongside the dataset, the authors propose Moment-DETR, a model inspired by the DETR (Detection Transformer) architecture used in object detection.
Unlike traditional methods that rely on complex, hand-crafted pipelines—like generating hundreds of candidate "proposals" and then filtering them with Non-Maximum Suppression (NMS)—Moment-DETR is elegantly simple. It views moment retrieval as a direct set prediction problem.
Key components of the architecture include:
- Feature Extraction: It uses SlowFast and CLIP (ViT-B/32) to extract visual features every 2 seconds, alongside a CLIP text encoder for the user query.
- Transformer Encoder-Decoder: The model processes these features to understand global temporal relations.
- Moment Queries: A set of trainable positional embeddings (moment queries) are fed into the decoder. Each query "learns" to look for moments at different temporal locations and scales.
- Joint Prediction Heads: The model simultaneously predicts span coordinates (center and width), classification (relevant foreground vs. background), and saliency scores (how "highlight-worthy" the segment is).
Key Insights and Results
The research yielded several compelling insights into video understanding:
- Joint Learning is Better: Training the model to detect saliency actually improves its ability to retrieve moments. The saliency signal helps the model better align text queries with visual content.
- The Power of ASR Pretraining: One of the biggest performance boosts came from "weakly-supervised pretraining." The authors used 236,000 caption-timestamp pairs from YouTube's Automatic Speech Recognition (ASR) to pretrain the model before fine-tuning it on the high-quality human annotations.
Moment-DETR sets a strong new baseline for the field. On the new QVHighlights dataset, the pretrained version achieves 59.78% [email protected] (Recall at IoU 0.5) and 36.14% average mAP, significantly outperforming well-engineered architectures. For highlight detection specifically, it achieves 37.43% mAP, proving that query-dependent highlights are a viable and useful task. Furthermore, when tested on the standard Charades-STA dataset, Moment-DETR achieved a new state-of-the-art (SOTA) with 55.65% [email protected], demonstrating excellent generalization.
Limitations and Future Work
While Moment-DETR is powerful, the authors note it is a "data-hungry" model. Because it doesn't use human-designed priors (like fixed-length proposals), it requires substantial training data to learn effectively.
Future work could involve developing more sophisticated pretraining tasks that better align with the goal of highlight detection, exploring how to make the model more efficient for hours-long videos, and addressing potential biases in the collected dataset to ensure the system works fairly across diverse types of content.
Summary updated