Natural Language Video Moment Localization Through Query-Controlled Temporal Convolution

Lingyu Zhang, Richard J. RadkeWACV 2022

TL;DR

This paper introduces PEARL, a novel framework for pinpointing specific video segments described by natural language queries. PEARL innovates by treating natural language queries as "filters" that are convolved across video and subtitle streams, effectively learning to detect temporal "edges" that signify the start and end of a target moment. This approach achieves state-of-the-art results on challenging video moment localization benchmarks.

What Problem Does This Paper Address?

The task of natural language video moment localization aims to automatically find a short segment within a long, untrimmed video that precisely matches a given text description. These descriptions can be complex, involving subjects, objects, sequential actions, and specific locations. Existing methods often struggle with:

  • False positives: Identifying "boundary-like" frames that don't actually mark the start or end of the described event, especially when similar actions appear elsewhere.
  • Limited consistency metrics: Relying on handcrafted metrics (like cosine similarity) that may not capture nuanced relationships between video content and queries.
  • Multimodal fusion challenges: Effectively integrating visual and subtitle information, particularly when their temporal dependencies are asynchronous.

This paper addresses these limitations by proposing a more robust and flexible approach to temporal localization.

Methods

PEARL introduces a novel "query-controlled temporal convolution" approach. The core idea is to transform the natural language query into a dynamic "filter" that acts as a temporal edge detector.

  • Query-Controlled Edge Detector: Instead of fixed filters (like Sobel or Laplacian), PEARL learns a query-specific filter. This filter is constructed by a "Perception and Abstraction Module" that processes the query embedding. This filter is designed to detect abrupt changes in consistency between the video content and the query.
  • Convolutional Response: The learned query filter is then convolved across the video (and subtitle, if available) sequences. This process generates "response maps" that highlight potential start and end points. The convolution operation, with its defined kernel size and stride, allows for capturing local temporal sequentiality and continuity.
  • Multimodal Fusion (with Subtitles): For videos with subtitles, a "Localization Module" fuses the response maps from visual and subtitle branches using an LSTM. This allows for modeling asynchronous dependencies between the two modalities to generate a final heatmap for boundary detection.
  • Boundary-Oriented Scheme: Similar to some prior work, PEARL focuses on directly estimating start and end boundaries rather than generating numerous proposals, which can be computationally expensive and prone to redundancy.

Datasets

The paper evaluates PEARL on three benchmark datasets:

  • TVR: A challenging dataset with 21,793 videos, 109,000 queries, and a significant portion of queries requiring understanding of multiple actions and complex relationships. It includes both video and subtitle information.
  • TACOS: A dataset of 17,344 query-moment pairs focused on indoor cooking activities. It primarily relies on visual information.
  • Charades-STA: A benchmark with 16,128 query-moment pairs for daily indoor activities. It also primarily relies on visual information.

Key Insights and Ideas

  • Queries as Dynamic Filters: The most significant insight is treating natural language queries not just as semantic descriptors but as active components that shape the detection mechanism itself. By converting queries into learnable filters, the model can adapt its detection strategy to the specific nuances of each query.
  • Temporal "Edges" for Localization: The analogy of temporal boundaries as "edges" in image processing is powerful. This framing allows for leveraging convolutional techniques to detect abrupt changes in query-video consistency, which is a more robust signal than simple frame-wise similarity.
  • Learnable Consistency: Moving beyond handcrafted metrics for consistency, PEARL's convolutional approach inherently learns a more flexible and powerful measure of how well a video segment aligns with a query.
  • Focus on Local Continuity: The convolutional approach naturally emphasizes local temporal continuity, which is crucial for accurately identifying the start and end of events, especially when dealing with complex or sequential actions.

Key Results

PEARL demonstrates state-of-the-art performance across all three evaluated datasets:

  • TVR Dataset: PEARL achieves the highest scores on multiple metrics, including a ranking score of 34.49 at IoU=0.5 and 15.53 at IoU=0.7, surpassing existing methods like XML and ExCL.
  • TACOS Dataset: PEARL achieves a high ranking score of 42.94 at IoU=0.3 and 32.07 at IoU=0.5, outperforming methods like VSLNet.
  • Charades-STA Dataset: PEARL also shows superior performance, with a ranking score of 71.9 at IoU=0.3 and 53.5 at IoU=0.5, exceeding VSLNet and ExCL.

Qualitative results (Figure 5) showcase PEARL's ability to handle challenging cases, including misleading frames, simultaneous actions, and queries relying solely on subtitles or a combination of visual and subtitle information.

Limitations and Future Work

  • Loss Function: The authors acknowledge that the current cross-entropy loss assigns equal penalties to all false predictions, regardless of their proximity to the ground truth. They suggest that a future direction could involve incorporating a regression component to better penalize predictions further away from the true boundary.
  • Charades-STA Performance: While PEARL performs well, its advantage is less pronounced on Charades-STA. This is attributed to the dataset's nature, where most queries involve single actions, making the sequential and complex relationship modeling less critical.
  • Future Work: The paper suggests exploring regression-based loss functions to refine boundary prediction accuracy.

Summary updated

BibTeX
@inproceedings{zhang2022natural,
  author = {Lingyu Zhang and Richard J. Radke},
  title = {{Natural Language Video Moment Localization Through Query-Controlled Temporal Convolution}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure