CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer

Ran Ran, Jiwei Wei, Xiangyi Cai, Xiang Guan, Jie Zou, Yang Yang, Heng Tao ShenAAAI 2025 · OpenReview

TL;DR

This paper introduces CDTR, a novel transformer-based model for Video Moment Retrieval (VMR). CDTR tackles challenges like fine-grained semantic understanding and sparse annotations by decomposing video moments and text queries into distinct "concepts." This concept-based approach allows for more robust cross-modal alignment, leading to state-of-the-art performance on benchmark VMR datasets.

What Problem Does This Paper Address?

Video Moment Retrieval (VMR) is the task of finding a specific video segment that matches a natural language query. Imagine asking for "the moment the cat jumps onto the table" in a long home video. Existing VMR methods often struggle with:

  • Limited Fine-Grained Semantics: They might grasp the general idea but miss subtle nuances in the video or query, like distinguishing between "a person walking" and "a person jogging."
  • Semantic Overlap: Similar-sounding or related concepts can confuse the model, leading to incorrect alignments.
  • Sparse Constraints: In many datasets, only a few moments are explicitly labeled for a given query, leaving most of the video as "unlabeled negative samples," which is hard for models to learn from.

These issues hinder the ability of VMR systems to accurately pinpoint precise moments in videos.

Methods

The core innovation of CDTR lies in its Concept Decomposition Transformer. Here's a breakdown:

  • Concept Decomposition Module: This is the heart of CDTR. It takes video clips and text queries and breaks them down into a set of "concept representations." Think of it as identifying the key entities, actions, and attributes present (e.g., "girl," "drives," "car" from a query like "teen girl drives her car"). This module aims to disentangle the rich semantics into more manageable, interpretable units.
  • Pseudo-Labeling: The decomposed concept representations are then used as "pseudo-labels." These labels help the model learn by creating positive and negative examples for specific concepts, even for moments that aren't explicitly annotated in the ground truth.
  • Adaptive Concept-Specific Thresholds: To determine what constitutes a "positive" or "negative" sample for a concept, CDTR uses adaptive thresholds. These thresholds are dynamically adjusted based on the model's current understanding, making the learning process more robust.
  • Fine-Grained Concept Alignment: CDTR performs alignment in two ways:
  • Textual-Visual Cross-Modal Alignment: It aligns the decomposed concepts from the video moments with the concepts derived from the text query.
  • Video Intra-Modal Alignment: It aligns conceptual components within different video moments themselves, helping to build a richer understanding of temporal relationships and semantic consistency within the video.
  • Transformer Architecture: The model leverages transformer mechanisms for cross-attention and self-attention to effectively fuse and process these concept representations, ultimately predicting the start and end timestamps of the target moment.

Datasets

The paper evaluates CDTR on three widely used VMR datasets:

  • QVHighlights: Known for its rich content and manually annotated text queries.
  • Charades-STA: Focuses on daily indoor activities.
  • TACOS: Features long-term videos of cooking activities, offering diverse scenarios.

The authors do not introduce new benchmarks but demonstrate strong performance on existing ones.

Key Insights and Ideas

  • Decomposition is Key: The central idea that breaking down complex video and text semantics into discrete concepts significantly improves alignment is a powerful insight. It moves beyond simple feature matching to a more structured understanding.
  • Pseudo-Labels for Sparse Data: Using decomposed concepts as pseudo-labels is an elegant way to leverage the vast amount of unlabeled data in VMR, directly addressing the sparse constraint problem.
  • Adaptive Thresholds for Robustness: The use of adaptive thresholds for defining positive/negative samples makes the learning process more flexible and less sensitive to noisy labels.
  • Intra-Modal Alignment Matters: The inclusion of video intra-modal alignment suggests that understanding relationships within the video content is as crucial as aligning it with the text query.

Key Results

CDTR consistently outperforms state-of-the-art methods across all three datasets. For example:

  • On QVHighlights, CDTR achieves new state-of-the-art performance, surpassing UVCOM by an average of 2.38% across metrics on the test split.
  • On Charades-STA, it achieves state-of-the-art results using CLIP+SlowFast features, exceeding UVCOM by 0.80% in [email protected].
  • On TACOS, CDTR leads the UVCOM model by 3.87% in [email protected], demonstrating significant improvements.

Qualitative analysis (Figure 3) shows that CDTR effectively focuses on relevant concepts and aligns closer to ground truth than previous methods like MESM.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, transformer-based models, especially those involving decomposition and multiple alignment stages, can be computationally intensive.
  • Concept Granularity: The effectiveness of the concept decomposition module might depend on the chosen vocabulary and the granularity of concepts extracted. Future work could explore more dynamic or learned concept spaces.
  • Generalization to New Domains: While tested on diverse datasets, further exploration of CDTR's generalization capabilities to entirely new video domains or query types would be valuable.

The paper's success suggests that concept-based reasoning is a promising avenue for future VMR research.

Summary updated

Abstract

Video Moment Retrieval (VMR) involves locating specific moments within a video based on natural language queries. However, existing VMR methods that employ various strategies for cross-modal alignment still face challenges such as limited understanding of fine-grained semantics, semantic overlap, and sparse constraints. To address these limitations, we propose a novel Concept Decomposition Transformer (CDTR) model for VMR. CDTR introduces a semantic concept decomposition module that disentangles video moments and sentence queries into concept representations, reflecting the relevance between various concepts and capturing fine-grained semantics which is crucial for cross-modal matching. These decomposed concept representations are then used as pseudo-labels, determined as positive or negative samples by adaptive concept-specific thresholds. Subsequently, fine-grained concept alignment is performed in video intra-modal and textual-visual cross-modal, aligning different conceptual components within features, enhancing the model's ability to distinguish fine-grained semantics, and alleviating issues related to semantic overlap and sparse constraints. Comprehensive experiments demonstrate the effectiveness of the CDTR, outperforming state-of-the-art methods on three widely used datasets: QVHighlights, Charades-STA, and TACoS.
BibTeX
@inproceedings{ran2025cdtr,
  author = {Ran Ran and Jiwei Wei and Xiangyi Cai and Xiang Guan and Jie Zou and Yang Yang and Heng Tao Shen},
  title = {{CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=M5sydfOoo0},
}
Figure