Hypotheses Tree Building for One-Shot Temporal Sentence Localization

Daizong Liu, Xiang Fang, Pan Zhou, Xing Di, Weining Lu, Yu Cheng

Abstract

Given an untrimmed video, temporal sentence localization (TSL) aims to localize a specific segment according to a given sentence query. Though respectable works have made decent achievements in this task, they severely rely on dense video frame annotations, which require a tremendous amount of human effort to collect. In this paper, we target another more practical and challenging setting: one-shot temporal sentence localization (one-shot TSL), which learns to retrieve the query information among the entire video with only one annotated frame. Particularly, we propose an effective and novel tree-structure baseline for one-shot TSL, called Multiple Hypotheses Segment Tree (MHST), to capture the query-aware discriminative frame-wise information under the insufficient annotations. Each video frame is taken as the leaf-node, and the adjacent frames sharing the same visual-linguistic semantics will be merged into the upper non-leaf node for tree building. At last, each root node is an individual segment hypothesis containing the consecutive frames of its leaf-nodes. During the tree construction, we also introduce a pruning strategy to eliminate the interference of query-irrelevant nodes. With our designed self-supervised loss functions, our MHST is able to generate high-quality segment hypotheses for ranking and selection with the query. Experiments on two challenging datasets demonstrate that MHST achieves competitive performance compared to existing methods.

Introduction

Temporal Sentence Localization (TSL) is a critical task in computer vision that involves retrieving a specific video segment corresponding to a natural language query. Conventional approaches typically rely on dense annotations, where human annotators specify the start and end timestamps for every action. This process is labor-intensive and limits the scalability of TSL models.

To address this, we explore the One-Shot TSL setting, where only a single relevant frame is annotated per video query. This setting is significantly more practical but introduces challenges in boundary estimation and background noise suppression. We propose the Multiple Hypotheses Segment Tree (MHST), a framework designed to localize segments by hierarchically grouping frames based on their semantic relevance to the query.

Methodology

The MHST framework operates through a structured pipeline that transforms frame-level features into segment-level hypotheses.

The MHST framework architecture, detailing the four stages: Preparation, Tree Building, Pruning & Selection, and Supervision.

1. Preparation and Tree Building

Video frames and the sentence query are first encoded into high-dimensional features. Every frame is initialized as a leaf node in a tree. Adjacent nodes are iteratively merged into parent nodes if they exhibit high visual-linguistic similarity. This merging process is guided by both visual consistency and linguistic relevance to the query, allowing the model to distinguish between similar actions based on textual context.

2. Pruning and Selection

To manage computational complexity and filter out query-irrelevant segments, an L-scan pruning strategy is employed. This step removes nodes with low query-relevance scores, ensuring that only the most promising hypotheses are retained for final selection.

3. Self-Supervised Learning

Given the limited supervision (one frame), the model is trained using three specialized loss functions:

  • Ranking Loss: Uses policy gradients to reward proposals with high linguistic relevance.
  • Inter-constraint Loss: Enhances discriminative power between matching and non-matching video-query pairs.
  • Intra-constraint Loss: A hinge loss that forces the model to rank the "positive" segment (containing the labeled frame) higher than other "negative" segments within the same video.

Experiments and Results

We evaluated MHST on two benchmarks: ActivityNet Captions and Charades-STA. In the one-shot setting, MHST demonstrated superior performance over existing weakly-supervised and one-shot baselines.

Qualitative results on ActivityNet Captions, showing MHST's ability to accurately predict temporal boundaries compared to ground truth.

On ActivityNet Captions, MHST achieved an R@1 (IoU=0.5) of 45.68%, outperforming the previous state-of-the-art one-shot method, ViGA (35.79%), and even rivaling fully-supervised models like DRN (45.45%). On Charades-STA, MHST reached 34.48% (R@1, IoU=0.7), more than doubling ViGA's performance. Ablation studies confirmed that both the query-guided merging and the pruning strategy are essential for these gains.

Conclusion

The Multiple Hypotheses Segment Tree (MHST) provides a robust solution for one-shot temporal sentence localization. By leveraging a hierarchical tree structure and self-supervised constraints, it effectively bridges the gap between weakly-supervised and fully-supervised methods. Future work will focus on optimizing the computational efficiency of the tree-building process and exploring its application in other multi-modal video understanding tasks.

Summary updated

Abstract

Given an untrimmed video, temporal sentence localization (TSL) aims to localize a specific segment according to a given sentence query. Though respectable works have made decent achievements in this task, they severely rely on dense video frame annotations, which require a tremendous amount of human effort to collect. In this paper, we target another more practical and challenging setting: one-shot temporal sentence localization (one-shot TSL), which learns to retrieve the query information among the entire video with only one annotated frame. Particularly, we propose an effective and novel tree-structure baseline for one-shot TSL, called Multiple Hypotheses Segment Tree (MHST), to capture the query-aware discriminative frame-wise information under the insufficient annotations. Each video frame is taken as the leaf-node, and the adjacent frames sharing the same visual-linguistic semantics will be merged into the upper non-leaf node for tree building. At last, each root node is an individual segment hypothesis containing the consecutive frames of its leaf-nodes. During the tree construction, we also introduce a pruning strategy to eliminate the interference of query-irrelevant nodes. With our designed self-supervised loss functions, our MHST is able to generate high-quality segment hypotheses for ranking and selection with the query. Experiments on two challenging datasets demonstrate that MHST achieves competitive performance compared to existing methods.
BibTeX
@inproceedings{liu2023hypotheses,
  author = {Daizong Liu and Xiang Fang and Pan Zhou and Xing Di and Weining Lu and Yu Cheng},
  title = {{Hypotheses Tree Building for One-Shot Temporal Sentence Localization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=b5dzLFJDhE},
}
Figure