Hypotheses Tree Building for One-Shot Temporal Sentence Localization

Daizong Liu, Xiang Fang, Pan Zhou, Xing Di, Weining Lu, Yu Cheng

Abstract

Temporal Sentence Localization (TSL) aims to identify specific segments within an untrimmed video that correspond to a linguistic query. While existing methods have achieved significant success, they typically rely on dense annotations—precise start and end timestamps—which are labor-intensive to produce. This paper explores the more practical and challenging "one-shot" TSL setting, where only a single annotated frame per video is available during training. We propose the Multiple Hypotheses Segment Tree (MHST), a novel framework that captures query-aware discriminative information by building a hierarchical tree of video segments. Through iterative merging of semantically similar frames and a robust pruning strategy, MHST generates high-quality segment hypotheses. Experimental results on ActivityNet Captions and Charades-STA demonstrate that MHST achieves competitive performance, even outperforming some fully-supervised baselines.

Introduction

The primary bottleneck in Temporal Sentence Localization is the requirement for exhaustive manual labeling. Fully-supervised models require thousands of hours of precise temporal boundaries, while weakly-supervised models—which only use video-level tags—often lack the precision needed for granular localization.

One-shot TSL offers a middle ground: an annotator provides a single "click" on a relevant frame. This is significantly faster than marking boundaries but provides a stronger spatial-temporal anchor than a simple tag. The challenge lies in expanding this single-point supervision into a full temporal segment. MHST addresses this by treating the video as a collection of potential hypotheses organized in a tree structure.

The Multiple Hypotheses Segment Tree (MHST)

Tree Construction and Pruning

Instead of regressing boundaries directly, MHST adopts a bottom-up approach to segment generation.

  1. Leaf-Node Initialization: Every video frame is initially treated as a leaf node in the tree.
  2. Iterative Merging: The model evaluates adjacent nodes based on their visual-linguistic semantics. If two adjacent nodes share high similarity and relate strongly to the text query, they are merged into a parent node. This process continues until root nodes are formed.
  3. L-Scan Pruning: To manage the complexity of untrimmed videos, the authors introduce a pruning strategy. Nodes with low linguistic relevance to the query are discarded early. This filters out "background" noise and irrelevant actions, ensuring the model focuses on the most promising segment candidates.

Hypothesis Selection

Each root node in the resulting tree represents a distinct segment hypothesis consisting of consecutive frames. During training, the model utilizes the single annotated frame as an anchor: any hypothesis containing this labeled frame is treated as a positive candidate, while others are treated as negatives or background.

Learning and Optimization

To optimize the tree under sparse supervision, MHST employs three specialized self-supervised loss functions:

  • Ranking Loss: This ensures that segments matching the query receive higher confidence scores than irrelevant segments.
  • Inter-Constraint: This loss compares query-video pairs across different videos in the dataset. It prevents the model from over-fitting to specific visual patterns and encourages it to learn the underlying relationship between language and action.
  • Intra-Constraint: By comparing different segments within the same video, this constraint helps the model distinguish the target action from similar-looking background activities or overlapping sub-actions.

Experimental Evaluation

Datasets

The framework was evaluated on two primary benchmarks:

  • ActivityNet Captions: 20,000 YouTube videos with 100,000 queries, averaging 2 minutes per video.
  • Charades-STA: 16,000 video-sentence pairs focusing on indoor, daily activities.

Performance

MHST demonstrated significant improvements over the previous state-of-the-art for one-shot TSL, ViGA. On ActivityNet Captions, MHST achieved a 64.34% R@1 (at IoU=0.3), compared to ViGA’s 59.61%.

Most notably, on the Charades-STA dataset, MHST (using only one-shot supervision) outperformed several well-established fully-supervised models, such as CTRL and 2D-TAN. This suggests that the hierarchical tree structure is a highly efficient way to capture temporal boundaries, potentially more effective than direct regression when data is noisy or complex.

Discussion and Conclusion

The success of MHST highlights two key insights. First, hierarchy matters: representing video as a tree naturally captures the compositional nature of human actions. Second, one frame provides sufficient "gravity": a single anchor point, when combined with semantic similarity and effective pruning, is enough to pull in relevant frames and define a coherent segment.

While the model is currently dependent on the quality of its pre-trained backbone (C3D) and involves higher computational costs due to the tree-building process, it represents a significant step toward reducing the human effort required for video AI training. Future work may extend this tree-based logic to zero-shot settings, further pushing the boundaries of label-efficient video understanding.

Summary updated

Abstract

Given an untrimmed video, temporal sentence localization (TSL) aims to localize a specific segment according to a given sentence query. Though respectable works have made decent achievements in this task, they severely rely on dense video frame annotations, which require a tremendous amount of human effort to collect. In this paper, we target another more practical and challenging setting: one-shot temporal sentence localization (one-shot TSL), which learns to retrieve the query information among the entire video with only one annotated frame. Particularly, we propose an effective and novel tree-structure baseline for one-shot TSL, called Multiple Hypotheses Segment Tree (MHST), to capture the query-aware discriminative frame-wise information under the insufficient annotations. Each video frame is taken as the leaf-node, and the adjacent frames sharing the same visual-linguistic semantics will be merged into the upper non-leaf node for tree building. At last, each root node is an individual segment hypothesis containing the consecutive frames of its leaf-nodes. During the tree construction, we also introduce a pruning strategy to eliminate the interference of query-irrelevant nodes. With our designed self-supervised loss functions, our MHST is able to generate high-quality segment hypotheses for ranking and selection with the query. Experiments on two challenging datasets demonstrate that MHST achieves competitive performance compared to existing methods.
BibTeX
@inproceedings{liu2023hypotheses,
  author = {Daizong Liu and Xiang Fang and Pan Zhou and Xing Di and Weining Lu and Yu Cheng},
  title = {{Hypotheses Tree Building for One-Shot Temporal Sentence Localization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=b5dzLFJDhE},
}
Figure