Abstract
Temporal Sentence Localization (TSL) aims to identify specific segments within an untrimmed video that correspond to a linguistic query. While existing methods have achieved significant success, they typically rely on dense annotations—precise start and end timestamps—which are labor-intensive to produce. This paper explores the more practical and challenging "one-shot" TSL setting, where only a single annotated frame per video is available during training. We propose the Multiple Hypotheses Segment Tree (MHST), a novel framework that captures query-aware discriminative information by building a hierarchical tree of video segments. Through iterative merging of semantically similar frames and a robust pruning strategy, MHST generates high-quality segment hypotheses. Experimental results on ActivityNet Captions and Charades-STA demonstrate that MHST achieves competitive performance, even outperforming some fully-supervised baselines.
Introduction
The primary bottleneck in Temporal Sentence Localization is the requirement for exhaustive manual labeling. Fully-supervised models require thousands of hours of precise temporal boundaries, while weakly-supervised models—which only use video-level tags—often lack the precision needed for granular localization.
One-shot TSL offers a middle ground: an annotator provides a single "click" on a relevant frame. This is significantly faster than marking boundaries but provides a stronger spatial-temporal anchor than a simple tag. The challenge lies in expanding this single-point supervision into a full temporal segment. MHST addresses this by treating the video as a collection of potential hypotheses organized in a tree structure.
The Multiple Hypotheses Segment Tree (MHST)
Tree Construction and Pruning
Instead of regressing boundaries directly, MHST adopts a bottom-up approach to segment generation.
- Leaf-Node Initialization: Every video frame is initially treated as a leaf node in the tree.
- Iterative Merging: The model evaluates adjacent nodes based on their visual-linguistic semantics. If two adjacent nodes share high similarity and relate strongly to the text query, they are merged into a parent node. This process continues until root nodes are formed.
- L-Scan Pruning: To manage the complexity of untrimmed videos, the authors introduce a pruning strategy. Nodes with low linguistic relevance to the query are discarded early. This filters out "background" noise and irrelevant actions, ensuring the model focuses on the most promising segment candidates.
Hypothesis Selection
Each root node in the resulting tree represents a distinct segment hypothesis consisting of consecutive frames. During training, the model utilizes the single annotated frame as an anchor: any hypothesis containing this labeled frame is treated as a positive candidate, while others are treated as negatives or background.
Learning and Optimization
To optimize the tree under sparse supervision, MHST employs three specialized self-supervised loss functions:
- Ranking Loss: This ensures that segments matching the query receive higher confidence scores than irrelevant segments.
- Inter-Constraint: This loss compares query-video pairs across different videos in the dataset. It prevents the model from over-fitting to specific visual patterns and encourages it to learn the underlying relationship between language and action.
- Intra-Constraint: By comparing different segments within the same video, this constraint helps the model distinguish the target action from similar-looking background activities or overlapping sub-actions.
Experimental Evaluation
Datasets
The framework was evaluated on two primary benchmarks:
- ActivityNet Captions: 20,000 YouTube videos with 100,000 queries, averaging 2 minutes per video.
- Charades-STA: 16,000 video-sentence pairs focusing on indoor, daily activities.
Performance
MHST demonstrated significant improvements over the previous state-of-the-art for one-shot TSL, ViGA. On ActivityNet Captions, MHST achieved a 64.34% R@1 (at IoU=0.3), compared to ViGA’s 59.61%.
Most notably, on the Charades-STA dataset, MHST (using only one-shot supervision) outperformed several well-established fully-supervised models, such as CTRL and 2D-TAN. This suggests that the hierarchical tree structure is a highly efficient way to capture temporal boundaries, potentially more effective than direct regression when data is noisy or complex.
Discussion and Conclusion
The success of MHST highlights two key insights. First, hierarchy matters: representing video as a tree naturally captures the compositional nature of human actions. Second, one frame provides sufficient "gravity": a single anchor point, when combined with semantic similarity and effective pruning, is enough to pull in relevant frames and define a coherent segment.
While the model is currently dependent on the quality of its pre-trained backbone (C3D) and involves higher computational costs due to the tree-building process, it represents a significant step toward reducing the human effort required for video AI training. Future work may extend this tree-based logic to zero-shot settings, further pushing the boundaries of label-efficient video understanding.
Summary updated