ClipTBP: Clip-Pair Based Temporal Boundary Prediction with Boundary-Aware Learning for Moment Retrieval

Ji-Hyeon Kim, Ho-Joong Kim, Seong-Whan Lee

TL;DR

ClipTBP is a plug-and-play framework for video moment retrieval that fixes a common flaw in existing models: ignoring the semantic relationships between multiple correct video segments for a single query. By introducing clip-pair similarity learning and boundary-aware losses, it prevents models from getting confused by visually similar background scenes and significantly sharpens temporal boundary predictions.

What Problem Does This Paper Address?

Video moment retrieval (VMR) is the task of pinpointing the exact start and end times of video segments that match a natural language query. Current state-of-the-art models typically evaluate the similarity between the text query and short video snippets independently.

However, this independent approach creates a major blind spot. If a query matches multiple distinct moments throughout a video, or if the background contains scenes that are visually similar but semantically irrelevant to the query, the model struggles. It often merges adjacent segments across a temporal gap or gets tricked by partial matches (for example, confusing a video of a woman walking during a trip with a video of her explicitly describing the trip).

Challenges with multiple answer segments

As shown in Figure 1, baseline models fail to recognize brief temporal gaps between adjacent answer segments, resulting in overly broad boundary predictions.

Methods

To resolve these boundary ambiguities, the authors propose ClipTBP (Clip-Pair based Temporal Boundary Prediction). Instead of altering the core architecture of existing VMR models, ClipTBP introduces three novel training losses that can be seamlessly integrated into both snippet-level regression and DETR-based frameworks:

Overall framework of ClipTBP
  • Clip-level similarity loss ($L_{clip}$): Rather than just matching video snippets to text, this loss uses contrastive learning to model the relationship between different video clips. Clips within the same ground-truth segment are treated as positive pairs, while highly similar clips outside the segment are mined as hard negatives. This forces the model to learn global semantic consistency across the video.
  • Main boundary loss ($L_{boun}$): A margin-based ranking loss that explicitly teaches the model to differentiate between snippets just inside and just outside the temporal boundaries. Crucially, the margin dynamically adjusts based on the length of the target segment.
  • Auxiliary boundary loss ($L_{b\_aux}$): To stabilize the regression training process, this loss provides precise supervision using SmoothL1 loss for four key structural elements: start time, end time, center position, and segment length.

Datasets

The authors evaluated ClipTBP on two widely used video moment retrieval benchmarks:

  • QVHighlights: A dataset containing 4,435 YouTube videos. It is uniquely suited for this task because it features a multi-moment annotation structure, averaging 2.4 answer segments per query.
  • TACoS (Temporal Activity Composition System): A challenging dataset focused on cooking activities. It features long videos (averaging 8 minutes) where the target moments are very short (averaging just 2.5% of the total video length), demanding extreme boundary precision.

Key Insights and Ideas

  • Inter-segment relationships matter: Traditional models treat video snippets as isolated units. ClipTBP demonstrates that enforcing semantic consistency across correct segments helps the model build a more robust global context, making it resilient to contextual "distractors."
  • Dynamic margins for varying lengths: The main boundary loss uses a dynamic margin scaled by the segment's duration. This prevents the model from over-penalizing or under-penalizing boundary errors based on whether a clip is long or short.
  • Embedding space separation: t-SNE visualizations in the paper reveal that ClipTBP successfully untangles the embedding space, cleanly grouping correct answer segments away from visually similar but irrelevant background clips.

Key Results

The authors integrated ClipTBP into four diverse baseline architectures: UniVTG, TD-DETR, Video Mamba Suite, and FlashVTG. It consistently improved performance across the board:

  • QVHighlights: Adding ClipTBP to the FlashVTG baseline boosted the strict average Mean Average Precision (mAP@Avg) from 47.59% to 50.02%. When paired with a stronger backbone (InternVideo2), it reached a state-of-the-art mAP@Avg of 53.47%.
  • TACoS: On long videos, ClipTBP significantly improved the strict [email protected] metric (which requires a high Intersection-over-Union threshold of 0.7) for FlashVTG from 24.74% to 27.49%, proving its effectiveness at capturing precise boundaries.

Limitations and Future Work

Not explicitly discussed in the paper.

Summary updated

Abstract

Video moment retrieval is the task of retrieving specific segments of a video corresponding to a given text query. Recent studies have been conducted to improve multimodal alignment performance through visual-linguistic similarity learning at the snippet-level and transformer-based temporal boundary regression. However, existing models do not calculate similarity by considering the relationships between multiple answer segments that match the query. Therefore, existing models are easily influenced by visually similar segments in the surrounding context. Existing models calculate similarity at the snippet-level and ignore the relationships between multiple answer segments corresponding to a single query. Therefore, they struggle to exclude segments irrelevant to the query. To address this issues, we propose ClipTBP, a clip-pair temporal boundary prediction framework based on boundary-aware learning. ClipTBP introduces a clip-level alignment loss for explicitly learning the semantic relationship between answer segments. ClipTBP also predicts accurate temporal boundaries by applying both main boundary loss and auxiliary boundary loss. ClipTBP consistently improves performance when applied to various existing models and demonstrates more robust boundary prediction performance even in ambiguous query scenarios.
BibTeX
@inproceedings{kim2026cliptbp,
  author = {Ji-Hyeon Kim and Ho-Joong Kim and Seong-Whan Lee},
  title = {{ClipTBP: Clip-Pair Based Temporal Boundary Prediction with Boundary-Aware Learning for Moment Retrieval}},
  booktitle = {ICPR},
  year = {2026},
  doi = {10.1007/978-3-032-31666-0\_16},
}

Loading related content...

Figure