Sim-DETR: Unlock DETR for Temporal Sentence Grounding

Jiajin Tang, Zhengxuan Wei, Yuchen Zhu, Cheng Shi, Guanbin Li, Liang Lin, Sibei Yang

Sim-DETR: Unlock DETR for Temporal Sentence Grounding

Introduction: The Scaling Paradox in TSG

Temporal Sentence Grounding (TSG) requires a model to identify the exact start and end timestamps of an event described in a text query within an untrimmed video. Although the Detection Transformer (DETR) architecture revolutionized object detection, its application to TSG has been hindered by a strange phenomenon: increasing the number of queries or decoder layers—typically a recipe for better performance in deep learning—often results in lower accuracy.

The authors identify two primary causes for this "scaling paradox":

  1. Inter-query Conflicts: In the standard one-to-one matching system of DETR, multiple queries representing similar time segments compete. This leads to unstable learning as the model may pick different queries for the same event across different layers.
  2. Intra-query Conflicts: A single query must simultaneously capture global context for semantic understanding and local frame details for precise boundary localization. These competing objectives often pull the model in opposite directions.

Proposed Method: Sim-DETR

Sim-DETR introduces two "minor yet key" modifications to the standard DETR decoder to resolve these conflicts: Query Grouping and Ranking (QGR) and Global-Local Bridging (GLB).

Overview of Sim-DETR architecture featuring Query Grouping and Ranking (QGR) and Global-Local Bridging (GLB).

Query Grouping and Ranking (QGR)

To mitigate inter-query conflicts, QGR implements a "soft grouping" mechanism based on the L2 distance of predicted time spans. By grouping overlapping queries and ranking them using a combination of classification confidence and a dedicated IoU head (which predicts how well the guess overlaps with the ground truth), Sim-DETR ensures that high-quality, consistent queries dominate the prediction process across all decoder layers.

Global-Local Bridging (GLB)

To address intra-query conflicts, GLB introduces a query-to-frame matching loss. Instead of focusing solely on boundary points, this loss encourages the query to align semantically with every frame within the target segment. This "bridge" ensures the model maintains a strong semantic grasp of the entire clip while refining its temporal boundaries.

Experimental Results

Sim-DETR was evaluated on three major benchmarks: QVHighlights, Charades-STA, and TACoS, using various backbones like CLIP, SlowFast, and I3D. The results demonstrate that Sim-DETR effectively solves the scaling paradox.

Performance comparison showing Sim-DETR's stability and improvement as query and layer numbers increase, solving the scaling paradox.

As shown in the figure above, while baseline models like CG-DETR and TaskWeave suffer performance drops when increasing the number of queries or layers, Sim-DETR remains stable or improves. Key performance highlights include:

  • QVHighlights: Achieved a state-of-the-art 46.93 mAP, outperforming previous leaders like BAM-DETR (45.36) and CG-DETR (42.86).
  • TACoS: Showed a significant +3.89% improvement in mIoU, highlighting its strength in long, fine-grained video sequences.
  • Efficiency: The model exhibits faster training convergence compared to vanilla DETR-based grounding models.

Conclusion and Future Work

Sim-DETR provides a powerful yet simple solution to the scaling issues of DETR in temporal grounding. By managing query interactions and bridging global-local semantic gaps, it sets a new standard for the field. While it provides a strong baseline, future work may address extremely subtle semantic differences or extend these grouping principles to other video-related tasks such as action detection and video summarization.

Summary updated

Abstract

Temporal sentence grounding aims to identify exact moments in a video that correspond to a given textual query, typically addressed with detection transformer (DETR) solutions. However, we find that typical strategies designed to enhance DETR do not improve, and may even degrade, its performance in this task. We systematically analyze and identify the root causes of this abnormal behavior: (1) conflicts between queries from similar target moments and (2) internal query conflicts due to the tension between global semantics and local localization. Building on these insights, we propose a simple yet powerful baseline, Sim-DETR, which extends the standard DETR with two minor modifications in the decoder layers: (1) constraining self-attention between queries based on their semantic and positional overlap and (2) adding query-to-frame alignment to bridge the global and local contexts. Experiments demonstrate that Sim-DETR unlocks the full potential of DETR for temporal sentence grounding, offering a strong baseline for future research.
BibTeX
@inproceedings{tang2025sim,
  author = {Jiajin Tang and Zhengxuan Wei and Yuchen Zhu and Cheng Shi and Guanbin Li and Liang Lin and Sibei Yang},
  title = {{Sim-DETR: Unlock DETR for Temporal Sentence Grounding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure