Sim-DETR: Unlock DETR for Temporal Sentence Grounding
Introduction: The Scaling Paradox in TSG
Temporal Sentence Grounding (TSG) requires a model to identify the exact start and end timestamps of an event described in a text query within an untrimmed video. Although the Detection Transformer (DETR) architecture revolutionized object detection, its application to TSG has been hindered by a strange phenomenon: increasing the number of queries or decoder layers—typically a recipe for better performance in deep learning—often results in lower accuracy.
The authors identify two primary causes for this "scaling paradox":
- Inter-query Conflicts: In the standard one-to-one matching system of DETR, multiple queries representing similar time segments compete. This leads to unstable learning as the model may pick different queries for the same event across different layers.
- Intra-query Conflicts: A single query must simultaneously capture global context for semantic understanding and local frame details for precise boundary localization. These competing objectives often pull the model in opposite directions.
Proposed Method: Sim-DETR
Sim-DETR introduces two "minor yet key" modifications to the standard DETR decoder to resolve these conflicts: Query Grouping and Ranking (QGR) and Global-Local Bridging (GLB).
Query Grouping and Ranking (QGR)
To mitigate inter-query conflicts, QGR implements a "soft grouping" mechanism based on the L2 distance of predicted time spans. By grouping overlapping queries and ranking them using a combination of classification confidence and a dedicated IoU head (which predicts how well the guess overlaps with the ground truth), Sim-DETR ensures that high-quality, consistent queries dominate the prediction process across all decoder layers.
Global-Local Bridging (GLB)
To address intra-query conflicts, GLB introduces a query-to-frame matching loss. Instead of focusing solely on boundary points, this loss encourages the query to align semantically with every frame within the target segment. This "bridge" ensures the model maintains a strong semantic grasp of the entire clip while refining its temporal boundaries.
Experimental Results
Sim-DETR was evaluated on three major benchmarks: QVHighlights, Charades-STA, and TACoS, using various backbones like CLIP, SlowFast, and I3D. The results demonstrate that Sim-DETR effectively solves the scaling paradox.
As shown in the figure above, while baseline models like CG-DETR and TaskWeave suffer performance drops when increasing the number of queries or layers, Sim-DETR remains stable or improves. Key performance highlights include:
- QVHighlights: Achieved a state-of-the-art 46.93 mAP, outperforming previous leaders like BAM-DETR (45.36) and CG-DETR (42.86).
- TACoS: Showed a significant +3.89% improvement in mIoU, highlighting its strength in long, fine-grained video sequences.
- Efficiency: The model exhibits faster training convergence compared to vanilla DETR-based grounding models.
Conclusion and Future Work
Sim-DETR provides a powerful yet simple solution to the scaling issues of DETR in temporal grounding. By managing query interactions and bridging global-local semantic gaps, it sets a new standard for the field. While it provides a strong baseline, future work may address extremely subtle semantic differences or extend these grouping principles to other video-related tasks such as action detection and video summarization.
Summary updated