Introduction
Temporal Sentence Grounding (TSG) is a critical task in video understanding, requiring a model to localize a specific moment in a long video based on a natural language query. For instance, given the prompt "a person starts to juggle," the system must identify the precise start and end timestamps. Recent advancements have leveraged the Detection Transformer (DETR) architecture, which treats moments as learnable queries. However, standard DETR-based models typically represent moments using a center point and a length $(c, l)$. This paper identifies a fundamental limitation in this approach: "center misalignment."
The Problem: Center Misalignment
In many video sequences, the mathematical center of an action is not visually salient or semantically relevant to the text description. If a model is forced to predict this arbitrary center accurately to localize the boundaries, any slight error in center prediction propagates to the boundary estimates. This dependency makes models fragile, especially in complex videos where the action's "middle" might be a lull or a transition. BAM-DETR aims to break this dependency on the precise center.
Proposed Method: BAM-DETR
To address these challenges, the authors introduce BAM-DETR, which shifts the focus from the center to the boundaries through three primary innovations.
1. Boundary-Oriented Formulation
Instead of the traditional $(c, l)$ representation, BAM-DETR uses a triplet $(p, d_s, d_e)$. Here, $p$ represents an anchor point—any salient point within the target moment—while $d_s$ and $d_e$ are the distances from that anchor to the start and end boundaries, respectively. This asymmetric design allows the model to successfully ground a moment by identifying any relevant part of the action, rather than a specific, often ambiguous, center.
2. Dual-Pathway Decoder
The architecture employs a dual-pathway transformer decoder to refine these predictions iteratively:
- Anchor Pathway: This pathway uses global cross-attention to scan the entire video memory, identifying the most representative anchor point ($p$) for the query.
- Boundary Pathway: Once an anchor is established, this pathway utilizes "boundary-focused attention." This is a specialized form of deformable attention that zooms in on local details specifically around the predicted start and end times to precisely adjust the boundaries ($d_s, d_e$).
3. Quality-Based Scoring
Standard models often rank proposals based on a classification score (how well the segment matches the text). BAM-DETR improves this by predicting the Intersection over Union (IoU) directly. By using IoU as a quality score for ranking, the model ensures that proposals with better temporal alignment are prioritized, even if multiple segments have high semantic similarity to the query.
Experimental Evaluation
BAM-DETR was evaluated on three major benchmarks: QVHighlights, Charades-STA, and TACoS.
- QVHighlights: This dataset includes saliency scores, providing a rich ground for testing localization. BAM-DETR achieved a significant boost in precision, notably a +6.45% improvement in [email protected] over previous state-of-the-art models like QD-DETR.
- TACoS: Known for its long videos and very short, precise moments, TACoS is a high-difficulty benchmark. BAM-DETR's boundary-focused approach proved particularly effective here, demonstrating its ability to handle fine-grained localization.
Key Insights
The success of BAM-DETR highlights that the "center" in temporal grounding is often an artificial construct. By allowing any salient point to act as an anchor, the model becomes more robust to visual variations. Furthermore, the separation of global context (for anchor selection) and local context (for boundary refinement) mimics human-like visual search patterns. In "anti-biased" tests, BAM-DETR maintained its performance while other models' accuracy plummeted, suggesting it learns actual content rather than dataset statistics.
Conclusion
BAM-DETR introduces a paradigm shift in temporal sentence grounding by replacing center-based modeling with a boundary-oriented approach. Through its dual-pathway decoder and quality-based ranking, it achieves superior precision and robustness across diverse video datasets. This philosophy of boundary alignment offers a promising direction for other video-based localization tasks where precise timing is critical.
Summary updated