TL;DR
Traditional video grounding models are forced to find a clip even if the text query doesn't exist in the video, leading to hallucinated results. This paper introduces TSG-RF, a new task requiring models to provide relevance feedback. To solve it, the authors propose RaTSG, a lightweight, end-to-end network that accurately identifies if a query is present and localizes it only when appropriate.
The Problem: Hallucinating Non-Existent Segments
Temporal Sentence Grounding (TSG) is the task of retrieving a specific segment from a long video based on a natural language query (e.g., "a person opens a door"). However, current TSG research operates under an idealistic assumption: that the target segment always exists in the video.
In the real world, this isn't true. If you ask an AI to find a "man drinking water" in a video of someone just "opening a door," a standard TSG model will still confidently predict a start and end time, essentially hallucinating a result. This lack of a "none of the above" option makes existing models unreliable for practical applications like robotics or video search.
To tackle this, the authors formalize a new task: Temporal Sentence Grounding with Relevance Feedback (TSG-RF). This task requires the model to first determine if the query is relevant to the video and only output temporal boundaries if it is.
Methods: The RaTSG Network
To address the TSG-RF task, the authors propose the Relation-aware Temporal Sentence Grounding (RaTSG) network. Instead of a clunky two-stage process (first classify, then ground), RaTSG is a unified, end-to-end model that leverages multi-task learning.
Key components include:
- Multi-Granularity Relevance Discriminator: This module evaluates relevance at two levels. At the fine-grained (frame-level), it uses Multiple Instance Learning (MIL) to treat the video as a "bag" of frames; if at least one frame matches the query, the video is considered relevant. At the coarse-grained (video-level), it compares the global semantics of the entire video against the full query text.
- Relation-aware Segment Grounding: This module selectively conducts the grounding. To handle cases where the segment is absent, the authors use a clever trick: they prepend a special token (index 0) to the video feature sequence. If the model predicts index 0 for the start and end boundaries, it signifies that no relevant segment exists.
- Multi-Task Learning: The model is trained with a joint loss function that balances boundary prediction, frame-level relevance, and video-level relevance.
New Benchmarks: Charades-RF & ActivityNet-RF
Since standard datasets assume the query is always present, the authors built their own benchmarks by reconstructing the popular Charades-STA and ActivityNet Captions datasets. For every original video-query pair (positive), they added a "negative" sample where the query is paired with a different, semantically unrelated video. This creates a 1:1 ratio of relevant vs. irrelevant samples, forcing the model to actually learn discrimination.
Key Insights
- Relevance as Foreground Detection: The authors reformulate relevance as a foreground-background problem. If a model can't find "foreground" frames that match the text, it shouldn't try to ground a segment.
- The Power of the "Zero Index": By using a special token to represent "not found," the model can use the same architecture for both classification and localization, making it more efficient than two-stage pipelines.
- Mutual Enhancement: Learning to ground segments actually helps the model become better at relevance discrimination, and vice versa. Shared knowledge across these tasks leads to a more robust representation.
Key Results
The authors compared RaTSG against several state-of-the-art TSG models (like VSLNet and EAMAT) adapted with an external relevance discriminator (denoted as "Model++").
- Superior Accuracy: On the Charades-RF dataset, RaTSG achieved a relevance accuracy of 76.85%, outperforming the best baseline (71.94%).
- Better Grounding: In terms of mIoU (mean Intersection over Union), RaTSG reached 59.93% on Charades-RF, while the best adapted baseline (EAMAT++) reached 56.23%.
- Lightweight Performance: RaTSG achieved these results with just 1.27M parameters, whereas EAMAT++ requires a staggering 98.3M parameters.
- Robustness: RaTSG showed much higher performance gains as the proportion of "negative" (irrelevant) samples in the test set increased, proving it is far more reliable in unpredictable, real-world scenarios.
Limitations and Future Work
While RaTSG is a major step forward, the model currently relies heavily on visual features. It struggles with actions primarily identified by sound (e.g., "sneezing") and sometimes confuses the chronological order of actions (e.g., mistaking "opening a laptop" for "closing a laptop"). Future work will likely involve integrating audio features and more sophisticated temporal modeling to distinguish between similar actions that differ only in their sequence.
Summary updated