Temporal Sentence Grounding with Relevance Feedback in Videos

Jianfeng Dong, Xiaoman Peng, Daizong Liu, Xiaoye Qu, Xun Yang, Cuizhu Bao, Meng WangNeurIPS 2024 · OpenReview

TL;DR

Traditional video grounding models are forced to find a clip even if the text query doesn't exist in the video, leading to hallucinated results. This paper introduces TSG-RF, a new task requiring models to provide relevance feedback. To solve it, the authors propose RaTSG, a lightweight, end-to-end network that accurately identifies if a query is present and localizes it only when appropriate.

The Problem: Hallucinating Non-Existent Segments

Temporal Sentence Grounding (TSG) is the task of retrieving a specific segment from a long video based on a natural language query (e.g., "a person opens a door"). However, current TSG research operates under an idealistic assumption: that the target segment always exists in the video.

In the real world, this isn't true. If you ask an AI to find a "man drinking water" in a video of someone just "opening a door," a standard TSG model will still confidently predict a start and end time, essentially hallucinating a result. This lack of a "none of the above" option makes existing models unreliable for practical applications like robotics or video search.

Illustration of the TSG-RF task providing relevance feedback for absent queries.

To tackle this, the authors formalize a new task: Temporal Sentence Grounding with Relevance Feedback (TSG-RF). This task requires the model to first determine if the query is relevant to the video and only output temporal boundaries if it is.

Methods: The RaTSG Network

To address the TSG-RF task, the authors propose the Relation-aware Temporal Sentence Grounding (RaTSG) network. Instead of a clunky two-stage process (first classify, then ground), RaTSG is a unified, end-to-end model that leverages multi-task learning.

Architecture of the Relation-aware Temporal Sentence Grounding (RaTSG) network.

Key components include:

  • Multi-Granularity Relevance Discriminator: This module evaluates relevance at two levels. At the fine-grained (frame-level), it uses Multiple Instance Learning (MIL) to treat the video as a "bag" of frames; if at least one frame matches the query, the video is considered relevant. At the coarse-grained (video-level), it compares the global semantics of the entire video against the full query text.
  • Relation-aware Segment Grounding: This module selectively conducts the grounding. To handle cases where the segment is absent, the authors use a clever trick: they prepend a special token (index 0) to the video feature sequence. If the model predicts index 0 for the start and end boundaries, it signifies that no relevant segment exists.
  • Multi-Task Learning: The model is trained with a joint loss function that balances boundary prediction, frame-level relevance, and video-level relevance.

New Benchmarks: Charades-RF & ActivityNet-RF

Since standard datasets assume the query is always present, the authors built their own benchmarks by reconstructing the popular Charades-STA and ActivityNet Captions datasets. For every original video-query pair (positive), they added a "negative" sample where the query is paired with a different, semantically unrelated video. This creates a 1:1 ratio of relevant vs. irrelevant samples, forcing the model to actually learn discrimination.

Key Insights

  • Relevance as Foreground Detection: The authors reformulate relevance as a foreground-background problem. If a model can't find "foreground" frames that match the text, it shouldn't try to ground a segment.
  • The Power of the "Zero Index": By using a special token to represent "not found," the model can use the same architecture for both classification and localization, making it more efficient than two-stage pipelines.
  • Mutual Enhancement: Learning to ground segments actually helps the model become better at relevance discrimination, and vice versa. Shared knowledge across these tasks leads to a more robust representation.

Key Results

The authors compared RaTSG against several state-of-the-art TSG models (like VSLNet and EAMAT) adapted with an external relevance discriminator (denoted as "Model++").

  • Superior Accuracy: On the Charades-RF dataset, RaTSG achieved a relevance accuracy of 76.85%, outperforming the best baseline (71.94%).
  • Better Grounding: In terms of mIoU (mean Intersection over Union), RaTSG reached 59.93% on Charades-RF, while the best adapted baseline (EAMAT++) reached 56.23%.
  • Lightweight Performance: RaTSG achieved these results with just 1.27M parameters, whereas EAMAT++ requires a staggering 98.3M parameters.
  • Robustness: RaTSG showed much higher performance gains as the proportion of "negative" (irrelevant) samples in the test set increased, proving it is far more reliable in unpredictable, real-world scenarios.

Limitations and Future Work

While RaTSG is a major step forward, the model currently relies heavily on visual features. It struggles with actions primarily identified by sound (e.g., "sneezing") and sometimes confuses the chronological order of actions (e.g., mistaking "opening a laptop" for "closing a laptop"). Future work will likely involve integrating audio features and more sophisticated temporal modeling to distinguish between similar actions that differ only in their sequence.

Summary updated

Abstract

As a widely explored multi-modal task, Temporal Sentence Grounding in videos (TSG) endeavors to retrieve a specific video segment matched with a given query text from a video. The traditional paradigm for TSG generally assumes that relevant segments always exist within a given video. However, this assumption is restrictive and unrealistic in real-world applications where the existence of a query-related segment is uncertain, easily resulting in erroneous grounding. Motivated by the research gap and practical application, this paper introduces a new task, named Temporal Sentence Grounding with Relevance Feedback (TSG-RF) in videos, which accommodates the possibility that a video may or may not include a segment related to the query. This task entails localizing precise video segments that semantically align with the query text when such content is present, while delivering definitive feedback on the non-existence of related segments when absent. Moreover, we propose a novel Relation-aware Temporal Sentence Grounding (RaTSG) network for addressing this challenging task. This network first reformulates the TSG-RF task as a foreground-background detection problem by investigating whether the query-related semantics exist in both frame and video levels. Then, a multi-granularity relevance discriminator is exploited to produce precise video-query relevance feedback and a relation-aware segment grounding module is employed to selectively conduct the grounding process, dynamically adapting to the presence or absence of query-related segments in videos. To validate our RaTSG network, we reconstruct two popular TSG datasets, establishing a rigorous benchmark for TSG-RF. Experimental results demonstrate the effectiveness of our proposed RaTSG for the TSG-RF task. Our source code is available at https://github.com/HuiGuanLab/RaTSG.
BibTeX
@inproceedings{dong2024temporal,
  author = {Jianfeng Dong and Xiaoman Peng and Daizong Liu and Xiaoye Qu and Xun Yang and Cuizhu Bao and Meng Wang},
  title = {{Temporal Sentence Grounding with Relevance Feedback in Videos}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=DKimhAdJDn},
}
Figure