Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding

Jin-Seop Lee, SungJoon Lee, SeongJun Jung, Boyang Li, Jee-Hyong LeearXiv 2025 · DOI · arXiv

Introduction

Video Temporal Grounding (VTG) is a critical task in computer vision that involves locating a specific segment within a video based on a natural language query. While current state-of-the-art models perform well when a relevant segment exists, they suffer from a significant "hallucination" problem. Most models operate under a "closed-world assumption," assuming that every query must have a corresponding timestamp in the video. Consequently, when presented with an irrelevant query, these models confidently predict a start and end time for an event that never occurred.

Comparison between existing VTG models that hallucinate answers and the proposed refusal-aware approach.

This paper distinguishes between two types of irrelevance: Coarse Irrelevance, where the query and video are semantically distant (e.g., a query about a "bicycle" for a video of a "cat"), and Hard Irrelevance. Hard-irrelevant queries are semantically similar to the video but factually incorrect—for example, asking for a segment where a chef is "cutting steaks" in a video where they are actually "cooking pasta." To address this, the authors introduce Refusal-Aware Reinforcement Fine-Tuning (RA-RFT) and the HI-VTG dataset to teach models to recognize these mismatches and provide detailed refusal explanations.

Methodology: Refusal-Aware Reinforcement Fine-Tuning (RA-RFT)

The core of the proposed solution is RA-RFT, a post-training strategy utilizing the Group Relative Policy Optimization (GRPO) framework. GRPO is an efficient alternative to traditional reinforcement learning (like PPO) because it optimizes the policy by comparing a group of candidate outputs against each other, eliminating the need for a complex "critic" model.

The RA-RFT framework, illustrating the reward functions, the HI-VTG dataset pipeline, and an inference example.

The model is trained using four specialized reward functions designed to enforce reasoning and accuracy:

  1. Format Reward: Ensures the model adheres to a structured output format including <think> (reasoning), <answer> (segment or refusal), and <correct> (query reconstruction).
  2. Refuse-IoU Reward: Penalizes predicting segments for irrelevant queries while rewarding high Intersection-over-Union (IoU) for relevant ones.
  3. Explain Reward: Uses SentenceBERT and LLM-based scoring to reward refusal explanations that accurately describe the semantic mismatch.
  4. Query Correction Reward: The most novel component, rewarding the model for "correcting" an irrelevant query back to the actual video content. For instance, if the query is "cutting steaks" but the video shows "cooking pasta," the model must reconstruct the latter to prove its understanding.

The HI-VTG Dataset

To support this training, the researchers developed the Hard-Irrelevant VTG (HI-VTG) dataset. They utilized 2.5K videos from existing datasets (HowTo100M, InternVID) and employed an LLM to generate 7.5K hard-irrelevant queries. These queries were created by modifying original relevant queries across 11 semantic categories, such as:

  • Action Sequence: Changing the order of events.
  • Object Existence: Adding or removing items from the description.
  • Attribute Value: Changing colors, sizes, or other descriptors.
  • Counting: Altering the number of people or objects present.

To ensure robustness, the authors also created "HI" versions of standard benchmarks (HI-ActivityNet, HI-TVGBench, HI-Charades) and a human-annotated RA-VTG set to validate that the model's performance wasn't merely an artifact of LLM-generated data.

Key Insights and Results

The study demonstrates that correction serves as a powerful form of reasoning. By requiring the model to reconstruct the correct query, the researchers ensure the model isn't just guessing "irrelevant" but actually understands the fine-grained temporal and semantic context. This "Chain-of-Thought" (CoT) approach, facilitated by the <think> block, allows the model to process context before committing to a decision.

Key performance metrics include:

  • Refusal Accuracy: On the HI-ActivityNet benchmark, the RA-RFT version of the Time-R1 model improved its F1 score from 70.5 to 76.3.
  • Explanation Quality: LLM-based consistency scores for refusal explanations increased by 20-30% across various model backbones.
  • Zero-Shot Generalization: Models trained on HI-VTG performed strongly on unseen human-annotated datasets, indicating that the refusal behavior is a robustly learned capability.

Conclusion

RA-RFT effectively breaks the "closed-world assumption" in VTG models without sacrificing performance on relevant queries. By teaching models "what they don't know" and requiring them to explain and correct mismatches, this work paves the way for more reliable and safer interactive video agents. Future work may involve scaling these methods to even larger vision-language models and exploring more complex multi-modal reasoning tasks.

Summary updated

BibTeX
@misc{lee2025learning,
  author = {Jin-Seop Lee and SungJoon Lee and SeongJun Jung and Boyang Li and Jee-Hyong Lee},
  title = {{Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding}},
  year = {2025},
  eprint = {2511.23151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.23151},
}
Figure