Abstract
Temporal grounding aims to localize specific actions in untrimmed videos using natural language queries. Traditional models often follow a unidirectional path from text to video, lacking a mechanism to verify the semantic consistency of the predicted interval. We present EVOQUER, a closed-loop framework that incorporates a "back-query generation" feedback loop. By simplifying queries into core visual pivots (verbs and nouns) and training the model to reconstruct these from predicted clips, EVOQUER achieves superior precision and semantic alignment. Experiments on Charades-STA and ActivityNet Captions demonstrate consistent improvements over state-of-the-art baselines, particularly at high Intersection over Union (IoU) thresholds.
Introduction
Temporal grounding is the task of identifying the start and end timestamps of an action described by a natural language query within an untrimmed video. While current models have made significant strides, they typically operate as "one-way streets," mapping text to video without a robust verification step. This lack of feedback limits the model's ability to correct internal misconceptions about actions and objects.
To address this, we introduce EVOQUER (Enhancing Temporal Grounding with VideO-Pivoted Back QUERy Generation). Inspired by "feedback-error-learning" from control theory, our system "double-checks" its predictions by attempting to describe the video clip it has localized.
Methods
The EVOQUER framework consists of two primary components integrated into a closed loop: a Temporal Grounding Module and a Translation Module.
- Temporal Grounding Module: We utilize the Local-Global Video-Text Interactions (LGI) model as our base grounding engine, which processes the full video and the input query to predict a temporal interval.
- Translation Module (Feedback Loop): Once an interval is predicted, frames are extracted and passed to a video-to-text translation module based on the VMT framework.
A key innovation is Query Simplification. Reconstructing full natural language sentences is noisy and difficult. Instead, we use Part-of-Speech (POS) tagging and lemmatization to reduce queries to essential "visual pivots"—primary verbs and nouns (e.g., "closing the door" becomes "close door"). The model is trained with a joint loss: the standard grounding loss plus a Negative Log-Likelihood (NLL) loss from the query generation task. This forces the grounding module to select intervals that are semantically rich enough to allow the translation module to accurately "re-generate" the core concepts.
Results
We evaluated EVOQUER on two benchmark datasets: Charades-STA (short videos, ~30s) and ActivityNet Captions (longer, complex videos, ~120s). EVOQUER consistently outperformed the baseline LGI model across both datasets.
On Charades-STA, EVOQUER improved the [email protected] metric by 1.05% and achieved a mean IoU (mIoU) of 50.48. On the more challenging ActivityNet dataset, the [email protected] metric improved by 1.31%. These results suggest that the feedback loop is especially effective at refining the boundaries of predicted intervals, leading to higher precision at stricter thresholds.
Discussion and Conclusion
Beyond quantitative metrics, the back-query generation provides a level of explainability. If a model predicts an incorrect interval but generates a query like "hold bag" instead of "hold glass," it reveals a specific failure in visual recognition (misidentifying the object) rather than a general failure of the grounding logic.
While effective, the model faced challenges with the large vocabulary of ActivityNet, where the translation module struggled to generate accurate simplified queries. Future work will explore more advanced translation architectures, such as Transformers, and refine the selection of visual pivots to include more granular details beyond basic verbs and nouns.
Summary updated