Abstract
Temporal Sentence Grounding (TSG) is a challenging task that requires identifying the start and end timestamps of activities in untrimmed videos based on natural language queries. Existing models often struggle with high annotation costs and a tendency to memorize dataset biases rather than understanding visual-textual relationships. We introduce the Equivariant Consistency Regulation Learning (ECRL) framework, which employs self-supervised consistency losses and spatio-temporal augmentations to improve grounding accuracy and robustness. ECRL achieves state-of-the-art performance across multiple benchmarks, including de-biased datasets, while maintaining high computational efficiency.
Introduction
The core challenge of TSG lies in finding the "needle" (the specific action) in the "haystack" (the untrimmed video). Traditional supervised methods require extensive manual labeling, which is both expensive and prone to capturing statistical shortcuts—where models predict timestamps based on common patterns in the training set rather than the actual query content.
To address these issues, we propose a self-supervised approach that focuses on equivariance. Unlike invariance, where a model's output remains unchanged despite input transformations, equivariance ensures that if a video is temporally transformed (e.g., sped up), the predicted timestamps shift accordingly. This forces the model to learn truly discriminative features that are robust to video transformations.
The ECRL Framework
The ECRL framework consists of four integrated components designed to foster robust feature learning:
- Spatio-Temporal Augmentation: This module creates synthetic training data by applying spatial transformations (cropping, color jitter) and temporal shifts (randomly scaling the duration of foreground and background segments).
- Self-Refine Module: To mitigate the "choppiness" introduced by random augmentations, this module uses a graph-based approach to smooth frame representations by considering both temporal and semantic similarities.
- Self-Supervised Consistency Loss (SSCL): Standard contrastive losses often fail in video tasks because adjacent frames are semantically similar. SSCL utilizes a prior Gaussian distribution to ensure that similarity between original and augmented frames follows a natural temporal decay, capturing invariant semantics without confusing the model with false negatives.
- Shared Grounding Head: By sharing a proposal-free grounding head across original and augmented streams, the model learns to map transformed visual features back to their corresponding temporal boundaries consistently.
Experimental Results
ECRL was evaluated on three major benchmarks: ActivityNet, TACoS, and Charades-STA. It consistently outperformed existing state-of-the-art (SOTA) methods.
- Accuracy: On ActivityNet, ECRL achieved a 54.24% R@1 (IoU=0.5), setting a new performance standard.
- Robustness: On de-biased benchmarks like ActivityNet-CD, ECRL showed a dramatic improvement (19.95% vs. 11.66% for the previous SOTA), indicating that the model relies on actual content understanding rather than dataset bias.
- Efficiency: The model is highly optimized, processing videos in just 0.06 seconds, making it significantly faster than traditional proposal-based architectures like ACRN.
Conclusion
ECRL demonstrates that transform-equivariant consistency learning is a powerful tool for temporal sentence grounding. By forcing the model to maintain semantic consistency across spatio-temporal transformations, we reduce the reliance on massive annotated datasets and mitigate the impact of distribution biases. Future work will explore deeper integration with raw pixel-level features to capture even finer temporal nuances.
Summary updated