Transform-Equivariant Consistency Learning for Temporal Sentence Grounding

Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Zichuan Xu, Haozhao Wang, Xing Di, Weining Lu, Yu Cheng

TL;DR

The Equivariant Consistency Regulation Learning (ECRL) framework improves temporal sentence grounding by teaching models that the "meaning" of a video segment stays the same even if you change its playback speed or appearance. By using self-supervised consistency losses and a novel smoothing module, ECRL achieves state-of-the-art accuracy while being significantly faster and more robust to dataset biases than previous methods.

What Problem Does This Paper Address?

Temporal Sentence Grounding (TSG) is the task of identifying the start and end timestamps of a specific activity within an untrimmed video based on a natural language query (e.g., "The woman adds oil to the pan").

While recent deep learning models have made progress, they face two major hurdles:

  1. Data Hunger: They require massive amounts of manually annotated video-query pairs, which are expensive to produce.
  2. Distribution Bias: Models often "cheat" by memorizing the statistical patterns of where annotations usually fall in a dataset (e.g., always guessing the middle of the video) rather than actually understanding the relationship between the text and the visual action.

The authors aim to solve these by introducing a self-supervised approach that forces the model to learn truly discriminative features that are robust to video transformations.

Methods

The core of the paper is the ECRL framework, which consists of four main components:

  • Spatio-Temporal Augmentation: To create "synthetic" training data, the authors apply transformations to both the foreground (the action) and background (irrelevant parts) of a video. This includes spatial changes (cropping, color jitter) and temporal changes (randomly speeding up or slowing down segments).
  • Self-Refine Module: Standard augmentations can make video frames look "choppy" or disconnected. This module uses a graph-based approach—calculating both temporal and semantic similarity between frames—to smooth out the representations and maintain continuity.
  • Self-Supervised Consistency Loss (SSCL): This is the "aha" moment of the paper. Instead of a standard contrastive loss (which might incorrectly treat similar neighboring frames as "negatives"), the SSCL uses a prior Gaussian distribution. It ensures that the similarity between the original and augmented video follows a bell curve based on timestamp distance, capturing the invariant semantics of the action.
  • Shared Grounding Head: A proposal-free head that predicts the start and end boundaries. Because it is shared across original and augmented videos, it learns equivariance: if the video is sped up, the predicted timestamps should shift accordingly.

Datasets

The authors evaluated ECRL on three standard TSG benchmarks:

  • ActivityNet: ~20,000 YouTube videos with diverse activities.
  • TACoS: Fine-grained cooking activities in a consistent kitchen setting.
  • Charades-STA: Indoor daily life activities.
  • De-biased Benchmarks: They also tested on ActivityNet-CD and Charades-CD, which are specifically designed to expose models that rely on distribution bias.

Key Insights and Ideas

  • Equivariance over Invariance: In many computer vision tasks, we want "invariance" (the output shouldn't change). In TSG, we want "equivariance"—if we stretch the video in time, the predicted boundaries should stretch by the exact same factor.
  • The Neighborhood Problem: Most self-supervised models fail in video because neighboring frames are almost identical. Treating them as "negative samples" confuses the model. By using a Gaussian prior, ECRL acknowledges that frames close in time should be similar.
  • Plug-and-Play Capability: The authors demonstrated that their ECRL modules can be added to existing "backbone" models (like LGI or VSLNet) to immediately boost their performance and robustness.

Key Results

  • State-of-the-Art Performance: ECRL outperformed all existing methods across all three datasets. For example, on ActivityNet, it achieved a 54.24% R@1 (IoU=0.5), a significant jump over previous leaders.
  • Beating the Bias: On the de-biased "OOD" (Out-of-Distribution) tests, ECRL showed massive improvements (e.g., 19.95% vs. 11.66% for the previous best on ActivityNet-CD), proving it actually "understands" the content rather than just memorizing patterns.
  • Efficiency: ECRL is remarkably lean. It processes a video in 0.06 seconds, compared to 4.31 seconds for older proposal-based methods like ACRN, and uses significantly fewer parameters.

Limitations and Future Work

While ECRL is highly effective, the authors note that the quality of the initial feature extraction (the "Video Encoder") still plays a large role. Future work could involve tighter integration between the self-supervised consistency learning and the raw pixel-level feature extraction to capture even more fine-grained temporal details. Additionally, exploring how this framework handles multi-step activities (queries involving "then" or "after") remains an open challenge.

Summary updated

Abstract

This paper addresses the temporal sentence grounding (TSG). Although existing methods have made decent achievements in this task, they not only severely rely on abundant video-query paired data for training, but also easily fail into the dataset distribution bias. To alleviate these limitations, we introduce a novel Equivariant Consistency Regulation Learning (ECRL) framework to learn more discriminative query-related frame-wise representations for each video, in a self-supervised manner. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted under various video-level transformations. Concretely, we first design a series of spatio-temporal augmentations on both foreground and background video segments to generate a set of synthetic video samples. In particular, we devise a self-refine module to enhance the completeness and smoothness of the augmented video. Then, we present a novel self-supervised consistency loss (SSCL) applied on the original and augmented videos to capture their invariant query-related semantic by minimizing the KL-divergence between the sequence similarity of two videos and a prior Gaussian distribution of timestamp distance. At last, a shared grounding head is introduced to predict the transform-equivariant query-guided segment boundaries for both the original and augmented videos. Extensive experiments on three challenging datasets (ActivityNet, TACoS, and Charades-STA) demonstrate both effectiveness and efficiency of our proposed ECRL framework.
BibTeX
@article{liu2024transform,
  author = {Daizong Liu and Xiaoye Qu and Jianfeng Dong and Pan Zhou and Zichuan Xu and Haozhao Wang and Xing Di and Weining Lu and Yu Cheng},
  title = {{Transform-Equivariant Consistency Learning for Temporal Sentence Grounding}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2024},
  url = {https://openreview.net/forum?id=BGe4pp1rd8},
}
Figure