Learning to Anticipate Future with Dynamic Context Removal

Xinyu Xu, Yong-Lu Li, Cewu Lu

Abstract

Anticipating future events is an essential feature for intelligent systems and embodied AI. However, compared to the traditional recognition task, the uncertainty of the future and reasoning ability requirements make the anticipation task very challenging and far from solved. In this field, previous methods usually focus more on model architecture design, with little attention paid to how to train an anticipation model with a proper learning policy. To this end, we propose a novel training scheme called Dynamic Context Removal (DCR), which dynamically schedules the visibility of observed future in the learning procedure. It follows a human-like curriculum learning process, gradually removing the event context to increase the anticipation difficulty until satisfying the final anticipation target. Our learning scheme is plug-and-play and easy to integrate with any reasoning model, including Transformers and LSTMs, offering advantages in both effectiveness and efficiency. In extensive experiments, the proposed method achieves state-of-the-art results on four widely-used benchmarks.

1. Introduction

While action recognition—identifying what is happening in the present—has achieved significant success, action anticipation remains a formidable challenge. Anticipation requires predicting future actions (e.g., one second ahead) based on current observations. This task suffers from two primary issues: temporal misalignment, where current visual cues do not match future semantics, and a training policy gap, where models often overfit to current visual biases rather than learning temporal logic.

Most existing research attempts to bridge this gap through complex neural architectures. In contrast, this work introduces Dynamic Context Removal (DCR), a training strategy that focuses on the learning process itself. By treating the future as an "auxiliary teacher," DCR guides the model through a curriculum that transitions from easy, context-rich scenarios to difficult, context-free anticipation.

2. Methodology: Dynamic Context Removal (DCR)

The DCR framework is inspired by human learning patterns, such as mastering Sudoku by starting with many hints and gradually removing them. It consists of two primary stages designed to instill temporal reasoning.

2.1 Order-Aware Pre-training

Before the main anticipation task, the model undergoes self-supervised pre-training to learn video sequential order. Uniquely, the authors remove standard positional encodings, forcing the model to discover temporal order by predicting the relative position of frames using a Gaussian affinity loss. This ensures the model understands the flow of time inherently.

2.2 Reconstruction-Driven Curriculum Design

This is the core of the DCR framework. During training, the model is provided with "auxiliary frames" from the future action.

The DCR framework follows a curriculum learning process, gradually removing future context to increase task difficulty, similar to learning Sudoku.
  • Early Training (Easy): Most auxiliary frames are visible, allowing the model to learn the association between past context and future visuals.
  • Later Training (Difficult): An "easiness factor" ($T_e$) dynamically decreases, masking these auxiliary frames.
  • Instance-Specific Scheduling: DCR utilizes a memory bank to track reconstruction performance for specific video clips. If the model successfully reconstructs the future of a specific clip, that clip's curriculum is accelerated, making it harder in subsequent epochs.

The model is optimized using a dual loss: a classification loss for action prediction and a reconstruction loss (MSE) to ensure its internal representation of future frames is accurate.

3. Experimental Evaluation

3.1 Benchmarks and Performance

DCR was evaluated on four major benchmarks: EPIC-KITCHENS-100 & 55, EGTEA GAZE+, and 50-Salads. On EPIC-KITCHENS-100, DCR achieved a state-of-the-art Top-1 action score of 16.1% (Recall@5), outperforming previous leaders like AVT.

3.2 Efficiency and Versatility

A key advantage of DCR is its efficiency. The DCR-Transformer is approximately 4x lighter than the 12-layer AVT model while delivering superior performance. Furthermore, the method is model-agnostic; a DCR-enhanced LSTM (DCR-LSTM) outperformed several larger Transformer-based models, demonstrating the power of the training policy over architecture alone.

DCR achieves superior action scores with significantly fewer parameters compared to existing methods like AVT.

4. Conclusion

Dynamic Context Removal represents a shift in focus from architecture design to learning policies for action anticipation. By leveraging future frames as auxiliary supervision through a dynamic curriculum, DCR enables models to learn robust temporal reasoning. Future work will explore applying this scheduling to broader temporal tasks such as long-term trajectory prediction and video generation.

Summary updated

Abstract

Anticipating future events is an essential feature for intelligent systems and embodied AI. However, compared to the traditional recognition task, the uncertainty of future and reasoning ability requirement make the anticipation task very challenging and far beyond solved. In this filed, previous methods usually care more about the model architecture design or but few attention has been put on how to train an anticipation model with a proper learning policy. To this end, in this work, we propose a novel training scheme called Dynamic Context Removal (DCR), which dynamically schedules the visibility of observed future in the learning procedure. It follows the human-like curriculum learning process, i.e., gradually removing the event context to increase the anticipation difficulty till satisfying the final anticipation target. Our learning scheme is plug-and-play and easy to integrate any reasoning model including transformer and LSTM, with advantages in both effectiveness and efficiency. In extensive experiments, the proposed method achieves state-of-the-art on four widely-used benchmarks. Our code and models are publicly released at https://github.com/AllenXuuu/DCR.
BibTeX
@inproceedings{xu2022learning,
  author = {Xinyu Xu and Yong-Lu Li and Cewu Lu},
  title = {{Learning to Anticipate Future with Dynamic Context Removal}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=gscUsCR6Ks5},
}
Figure