Abstract
Anticipating future events is an essential feature for intelligent systems and embodied AI. However, compared to the traditional recognition task, the uncertainty of the future and reasoning ability requirements make the anticipation task very challenging and far from solved. In this field, previous methods usually focus more on model architecture design, with little attention paid to how to train an anticipation model with a proper learning policy. To this end, we propose a novel training scheme called Dynamic Context Removal (DCR), which dynamically schedules the visibility of observed future in the learning procedure. It follows a human-like curriculum learning process, gradually removing the event context to increase the anticipation difficulty until satisfying the final anticipation target. Our learning scheme is plug-and-play and easy to integrate with any reasoning model, including Transformers and LSTMs, offering advantages in both effectiveness and efficiency. In extensive experiments, the proposed method achieves state-of-the-art results on four widely-used benchmarks.
1. Introduction
While action recognition—identifying what is happening in the present—has achieved significant success, action anticipation remains a formidable challenge. Anticipation requires predicting future actions (e.g., one second ahead) based on current observations. This task suffers from two primary issues: temporal misalignment, where current visual cues do not match future semantics, and a training policy gap, where models often overfit to current visual biases rather than learning temporal logic.
Most existing research attempts to bridge this gap through complex neural architectures. In contrast, this work introduces Dynamic Context Removal (DCR), a training strategy that focuses on the learning process itself. By treating the future as an "auxiliary teacher," DCR guides the model through a curriculum that transitions from easy, context-rich scenarios to difficult, context-free anticipation.
2. Methodology: Dynamic Context Removal (DCR)
The DCR framework is inspired by human learning patterns, such as mastering Sudoku by starting with many hints and gradually removing them. It consists of two primary stages designed to instill temporal reasoning.
2.1 Order-Aware Pre-training
Before the main anticipation task, the model undergoes self-supervised pre-training to learn video sequential order. Uniquely, the authors remove standard positional encodings, forcing the model to discover temporal order by predicting the relative position of frames using a Gaussian affinity loss. This ensures the model understands the flow of time inherently.
2.2 Reconstruction-Driven Curriculum Design
This is the core of the DCR framework. During training, the model is provided with "auxiliary frames" from the future action.
- Early Training (Easy): Most auxiliary frames are visible, allowing the model to learn the association between past context and future visuals.
- Later Training (Difficult): An "easiness factor" ($T_e$) dynamically decreases, masking these auxiliary frames.
- Instance-Specific Scheduling: DCR utilizes a memory bank to track reconstruction performance for specific video clips. If the model successfully reconstructs the future of a specific clip, that clip's curriculum is accelerated, making it harder in subsequent epochs.
The model is optimized using a dual loss: a classification loss for action prediction and a reconstruction loss (MSE) to ensure its internal representation of future frames is accurate.
3. Experimental Evaluation
3.1 Benchmarks and Performance
DCR was evaluated on four major benchmarks: EPIC-KITCHENS-100 & 55, EGTEA GAZE+, and 50-Salads. On EPIC-KITCHENS-100, DCR achieved a state-of-the-art Top-1 action score of 16.1% (Recall@5), outperforming previous leaders like AVT.
3.2 Efficiency and Versatility
A key advantage of DCR is its efficiency. The DCR-Transformer is approximately 4x lighter than the 12-layer AVT model while delivering superior performance. Furthermore, the method is model-agnostic; a DCR-enhanced LSTM (DCR-LSTM) outperformed several larger Transformer-based models, demonstrating the power of the training policy over architecture alone.
4. Conclusion
Dynamic Context Removal represents a shift in focus from architecture design to learning policies for action anticipation. By leveraging future frames as auxiliary supervision through a dynamic curriculum, DCR enables models to learn robust temporal reasoning. Future work will explore applying this scheduling to broader temporal tasks such as long-term trajectory prediction and video generation.
Summary updated