Learning to Anticipate Future with Dynamic Context Removal

Xinyu Xu, Yong-Lu Li, Cewu Lu

TL;DR

Action anticipation is a "holy grail" for embodied AI, but training models to predict the future is notoriously difficult due to the inherent uncertainty of what comes next. This paper introduces Dynamic Context Removal (DCR), a curriculum learning scheme that starts by allowing a model to see "auxiliary" future frames and gradually hides them as the model improves. By forcing the model to reconstruct the missing future from past observations, DCR achieves state-of-the-art results across four major benchmarks while remaining compatible with both Transformer and LSTM architectures.

What Problem Does This Paper Address?

In computer vision, action recognition (identifying what is happening now) has seen massive success. However, action anticipation (predicting what will happen in, say, one second) remains a major hurdle.

The challenge is twofold:

  1. Temporal Misalignment: The visual cues in the current frame often don't match the semantics of the future action (e.g., looking at a fridge doesn't visually "look like" the act of slicing an apple, which happens later).
  2. Training Policy Gap: Most research focuses on designing better neural architectures (like deeper Transformers), but very few look at how we should train these models. Standard training often leads to models that overfit on current visual biases rather than learning the underlying temporal logic.

Methods

The authors propose a "human-like" learning process inspired by how a child learns Sudoku: you start with many hints (easy) and gradually remove them until you can solve the puzzle with almost no information (hard).

The DCR framework consists of two main stages:

  1. Order-Aware Pre-training: Before the main task, the model undergoes self-supervised training to learn video sequential order. Interestingly, the authors remove standard positional encodings. Instead, they force the Transformer to "discover" the temporal order by predicting the relative position of frames using a Gaussian affinity loss.
  2. Reconstruction-Driven Curriculum Design: This is the core of DCR. During training, the model is given "auxiliary frames"—actual frames from the future action occurrence.
  3. Early Training: Most auxiliary frames are visible (Easy). The model learns to associate past context with these future visuals.
  4. Later Training: An "easiness factor" ($T_e$) dynamically decreases, masking out these auxiliary frames.
  5. Instance-Specific Scheduling: Unlike a simple linear decay, DCR uses a memory bank to track how well the model reconstructs specific video clips. If the model gets "good" at imagining the future of a specific clip, the curriculum makes that specific clip harder in the next epoch.

Datasets

The authors evaluated DCR on four widely-used benchmarks, covering both egocentric (first-person) and third-person perspectives:

  • EPIC-KITCHENS-100 & 55: Massive datasets of unscripted daily activities in kitchens.
  • EGTEA GAZE+: An egocentric dataset that includes eye-tracking data.
  • 50-Salads: A third-person dataset focused on long-term salad preparation tasks.

Key Insights and Ideas

  • The Future as an Auxiliary Teacher: Instead of just using a class label (e.g., "wash cup") as supervision, DCR uses the actual pixels of the future as a temporary crutch. This helps the model learn a richer representation of what "the future" looks like before it's forced to guess it blindly.
  • Reconstruction vs. Classification: The model is trained with a dual loss: a classification loss to get the action right, and a reconstruction loss (MSE) to ensure its internal "imagination" of the future frames is accurate. The authors found that removing the reconstruction loss caused a significant performance drop.
  • Model Agnostic: While the paper leans into Transformers, they proved DCR works for LSTMs too. In fact, their DCR-LSTM outperformed several much larger, more complex Transformer-based models.

Key Results

DCR set new state-of-the-art (SOTA) records across the board:

  • EPIC-KITCHENS-100: Achieved a top action score of 16.1% (Recall@5), outperforming the previous leader, AVT, while being significantly more parameter-efficient.
  • Efficiency: The DCR-Transformer is roughly 4x lighter than the 12-layer AVT model but provides better performance.
  • Qualitative Success: t-SNE visualizations showed that the order-aware pre-training successfully allowed the model to map temporal dynamics into a coherent manifold, proving it "understands" the flow of time even without explicit position markers.

Limitations and Future Work

The authors acknowledge that their local easiness scheduling is somewhat empirical. While the memory bank approach works well, a more theoretical or finer-grained analysis of the curriculum could yield even better results. Additionally, like all models trained on human-annotated data, there is a risk of inheriting biases from the way humans label "start" and "end" times of actions.

Future work may involve applying DCR to other temporal tasks, such as video generation or long-term trajectory prediction for autonomous vehicles.

Summary updated

Abstract

Anticipating future events is an essential feature for in-telligent systems and embodied AI. However, compared to the traditional recognition task, the uncertainty of future and reasoning ability requirement make the anticipation task very challenging and far beyond solved. In this filed, previous methods usually care more about the model ar-chitecture design or but few attention has been put on how to train an anticipation model with a proper learning policy. To this end, in this work, we propose a novel training scheme called Dynamic Context Removal (DCR), which dynamically schedule the visibility of observed future in the learning procedure. It follows the human-like curriculum learning process, i.e., gradually removing the event context to increase the anticipation difficulty till satisfying the final anticipation target. Our learning scheme is plug-and-play and easy to integrate any reasoning model including transformer and LSTM, with advantages in both effectiveness and efficiency. In extensive experiments, the pro-posed method achieves state-of-the-art on four widely-used benchmarks. Our code and models are publicly released at https://github.com/AllenXuuuIDCR.
BibTeX
@inproceedings{xu2022learning,
  author = {Xinyu Xu and Yong-Lu Li and Cewu Lu},
  title = {{Learning to Anticipate Future with Dynamic Context Removal}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=gscUsCR6Ks5},
}

Loading related content...

Figure