Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation

Razvan-George Pasca, Alexey Gavryushin, Muhammad Hamza, Yen-Ling Kuo, Kaichun Mo, Luc Van Gool, Otmar Hilliges, Xi Wang

TL;DR

TransFusion improves future object interaction prediction in first-person videos by summarizing past video frames into concise natural language descriptions (e.g., "hold pan", "stir"). By fusing these text summaries with the current video frame using a multimodal transformer, the model leverages large language model priors to achieve a massive 40.4% relative performance boost on the Ego4D dataset, particularly excelling at rare, long-tailed actions.

What Problem Does This Paper Address?

Anticipating how a human will interact with objects in the near future—such as predicting what object they will grab, what action they will perform, and exactly when the contact will happen—is a foundational capability for assistive robotics and smart augmented reality (AR) glasses.

However, doing this from egocentric (first-person) video is incredibly challenging. Videos are full of visual clutter, changing perspectives, and ambiguities. To predict the next step, a model must understand the spatio-temporal "action context" of what just happened. Traditional methods rely on heavy 3D convolutional networks (like SlowFast) to process raw video chunks. These models struggle to filter out irrelevant background noise and often fail to generalize to rare or "long-tailed" activities because they lack semantic commonsense reasoning.

Methods

The authors introduce TransFusion, a multimodal transformer-based architecture that replaces raw past video features with compact text summaries.

The pipeline consists of two main phases:

  1. Past Summarization: For past video frames, the model extracts three types of language context using off-the-shelf models:
  2. Action descriptions: Verb-noun pairs (e.g., "cut tomato") generated via an image captioner (OFA) and part-of-speech tagging.
  3. Held objects: Identified using a hand-object interaction detector.
  4. Salient objects: Detected using a zero-shot CLIP classifier to capture environmental context. These frame-wise text snippets are aggregated and deduplicated over time into clean temporal segments.

  5. Multimodal Fusion: The current "prediction frame" is processed by a standard CNN (ResNet-50) to extract multiscale visual feature maps, which are flattened into visual tokens. Concurrently, the text summary is encoded into language tokens using Sentence-BERT (SBERT).

As shown in Figure 2, these visual and language tokens are concatenated and passed through a series of transformer self-attention layers at multiple scales. This allows the network to dynamically condition its visual search on the textual context before passing the fused features to a Faster R-CNN head to predict the final bounding box, noun, verb, and time-to-contact (TTC).

Figure 2: Overview of the TransFusion model. TransFusion takes the prediction frame and the action context summary as input and predicts the bounding box of the next-active object, the noun-verb pair describing the associated action, and the time to contact (TTC). Feature maps of different scales are extracted from the visual encoder and then fused via a multimodal fusion module with the encoded language features. Their output is then processed by a regular feature pyramid network (FPN), denoted as multi-scale region proposal networks, before being fed into the Faster R-CNN detector.

Datasets

  • Ego4D: A massive, diverse dataset of daily life egocentric videos. The authors evaluate specifically on the short-term object interaction anticipation task, which contains over 64,000 annotated clips.
  • EPIC-KITCHENS-100 (EK100): An unscripted first-person kitchen activity dataset used to validate how well the model generalizes to a more fine-grained, cluttered domain.

Key Insights and Ideas

  • Language as a Semantic Bottleneck: Text summaries act as an elegant filter. Instead of forcing a network to parse millions of raw pixels from past frames, language strips away visual clutter and preserves only the core semantic narrative of the past.
  • Unlocking Commonsense Priors: By converting video context into text, the model can leverage massive pre-trained language models. This injects "commonsense" into the network—knowing that if someone was just "holding a knife" and "holding a vegetable," the most plausible next action is "slice," even if the visual scene is highly ambiguous.
  • Extreme Data Efficiency: Text descriptions are incredibly compact. One second of video can be summarized in just a couple of words, taking up more than 12 times less space than standard 3D video embeddings (like SlowFast) while yielding superior predictive power.

Key Results

  • State-of-the-Art Performance: On the Ego4D test set, TransFusion achieves a 40.4% relative improvement in overall mean Average Precision (mAP) over the standard Ego4D baseline.
  • Conquering the Long Tail: The model shows extraordinary gains on rare, long-tailed classes. As detailed in Figure 5, it achieves a 161% relative improvement on tail verbs and a 22% relative improvement on tail nouns.
Figure 5: Classification performance on top/tail cate- gories. We show the relative and absolute gains of TransFu- sion over Ego4D for Noun-Only and Verb-Only mAP (with- out the IOU constraint). Relative improvements are written on top of the red bars.
  • Qualitative Success: Counterfactual analysis demonstrated that changing the text input dynamically alters the model's visual predictions, proving that TransFusion genuinely conditions its visual attention on the language context to resolve ambiguities.

Limitations and Future Work

  • Resolution Dependency: The summarization pipeline relies heavily on the quality of off-the-shelf vision-language models. When tested on lower-resolution datasets like EPIC-KITCHENS-100 (256x456), the quality of the generated captions degraded, which narrowed the performance gap between TransFusion and video-based models.
  • Lack of Explicit Motion Modeling: TransFusion currently ignores explicit temporal motion cues, such as optical flow. The authors note that integrating motion features could help refine the "time-to-contact" (TTC) estimations.
  • Future Directions: The authors suggest extending this language-bottleneck approach to long-term interaction anticipation and other complex video reasoning tasks.

Summary updated

Abstract

We study object interaction anticipation in egocentric videos. This task requires an understanding of the spatio-temporal context formed by past actions on objects, coined action context. We propose TransFusion, a multimodal transformer-based architecture. It exploits the representational power of language by summarizing the action context. TransFusion leverages pre-trained image captioning and vision-language models to extract the action context from past video frames. This action context together with the next video frame is processed by the multimodal fusion module to forecast the next object interaction. Our model enables more efficient end-to-end learning. The large pre-trained language models add common sense and a generalisation capability. Experiments on Ego4D and EPIC-KITCHENS-100 show the effectiveness of our multimodal fusion model. They also highlight the benefits of using language-based context summaries in a task where vision seems to suffice. Our method outperforms state-of-the-art approaches by 40.4% in relative terms in overall mAP on the Ego4D test set. We validate the effectiveness of TransFusion via experiments on EPIC-KITCHENS-100. Video and code are available at https://eth-ait.github.io/transfusion-proj/.
BibTeX
@inproceedings{pasca2024summarize,
  author = {Razvan-George Pasca and Alexey Gavryushin and Muhammad Hamza and Yen-Ling Kuo and Kaichun Mo and Luc Van Gool and Otmar Hilliges and Xi Wang},
  title = {{Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=g0u6xNAChC},
}
Figure