Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction

Razvan-George Pasca, Alexey Gavryushin, Yen-Ling Kuo, Otmar Hilliges, Xi Wang

Abstract

Anticipating human-object interactions in egocentric video is a fundamental challenge for assistive robotics and augmented reality. Traditional vision-only models often struggle with visual clutter and the long-tail distribution of real-world actions. We present TransFusion, a multimodal transformer architecture that leverages natural language summaries of past video frames to provide semantic context for future predictions. By converting visual history into a compact linguistic representation, TransFusion utilizes the common-sense reasoning capabilities of pre-trained language models. Our approach achieves a 40.4% relative improvement in mean Average Precision (mAP) over state-of-the-art methods on the Ego4D benchmark, demonstrating the efficacy of language as a robust medium for temporal context.

1. Introduction

Short-term object interaction anticipation requires a model to predict the next object a user will interact with, the action they will perform, and the time to contact (TTC). For instance, if a user is in a kitchen holding a vegetable, a predictive system must determine if the next likely action is "slicing" (requiring a knife) or "washing" (requiring a sink).

Existing methods typically rely on raw video features extracted from fixed-size temporal windows. However, visual data is often ambiguous and noisy. Furthermore, these models frequently fail on "long-tail" actions—rare interactions that appear infrequently in training sets. We propose that natural language provides a more efficient and semantically rich way to represent past context. By summarizing the past into text, we can leverage the inherent "world knowledge" found in large language models to bridge the gap between visual observation and intent prediction.

2. The TransFusion Architecture

TransFusion is a multimodal fusion transformer designed to integrate current visual observations with linguistic summaries of the past. The pipeline consists of three primary stages:

2.1 Context Extraction and Aggregation

Rather than processing raw video history directly, TransFusion uses a suite of "expert" models to generate a textual summary of previous frames. An image captioning model (OFA) generates descriptions of actions, CLIP identifies salient objects in the environment, and a hand-object interaction detector tracks objects currently in use. To ensure stability and remove detection flicker, these per-frame summaries are aggregated into "action segments," resulting in a concise string such as: "spoon; hold pan; stir pan."

2.2 Multimodal Fusion

The model processes two distinct inputs:

  1. Visual Stream: The current video frame is processed via a ResNet-50 backbone to extract spatial features.
  2. Linguistic Stream: The aggregated text summary is encoded using a Sentence-BERT (SBERT) model.

These features are projected into a shared embedding space and processed by a Transformer encoder. The joint attention mechanism allows the model to use the linguistic context to attend to specific regions or objects within the current visual frame, effectively using the "past" to filter the "present."

2.3 Prediction Head

The final output of the model includes the bounding box of the next active object, the object category (noun), the predicted interaction (verb), and the Time to Contact (TTC).

3. Experimental Results

We evaluated TransFusion on two major egocentric benchmarks: Ego4D and EPIC-KITCHENS-100 (EK100). Ego4D is particularly challenging due to its massive scale and highly imbalanced distribution of actions.

3.1 Quantitative Performance

TransFusion established a new state-of-the-art on the Ego4D benchmark, achieving a 40.4% relative improvement in mAP compared to previous vision-only baselines. When focusing specifically on the accuracy of Noun-Verb pair predictions, the model demonstrated a 47% improvement.

3.2 Efficiency and Generalization

One of the most striking findings was the efficiency of the linguistic representation. The text-based features used by TransFusion were 12x smaller in size than standard video features (such as SlowFast), yet they provided superior context. This compactness allows the model to maintain a much longer historical memory without the computational costs associated with high-dimensional video buffers.

Furthermore, TransFusion excelled in the "long tail." While vision-only models often fail when they lack thousands of visual examples for a specific action, TransFusion uses the semantic relationships learned during SBERT's pre-training to make informed guesses. For example, the model "knows" that a "knife" is semantically related to "cutting," even if it has rarely seen that specific visual sequence in the training data.

4. Discussion and Limitations

While TransFusion represents a significant step forward, it remains dependent on the quality of the initial "expert" models. If the captioning or detection models fail to identify a critical object in the past, the downstream prediction may be compromised. Additionally, while language is excellent for high-level semantics, it can lack the fine-grained temporal granularity (such as the specific velocity of a hand movement) necessary for perfect TTC estimation. Future work may explore the integration of motion cues, such as optical flow, alongside the linguistic context.

5. Conclusion

TransFusion demonstrates that "reading" the past is a powerful alternative to simply "seeing" it. By using natural language as a semantic bottleneck, we can filter out visual noise, leverage common-sense reasoning, and significantly improve the accuracy of human-object interaction anticipation in complex, real-world environments.

Summary updated

Abstract

We study object interaction anticipation in egocentric videos. This task requires an understanding of the spatiotemporal context formed by past actions on objects, coined action context. We propose TransFusion, a multimodal transformer-based architecture. It exploits the representational power of language by summarising the action context. TransFusion leverages pre-trained image captioning and vision-language models to extract the action context from past video frames. This action context together with the next video frame is processed by the multimodal fusion module to forecast the next object interaction. Our model enables more efficient end-to-end learning. The large pre-trained language models add common sense and a generalisation capability. Experiments on Ego4D and EPIC-KITCHENS-100 show the effectiveness of our multimodal fusion model. They also highlight the benefits of using language-based context summaries in a task where vision seems to suffice. Our method outperforms state-of-the-art approaches by 40.4% in relative terms in overall mAP on the Ego4D test set. We validate the effectiveness of TransFusion via experiments on EPIC-KITCHENS-100. Video and code are available at https://eth-ait.github.io/transfusion-proj/.
BibTeX
@inproceedings{pasca2024summarize,
  author = {Razvan-George Pasca and Alexey Gavryushin and Yen-Ling Kuo and Otmar Hilliges and Xi Wang},
  title = {{Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=g0u6xNAChC},
}
Figure