Abstract
Abductive reasoning aims to make the most likely inference for a given set of incomplete observations. In this paper, we introduce "Abductive Past Action Inference", a novel research task aimed at identifying the past actions performed by individuals within homes to reach specific states captured in a single image, using abductive inference. The research explores three key abductive inference problems: past action set prediction, past action sequence prediction, and abductive past action verification. We introduce several models tailored for abductive past action inference, including a relational graph neural network, a relational bilinear pooling model, and a relational transformer model. Notably, the newly proposed object-relational bilinear graph encoder-decoder (BiGED) model emerges as the most effective among all methods evaluated, demonstrating good proficiency in handling the intricacies of the Action Genome dataset. The contributions of this research significantly advance the ability of deep learning models to reason about current scene evidence and make highly plausible inferences about past human actions. This advancement enables a deeper understanding of events and behaviors, which can enhance decision-making and improve system capabilities across various real-world applications such as Human-Robot Interaction and Elderly Care and Health Monitoring.
Introduction
Humans possess a unique cognitive ability called abductive reasoning: the power to look at a "result" (e.g., a kitchen with an open flour bag, a dirty bowl, and a preheating oven) and infer the most likely "cause" (someone is baking). While computer vision has made strides in action recognition ("what is happening now?") and action anticipation ("what will happen next?"), the inverse problem—inferring the sequence of actions that led to a current state from a single static image—remains a significant challenge.
This paper formalizes Abductive Past Action Inference (APAI) as a novel research task. The core difficulty lies in the fact that a single image represents an "incomplete observation"; multiple different historical sequences could potentially lead to the same visual state. Current end-to-end models, such as ResNets or Vision Transformers (ViTs), often lack the logical framework to connect visual evidence to these causal chains.
The APAI Task and Methodology
The authors argue that the key to "reading" a scene lies in understanding human-object relationships. Rather than relying on global image features, the proposed framework moves toward a structured, relational approach. The task is evaluated across three sub-problems:
- Set Prediction: Identifying the unordered set of all past actions.
- Sequence Prediction: Determining the chronological order of those actions.
- Verification: A binary "Yes/No" task asking if a specific action (e.g., "Did the person open the fridge?") occurred.
The BiGED Model
The primary contribution of this work is the Object-Relational Bilinear Graph Encoder-Decoder (BiGED). This model integrates two sophisticated techniques to handle scene complexity:
- Graph Neural Networks (GNNs): Used to model how different objects in a room relate to one another spatially and semantically.
- Bilinear Pooling: Employed to capture high-order interactions between the human's state and the objects they might have interacted with, providing a more nuanced fusion than standard attention mechanisms.
The system utilizes FasterRCNN for visual feature extraction and GloVe word embeddings for semantic features, ensuring the model understands not just what an object looks like, but its functional purpose.
Datasets and Experimental Setup
To benchmark this task, the researchers extended the Action Genome (AG) dataset. While the original AG focuses on frame-by-frame relationships, the authors mapped the entire history of a video to its final frames. The resulting dataset includes 9,848 videos with over 1.72 million visual relationships. They established two primary setups: Abduct at T (reasoning from any point in a sequence) and Abduct last image (reasoning from the final state of a completed activity).
Key Results and Insights
Experimental evaluations reveal a massive "reasoning gap" between AI and humans. Humans achieve approximately 80 mAP on action set prediction, while even the most advanced models struggle to reach half of that performance.
As shown in the results, BiGED is the state-of-the-art, achieving 35.75 mAP in the "Abduct at T" task, significantly outperforming standard end-to-end models like ResNet101-2D (9.27 mAP) and ViT (7.27 mAP).
Several key insights emerged from the study:
- Relations > Pixels: Explicitly modeling relationships (e.g., Person + Cabinet + Open) is far more effective than holistic image processing. Standard models fail because they don't "understand" that an open cabinet is a causal clue.
- Semantics Matter: Adding language-based knowledge via GloVe significantly boosts performance over using visual pixels alone.
- Sequence Prediction is Brutal: Predicting the exact order of actions is the hardest task for both humans and AI, with human accuracy at only 14% and models slightly lower at ~10.5%.
- LLM Limitations: Testing GPT-3.5 and VILA (a Vision-Language Model) showed they were often too conservative or prone to hallucinating actions not supported by visual evidence.
Conclusion and Future Work
This research advances the ability of deep learning models to reason about current scene evidence to make plausible inferences about the past. While BiGED represents a significant step forward, limitations remain. Some actions leave no visual trace, creating inherent causal ambiguity. Furthermore, current models struggle with temporal depth—distinguishing between an action that happened seconds ago versus minutes ago. Future work will explore "open-world" reasoning, allowing models to describe past actions in natural language rather than selecting from a fixed set.
Summary updated