Inferring Past Human Actions in Homes with Abductive Reasoning

Clement Tan, Chai Kiat Yeo, Cheston Tan, Basura Fernando

Introduction

While computer vision has made significant strides in recognizing current actions (Action Recognition) and predicting future ones (Action Anticipation), the ability to infer the past remains a relatively unexplored frontier. Humans naturally perform abductive reasoning—the process of "inference to the best explanation"—when observing a scene. For instance, seeing an open cabinet and a glass on a counter allows a human to conclude that someone recently retrieved the glass. This paper formalizes this capability as a new research task: Abductive Past Action Inference (APAI).

The APAI Challenge

APAI requires an AI to analyze a single static image and reconstruct the sequence of actions that led to the observed state. Unlike standard action recognition, the visual cues for the actions themselves (like motion blur or active interaction) are often absent. Instead, the model must rely on the effects of past actions—the state of objects and their spatial relationships to the human. This task is crucial for developing intelligent robots and assistants capable of understanding context in environments like elderly care or forensic investigation.

Methodology: The BiGED Model

The authors argue that the key to solving APAI lies in modeling human-object relationships rather than relying solely on raw pixel data. They propose BiGED (Bilinear Graph Encoder-Decoder), a framework designed to capture both global scene context and fine-grained interactions.

The APAI pipeline: from object detection to abductive action inference.

The BiGED architecture consists of several integrated components:

  1. Relational Representation: The system uses Faster R-CNN to extract features for the human, relevant objects, and the "union" area between them. Semantic knowledge is incorporated via GloVe word embeddings to provide the model with a conceptual understanding of the objects.
  2. GNNED (Graph Neural Network Encoder-Decoder): This module models the interdependencies between various relationships in the room, allowing the model to understand how "person-near-fridge" might correlate with "person-holding-milk."
  3. RBP (Relational Bilinear Pooling): To capture high-order interactions, the model employs bilinear pooling. This allows for a more sophisticated fusion of human and object features than simple concatenation, enabling the model to distinguish between subtle interaction types.

Experimental Setup and Datasets

The research utilizes an adapted version of the Action Genome (AG) dataset, derived from the Charades video dataset. The authors established two primary evaluation setups:

  • Abduct at T: Predicting all actions from the start of a sequence up to a specific frame $T$.
  • Abduct last image: A more challenging task where the model must reconstruct the entire action history based solely on the final frame of a video.

The dataset encompasses 157 action classes and over 1.7 million visual relationships, providing a robust benchmark for abductive reasoning.

Results and Discussion

The experimental results demonstrate that standard "black box" vision models, such as ResNet-101 or Vision Transformers (ViTs), struggle with APAI, often achieving less than 10% mean Average Precision (mAP). This failure underscores that abductive reasoning requires structural, relational logic rather than just high-quality feature extraction.

Comparative performance of various models on the APAI task.

In contrast, the BiGED model significantly outperforms these baselines. As shown in the results table, BiGED achieved 35.75 mAP on the "Abduct at T" task when using ground-truth human and object detections. Even in a binary verification task—answering "Yes/No" queries about past actions—the model showed reasonable proficiency.

Despite these gains, a substantial "human gap" remains. While the best AI models hover around 37% mAP, human performance on the same tasks reaches approximately 80%. This gap highlights APAI as a significant and challenging new frontier for machine reasoning.

Conclusion and Future Directions

The introduction of APAI and the BiGED model marks a pivotal step toward AI that understands the temporal context of human environments. Current limitations include a dependence on accurate object detection and the constraints of a closed-set classification problem. Future work may involve integrating Large Language Models (LLMs) to handle more open-ended abductive queries and improving the underlying vision systems to better handle real-world noise.

Summary updated

Abstract

Abductive reasoning aims to make the most likely inference for a given set of incomplete observations. In this paper, we introduce “Abductive Past Action Inference”, a novel research task aimed at identifying the past actions performed by individuals within homes to reach specific states captured in a single image, using abductive inference. The research explores three key abductive inference problems: past action set prediction, past action sequence prediction, and abductive past action verification. We introduce several models tailored for abductive past action inference, including a relational graph neural network, a relational bilinear pooling model, and a relational transformer model. Notably, the newly proposed object-relational bilinear graph encoder-decoder (BiGED) model emerges as the most effective among all methods evaluated, demonstrating good proficiency in handling the intricacies of the Action Genome dataset. The contributions of this research significantly advance the ability of deep learning models to reason about current scene evidence and make highly plausible inferences about past human actions. This advancement enables a deeper understanding of events and behaviors, which can enhance decision-making and improve system capabilities across various real-world applications such as Human-Robot Interaction and Elderly Care and Health Monitoring. Code and data available at https://github.com/LUNAProject22/AAR
BibTeX
@inproceedings{tan2025inferring,
  author = {Clement Tan and Chai Kiat Yeo and Cheston Tan and Basura Fernando},
  title = {{Inferring Past Human Actions in Homes with Abductive Reasoning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00800},
}
Figure