Abductive Action Inference

Clement Tan, Chai Kiat Yeo, Cheston Tan, Basura Fernando

Abstract

Abductive reasoning aims to make the most likely inference for a given set of incomplete observations. In this work, given a situation or a scenario, we aim to answer the question 'what is the set of actions that were executed by the human in order to come to this current state?', which we coin as abductive action inference. We provide a solution based on the human-object relations and their states in the given scene. Specifically, we first detect objects and humans in the scene, and then generate representations for each human-centric relation. Using these human-centric relations, we derive the most likely set of actions the human may have executed to arrive in this state. To generate human-centric relational representations, we investigate several models such as Transformers, a novel graph neural network-based encoder-decoder, and a new relational bilinear pooling method. We obtain promising results using these new models on this challenging task on the Action Genome dataset.

Introduction

Most computer vision research focuses on the "now" (action recognition) or the "next" (action anticipation). However, humans excel at abductive reasoning: looking at evidence—like an open cabinet and a glass on a table—and concluding that someone must have recently poured a drink. This ability is crucial for applications like elderly care (e.g., a robot noticing a fall or missed routine) and forensic investigation.

The challenge is that a single image contains no motion cues. To succeed, a model must understand "causal chains"—how specific actions change the environment—and use those changes as breadcrumbs to reconstruct the past. This paper introduces Abductive Past Action Inference, a task that challenges AI to look at a single static image and infer the sequence of human actions that led to that state.

Methodology

The researchers argue that standard "holistic" models (like vanilla CNNs or Transformers) fail because they don't explicitly reason about how humans interact with objects. They propose a suite of Object-Relational models that represent a scene as a set of triplets (e.g., <person, glass, table>).

Overview of the Abductive Action Inference pipeline, from object detection to inferring action sequences.

The core of the proposed solution is the BiGED (Bilinear Graph Encoder-Decoder). It combines two powerful techniques:

  1. Relational Graph Neural Networks (GNNs): These model the complex inter-dependencies between different objects in a room.
  2. Bilinear Pooling: This captures high-order interactions between human features and object features (e.g., not just "person" and "cup," but the specific way a person might be holding that cup).

The architecture uses a GNN-based encoder-decoder to summarize relational information into a vector, which is then passed through a classifier to predict the set or sequence of past actions.

Experimental Setup

The study utilizes an extension of the Action Genome (AG) dataset, which is built upon the Charades video dataset. The authors mapped 157 action classes to specific frames, creating ground-truth "past action sets" for every image. They tested models on two scenarios: Abduct at T (inferring actions from any random point in a sequence) and Abduct last image (inferring the entire history of a video from its final frame).

Key Results and Insights

Performance comparison of various models on the Abductive Action Inference task using the Action Genome dataset.
  • The "Reasoning Gap": Human performance on this task is roughly 80.6 mAP, while the best AI models hover around 35.7 mAP. This highlights a massive gap in logical reasoning capabilities in current vision systems.
  • Relations > Pixels: Standard end-to-end models (like ResNet or ViT) performed poorly. Success requires moving from raw pixels to a symbolic understanding of human-object relationships.
  • Bilinear Magic: Bilinear pooling (RBP) consistently outperformed standard Transformers. The authors suggest this is because bilinear operators are better at modeling the "fine-grained" physics of how an action (like "pouring") leaves a specific visual trace on an object.
  • Top Performer: The BiGED model achieved the highest scores across most metrics, reaching 35.75 mAP and 60.55% Recall@10 on the "Abduct at T" task.

Limitations and Future Work

The primary limitation is the "Object Detection Bottleneck." The models perform significantly better when given ground-truth object labels. When using predicted labels from a standard detector (FasterRCNN), performance drops by about 10-12 points.

Future work could focus on improving how models extract clean relational data from messy, real-world images and moving beyond fixed action classes to more generalized, "in-the-wild" abductive reasoning. Integrating more explicit physical "world models" could also help the AI understand why certain actions leave specific visual marks.

Summary updated

Abstract

Abductive reasoning aims to make the most likely inference for a given set of incomplete observations. In this work, given a situation or a scenario, we aim to answer the question 'what is the set of actions that were executed by the human in order to come to this current state?', which we coin as abductive action inference. We provide a solution based on the human-object relations and their states in the given scene. Specifically, we first detect objects and humans in the scene, and then generate representations for each human-centric relation. Using these human-centric relations, we derive the most likely set of actions the human may have executed to arrive in this state. To generate human-centric relational representations, we investigate several models such as Transformers, a novel graph neural network-based encoder-decoder, and a new relational bilinear pooling method. We obtain promising results using these new models on this challenging task on the Action Genome dataset.
BibTeX
@inproceedings{tan2025abductive,
  author = {Clement Tan and Chai Kiat Yeo and Cheston Tan and Basura Fernando},
  title = {{Abductive Action Inference}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00800},
}
Figure