This paper introduces a fascinating new challenge for computer vision: Abductive Past Action Inference. While most AI research focuses on recognizing what is happening now or predicting what will happen next, this work asks a model to look at a single static image and figure out what happened before to reach that state.
TL;DR
The authors propose a novel task and dataset for inferring past human actions from a single image using abductive reasoning. They introduce BiGED, a specialized architecture that combines graph neural networks and bilinear pooling to reason about human-object relationships, significantly outperforming standard end-to-end vision models.
What Problem Does This Paper Address?
Imagine a rescue robot entering an elderly person's home. It sees an open back door and a half-empty glass of water on a table. A human would immediately infer that the resident likely poured water, drank some, and then walked out the door. This is abductive reasoning—inferring the most plausible explanation from incomplete observations.
Current AI excels at action recognition (identifying actions in a video) and action anticipation (predicting the future). However, "looking backward" from a static image is much harder because there are no motion cues. The model must understand the "causal chain" of how actions change the world. The authors argue this capability is vital for human-robot interaction, accident investigation, and assistive robotics.
Methods
The authors frame the problem through three sub-tasks:
- Past Action Set Prediction: Identifying all actions that occurred.
- Past Action Sequence Prediction: Determining the chronological order of those actions.
- Abductive Verification: Answering "Yes/No" to whether a specific action occurred.
To solve this, they move away from holistic image features (which struggle to find subtle clues) and focus on human-object relational representations. Their proposed model, the Bilinear Graph Encoder-Decoder (BiGED), uses a two-pronged approach:
- Relational Bilinear Pooling (RBP): Models the complex, high-order interactions within a single human-object pair (e.g., the specific way a hand holds a glass).
- Graph Neural Network (GNNED): Models the interactions between different relations in the scene (e.g., how the state of the cabinet relates to the glass on the table).
The architecture of the BiGED model is illustrated below:
Datasets
The authors introduce the Action Genome Past Action Inference dataset, built upon the existing Action Genome and Charades benchmarks.
- They mapped 157 action classes to specific keyframes.
- The dataset includes over 234,000 frames where the "ground truth" is the set of actions that occurred in the video prior to that frame.
- They evaluated two setups: Abduct at T (inferring actions up to a random point) and Abduct last image (inferring the entire history from the final frame of a sequence).
Key Insights and Ideas
- Objects are Clues: Standard vision models (ResNet, ViT) perform poorly because they don't explicitly reason about objects as "evidence." Relational models that treat the scene as a set of human-object triplets are far more effective.
- The Power of "Bilinear + Graph": The authors found that GNNs are great at global reasoning, but Bilinear pooling is better at capturing the nuances of a specific interaction. Combining them (BiGED) provides the best of both worlds.
- LLMs Aren't a Silver Bullet: Interestingly, the authors tested Large Language Models (GPT-3.5) and Vision-Language Models (VILA). Even when provided with object labels, these models struggled with the logic of past actions, often being too conservative or failing to understand the physical constraints of a home environment.
- Future Context Helps Training: During training, providing the model with a few "future" snapshots (K-frames) improved its ability to learn the causal relationships, even if those frames aren't available during final inference.
Key Results
The BiGED model established a new state-of-the-art for this task, significantly outperforming end-to-end baselines.
As shown in Table 1, BiGED achieved 35.75 mAP, nearly four times higher than a standard ResNet101-2D (9.27 mAP). However, a significant "human-machine gap" remains: human evaluators achieved over 80% Recall@10, while the best models hover around 60%.
Limitations and Future Work
- Semantic Bottleneck: The models perform much better when given ground-truth object labels. When using predicted labels from an object detector (FasterRCNN), performance drops significantly, showing that errors in basic perception cascade into reasoning failures.
- Sequence Difficulty: Predicting the exact order of past actions remains an open challenge. The models only achieved ~10% accuracy in sequence prediction, likely because many different sequences can lead to the same final state (the "permutation problem").
- Real-world Complexity: The current work is limited to closed-set classification in home environments. Future work could explore open-vocabulary abduction in more diverse settings.
Summary updated