Towards Explainable Action Recognition by Salient Qualitative Spatial Object Relation Chains

Hua Hua, Dongxu Li, Ruiqi Li, Peng Zhang, Jochen Renz, Anthony G. CohnAAAI 2022 · OpenReview

Abstract

In order to be trusted by humans, Artificial Intelligence agents should be able to describe rationales behind their decisions. This paper presents a novel neuro-symbolic approach that recognizes actions from videos with human-understandable explanations. We propose representing videos symbolically by qualitative spatial relations between objects, termed qualitative spatial object relation chains. A neural saliency estimator is developed to capture the correlation between these chains and action occurrences, identifying which relations are most important for a given recognition. Evaluated on real-life video datasets, our approach achieves superior performance in both accuracy and explanation quality compared to previous symbolic methods.

Introduction

Deep learning has significantly advanced video action recognition, yet most state-of-the-art models function as "black boxes." In critical scenarios like autonomous driving or forensic investigations, understanding why a system made a specific decision is as important as the decision itself. For instance, knowing a car parked because it moved from being "disconnected" to "overlapping" with a parking space provides a verifiable rationale.

Existing symbolic approaches offer interpretability but often lack the accuracy of neural networks or require labor-intensive manual rule-tuning. This research bridges the gap by introducing a neuro-symbolic framework that utilizes Qualitative Spatial Relations (QSRs) to represent actions in a semantic space inherently closer to human cognition.

Methodology: Qualitative Spatial Object Relation Chains

The core of the proposed system is the symbolic representation of video content through spatial interactions. The pipeline consists of three main stages:

1. Object Relation Chain Extraction

The system tracks objects and extracts "chains" of spatial relations over time. These relations are defined using established spatial calculi such as the Region Connection Calculus (RCC). For example, a "car parking" action is represented as a sequence of topological states: disconnected -> overlapping -> inside.

Region Connection Calculus (RCC) relations (a) and their hierarchy (b) used to define the symbolic vocabulary for spatial states.

2. Neural Saliency Estimator

A video often contains many simultaneous object interactions, many of which are irrelevant to the primary action (e.g., a pedestrian walking past a car while it parks). To address this, the authors developed a neural saliency estimator. This component learns a "pairwise co-occurrence saliency mapping matrix" to determine the importance of specific relation chains relative to others in the clip, effectively filtering out noise.

Comparison between a salient relation chain (a) representing an object entering another, and a non-salient 'noise' chain (b) representing an object passing by.

3. Explanation and Classification

Weighted by their saliency scores, these chains are fed into a classifier. Simultaneously, the system generates an explanation by selecting the top-$k$ highest-scoring chains, providing a human-understandable justification for the predicted action.

Key Results and Discussion

The approach was evaluated on the CAD120 and CAD120++ datasets. CAD120++ is particularly notable as it includes ground-truth explanations, allowing for a quantitative assessment of interpretability.

  • Recognition Accuracy: On CAD120++, the model achieved 69.5% accuracy, outperforming traditional symbolic baselines (53.9%). On the original CAD120, it reached 95.9% accuracy.
  • Explanation Quality: The model's most salient chain matched human ground-truth explanations in 61.6% of cases, nearly doubling the performance of baseline symbolic methods.
  • Data Efficiency: By operating in a compact symbolic space rather than raw pixels, the model remains highly effective even with smaller training sets, avoiding the massive data requirements of 3D-Convolutional Neural Networks.

Conclusion

This neuro-symbolic framework demonstrates that interpretability does not have to come at the cost of performance. By leveraging qualitative spatial relations and neural saliency, the system provides a robust path toward trustworthy AI in action recognition. Future work will focus on reducing dependency on pre-processing (object tracking) and scaling the system to handle more complex multi-object interactions.

Summary updated

Abstract

In order to be trusted by humans, Artificial Intelligence agents should be able to describe rationales behind their decisions. One such application is human action recognition in critical or sensitive scenarios, where trustworthy and explainable action recognizers are expected. For example, reliable pedestrian action recognition is essential for self-driving cars and explanations for real-time decision making are critical for investigations if an accident happens. In this regard, learning-based approaches, despite their popularity and accuracy, are disadvantageous due to their limited interpretability. This paper presents a novel neuro-symbolic approach that recognizes actions from videos with human-understandable explanations. Specifically, we first propose to represent videos symbolically by qualitative spatial relations between objects called qualitative spatial object relation chains. We further develop a neural saliency estimator to capture the correlation between such object relation chains and the occurrence of actions. Given an unseen video, this neural saliency estimator is able to tell which object relation chains are more important for the action recognized. We evaluate our approach on two real-life video datasets, with respect to recognition accuracy and the quality of generated action explanations. Experiments show that our approach achieves superior performance on both aspects to previous symbolic approaches, thus facilitating trustworthy intelligent decision making. Our approach can be used to augment state-of-the-art learning approaches with explainabilities.
BibTeX
@inproceedings{hua2022towards,
  author = {Hua Hua and Dongxu Li and Ruiqi Li and Peng Zhang and Jochen Renz and Anthony G. Cohn},
  title = {{Towards Explainable Action Recognition by Salient Qualitative Spatial Object Relation Chains}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=_K1K6-kqf-},
}
Figure