Abstract
In order to be trusted by humans, Artificial Intelligence agents should be able to describe rationales behind their decisions. This paper presents a novel neuro-symbolic approach that recognizes actions from videos with human-understandable explanations. We propose representing videos symbolically by qualitative spatial relations between objects, termed qualitative spatial object relation chains. A neural saliency estimator is developed to capture the correlation between these chains and action occurrences, identifying which relations are most important for a given recognition. Evaluated on real-life video datasets, our approach achieves superior performance in both accuracy and explanation quality compared to previous symbolic methods.
Introduction
Deep learning has significantly advanced video action recognition, yet most state-of-the-art models function as "black boxes." In critical scenarios like autonomous driving or forensic investigations, understanding why a system made a specific decision is as important as the decision itself. For instance, knowing a car parked because it moved from being "disconnected" to "overlapping" with a parking space provides a verifiable rationale.
Existing symbolic approaches offer interpretability but often lack the accuracy of neural networks or require labor-intensive manual rule-tuning. This research bridges the gap by introducing a neuro-symbolic framework that utilizes Qualitative Spatial Relations (QSRs) to represent actions in a semantic space inherently closer to human cognition.
Methodology: Qualitative Spatial Object Relation Chains
The core of the proposed system is the symbolic representation of video content through spatial interactions. The pipeline consists of three main stages:
1. Object Relation Chain Extraction
The system tracks objects and extracts "chains" of spatial relations over time. These relations are defined using established spatial calculi such as the Region Connection Calculus (RCC). For example, a "car parking" action is represented as a sequence of topological states: disconnected -> overlapping -> inside.
2. Neural Saliency Estimator
A video often contains many simultaneous object interactions, many of which are irrelevant to the primary action (e.g., a pedestrian walking past a car while it parks). To address this, the authors developed a neural saliency estimator. This component learns a "pairwise co-occurrence saliency mapping matrix" to determine the importance of specific relation chains relative to others in the clip, effectively filtering out noise.
3. Explanation and Classification
Weighted by their saliency scores, these chains are fed into a classifier. Simultaneously, the system generates an explanation by selecting the top-$k$ highest-scoring chains, providing a human-understandable justification for the predicted action.
Key Results and Discussion
The approach was evaluated on the CAD120 and CAD120++ datasets. CAD120++ is particularly notable as it includes ground-truth explanations, allowing for a quantitative assessment of interpretability.
- Recognition Accuracy: On CAD120++, the model achieved 69.5% accuracy, outperforming traditional symbolic baselines (53.9%). On the original CAD120, it reached 95.9% accuracy.
- Explanation Quality: The model's most salient chain matched human ground-truth explanations in 61.6% of cases, nearly doubling the performance of baseline symbolic methods.
- Data Efficiency: By operating in a compact symbolic space rather than raw pixels, the model remains highly effective even with smaller training sets, avoiding the massive data requirements of 3D-Convolutional Neural Networks.
Conclusion
This neuro-symbolic framework demonstrates that interpretability does not have to come at the cost of performance. By leveraging qualitative spatial relations and neural saliency, the system provides a robust path toward trustworthy AI in action recognition. Future work will focus on reducing dependency on pre-processing (object tracking) and scaling the system to handle more complex multi-object interactions.
Summary updated