Abstract
Deep learning models for object detection often operate as "black boxes," providing high accuracy but little insight into their decision-making process. This paper introduces the AOGParsing operator, which integrates hierarchical AND-OR Graphs (AOGs) into the R-CNN framework. By unfolding latent structures of objects into parse trees, the model provides visual rationales for its detections without requiring part-level supervision. Experimental results on PASCAL VOC and COCO datasets demonstrate that this structural interpretability is achieved without compromising detection performance.
Introduction
While deep neural networks have achieved human-level accuracy in object detection, they are notoriously difficult to interpret. A standard Faster R-CNN might correctly identify a "cat," but the decision is made within a "flat" fully-connected layer that pools features into a single, uninterpretable vector. This lack of transparency is problematic in high-stakes environments like autonomous driving or medical imaging, where understanding the basis of a model's prediction is crucial. Existing interpretability methods are often post-hoc or produce blurry heatmaps. This research aims to build interpretability directly into the architecture by forcing the model to represent objects as structured hierarchies of parts.
Methodology: The AOGParsing Operator
The core innovation is the AOGParsing operator, integrated into the two-stage R-CNN framework. Instead of a standard flat layer, the authors use a top-down grammar model known as an AND-OR Graph (AOG).
Structural Components
- AND-nodes represent the decomposition of a large part into smaller ones (e.g., a car is composed of a body and wheels).
- OR-nodes represent alternative configurations or viewpoints (e.g., a car seen from the front vs. the side).
- Terminal-nodes correspond to specific feature maps for localized regions.
Learning and Inference
Crucially, the model is trained under weak supervision; it is not given labels for "wheels" or "heads." It learns to "unfold" these latent structures automatically using only standard bounding box and category labels. A Value Sub-network acts as an attention mechanism, weighting different part configurations based on their discriminative power. To ensure robust learning, a Folding-Unfolding Strategy is employed: the model first trains all parts equally ("folding") before allowing it to select the best structural explanation ("unfolding").
Experimental Results
The authors evaluated their method on two gold-standard benchmarks: PASCAL VOC 2007 and COCO 2017.
Performance and Interpretability
One of the most significant findings is that structural interpretability does not come at the cost of performance. On PASCAL VOC 2007, the method achieved a Mean Average Precision (mAP) of 82.1%, identical to the baseline Faster R-CNN. On the more challenging COCO dataset, it achieved a Box AP of 38.2%, remaining competitive with state-of-the-art black-box models.
The "best parse tree" selected by the model serves as a qualitative explanation. As shown in the visualizations, the model correctly identifies the layouts of various objects, effectively justifying its bounding boxes with structural evidence.
Conclusion and Future Work
By using a hierarchical grammar as an inductive bias, the researchers provide the model with a prior about the world's structure—that objects are composed of parts. While the results are promising, quantifying interpretability remains a challenge. Future work may involve developing "interpretability-sensitive" loss functions and optimizing the computational efficiency of the parsing process for real-time applications. Additionally, moving from predefined AOG structures to learning the grammar itself represents a significant future direction.
Summary updated