Introduction
Visual Relation Detection (VRD) is a fundamental task in computer vision that involves identifying triples of the form <subject, predicate, object> within an image. While deep learning models have achieved significant success in this domain, they often operate as "black boxes," requiring vast amounts of labeled data and offering little insight into their decision-making processes. At AAAI-21, Kezhen Chen and Ken Forbus proposed a hybrid approach that integrates deep learning's perceptual capabilities with the logic-based analogical learning framework from cognitive science. This system aims to achieve high performance with greater data efficiency and inherent explainability.
The Hybrid Pipeline
The proposed system utilizes a two-stage pipeline that bridges connectionist perception with symbolic reasoning.
- Perception Stage: The system employs state-of-the-art deep learning models, specifically Faster-RCNN and Mask-RCNN, to perform object detection. These models identify object categories, bounding boxes, and segmentation masks from raw pixel data.
- Representation Stage: Instead of feeding high-dimensional feature vectors into a classifier, the system converts visual data into qualitative representations using CogSketch. This tool encodes spatial and geometric relationships into symbolic predicates. These include topological relations (e.g., Region Connection Calculus), positional relations (e.g.,
above,rightOf), and geometric attributes (e.g.,highRectangularity). - Reasoning Stage: The symbolic descriptions are processed by the Sequential Analogical Generalization Engine (SAGE). SAGE leverages the Structure Mapping Engine (SME) to compare new instances against a library of previously learned examples, organized into "generalization pools."
Analogical Learning and Explainability
A core advantage of this hybrid approach is its efficiency. While traditional neural networks often require dozens of training epochs to converge, the analogical learner can reach peak performance in just a single pass (one epoch) through the training data. This is possible because the system focuses on the underlying structural commonalities between examples rather than pixel-level variations.
Furthermore, the system is inherently explainable. Because SAGE builds generalization pools based on symbolic predicates, the "knowledge" captured by the model is human-readable. For instance, a generalization for the predicate "Above" might explicitly include the rule that the center of the subject must be vertically higher than the object, with associated probabilities for each feature.
Experimental Evaluation
The model was evaluated on the Visual Relationship Dataset (VRD), which consists of 5,000 images and 70 predicates. The researchers tested the system across two primary tasks:
- Predicate Detection (PREDT): Given ground-truth object labels and bounding boxes, the task is to predict the relationship. The hybrid system achieved a Recall@50 of 52.38%, outperforming several deep learning baselines such as VTransE (44.76%) and VRD (47.87%).
- Relationship Detection (RELDT): This more challenging task requires detecting both objects and their relations from scratch. The model achieved a Recall@50 of 16.12%. While this outperformed some established baselines like SA-Full (15.80%), it trailed behind the most advanced end-to-end neural architectures.
Conclusion and Future Directions
The hybrid analogical learning approach demonstrates that symbolic reasoning can complement deep learning to create more efficient and transparent AI systems. By stripping away visual noise and focusing on qualitative spatial structures, the model achieves competitive results with significantly less training overhead than pure connectionist models.
Future work will focus on refining the qualitative descriptions. The authors plan to incorporate part-level information (e.g., recognizing that a "handle" is part of a "cup") and richer shape encodings to better capture object poses. Additionally, the system's modular nature allows for the integration of more advanced object detectors as they become available.
Summary updated