Introduction
Modern computer vision has achieved remarkable success in object detection and tracking through deep learning. However, these systems often lack "common sense" reasoning, struggling with scenarios where objects are occluded, enter/exit scenes, or when detectors fail intermittently. Humans, by contrast, use abductive reasoning to fill these gaps—inferring that an object hidden behind a pillar still exists. This paper, presented at AAAI-18, introduces a hybrid framework that bridges low-level visual tracking with high-level symbolic reasoning to provide logical, human-understandable explanations for object dynamics.
A Hybrid Architecture for Visual Reasoning
The proposed system integrates two distinct layers to move beyond purely statistical tracking:
- Low-Level Visual Pipeline ($\Sigma_{trk}$): Utilizing standard tools like Faster R-CNN and Kalman filters, this layer generates "tracklets"—short, fragmented segments of an object's trajectory.
- High-Level Abductive Reasoning ($\Sigma_{abd}$): This core component uses Answer-Set Programming (ASP) to evaluate tracklets against spatio-temporal rules based on Region Connection Calculus (RCC8).
The system operates through three levels of Visual Explanation ($VXP$):
- $VXP_1$ (Hypothesizing): Proposing theories about events such as scene entries or occlusions.
- $VXP_2$ (Belief Revision): Resolving logical contradictions, such as ensuring a single object does not occupy two distinct locations simultaneously.
- $VXP_3$ (Default Assumptions): Maintaining object persistence across frames unless evidence suggests otherwise.
Reasoning as Global Optimization
Unlike traditional trackers that operate frame-by-frame, this framework treats perception as a global optimization problem. It seeks the "cheapest" explanation by minimizing a cost function where, for instance, assuming a temporary occlusion is preferred over the spontaneous creation of a new object. This allows the system to maintain a consistent "story" across an entire video sequence by filling in the blanks of raw visual data.
Experimental Evaluation
The framework was tested on the MOTChallenge (MOT16) benchmark and a specialized "Movie Dataset" featuring complex movements from films by directors like Wes Anderson and Akira Kurosawa. These scenes were chosen specifically for their long-term occlusions that test the limits of logical consistency.
Key findings include:
- Significant Accuracy Gains: In a complex scene from Kurosawa’s The Bad Sleep Well, the reasoning engine improved Multi-Object Tracking Accuracy (MOTA) from 58.5% to 100% by resolving intricate occlusions.
- Reduced Identity Switches: On the Venice-2 dataset (MOT16), the system achieved a 43.8% reduction in "miss-matches" compared to the base tracker.
- Computational Efficiency: Despite the symbolic complexity, the ASP engine is highly performant, abducing hundreds of optimal models for 11 tracks in under one second (0.973s).
Conclusion and Future Directions
While the system demonstrates the power of combining qualitative reasoning with quantitative tracking, challenges remain regarding scalability as the hypothesis space grows. Future work aims to implement incremental reasoning for real-time applications and explore Inductive Logic Programming (ILP) to automatically learn scene-specific rules, reducing the reliance on hand-coded spatio-temporal logic.
Summary updated