Abstract
This paper presents a hybrid AI framework that integrates low-level computer vision tracking with high-level symbolic reasoning. By employing Answer Set Programming (ASP) for abductive inference, the system can hypothesize missing events such as occlusions and missed detections. This approach allows for the correction of fragmented tracks and identity switches, providing both improved tracking performance and human-readable explanations for object behavior in complex video sequences.
Introduction
Modern computer vision models excel at object detection but often lack the "commonsense" reasoning required for robust object permanence. When an object is temporarily hidden—for instance, a person walking behind a pillar—standard tracking algorithms frequently lose the detection, resulting in fragmented tracks and identity switches. Humans, by contrast, use intuitive physics to understand that the object still exists. This research bridges the gap between sub-symbolic visual processing and high-level semantic understanding to create a more robust and explainable tracking system.
Methodology: A Hybrid Architecture
The proposed system utilizes a tightly integrated two-part architecture designed to reconcile noisy visual data with logical constraints.
- Low-Level Visual Tracking ($\Sigma_{trk}$): This component follows a standard tracking-by-detection pipeline. It uses object detectors (e.g., Faster R-CNN) and Kalman filters to generate initial "space-time tracklets."
- High-Level Abductive Reasoning ($\Sigma_{abd}$): These noisy tracklets are fed into an Answer Set Programming (ASP) environment. ASP is a declarative logic language used here to encode commonsense rules about spatial and temporal continuity.
Rather than simple deduction, the system performs abduction—inference to the best explanation. When a track is interrupted, the ASP solver hypothesizes events like missing_detection or occludes to bridge the gap. An optimization layer then selects the most mathematically logical explanation, favoring smooth motion and penalizing physically unlikely scenarios.
Key Insights and Results
By translating spatial topologies into logical constraints, the system enforces physical reality on messy sensor data. This "top-down" logic actively fixes "bottom-up" errors, such as stitching together fragmented tracklets that logically belong to the same entity. Furthermore, the system provides explainability by design; its output includes a human-readable log of events (e.g., enters(from_left, trk9, 1500)).
Performance Evaluation
The abductive reasoning module demonstrated significant improvements across various datasets:
- Cinematic Scenes: In a scene from The Bad Sleep Well, Multi-Object Tracking Accuracy (MOTA) improved from 58.5% to 100%, completely eliminating identity mismatches.
- MOT16 Benchmark: On the complex "Venice-2" sequence, the ASP logic improved the baseline MOTA and reduced identity mismatches by 43.8% (from 153 to 86).
While the system significantly improves tracking consistency, the authors noted a slight decrease in tracking precision (MOTP) due to the use of linear interpolation for bounding boxes during occluded periods.
Conclusion and Future Work
The primary challenge for this approach is computational scalability, as the number of hypotheses grows exponentially with the number of tracks. Future work will focus on incremental computation strategies for real-time processing and the potential for Inductive Logic Programming to allow the system to learn physical rules directly from data rather than relying on hand-coded constraints.
Summary updated