Visual Explanation by High-Level Abduction: On Answer-Set Programming Driven Reasoning About Moving Objects

Jakob Suchan, Mehul Bhatt, Przemyslaw Andrzej Walega, Carl P. L. Schultz

Abstract

This paper presents a hybrid AI framework that integrates low-level computer vision tracking with high-level symbolic reasoning. By employing Answer Set Programming (ASP) for abductive inference, the system can hypothesize missing events such as occlusions and missed detections. This approach allows for the correction of fragmented tracks and identity switches, providing both improved tracking performance and human-readable explanations for object behavior in complex video sequences.

Introduction

Modern computer vision models excel at object detection but often lack the "commonsense" reasoning required for robust object permanence. When an object is temporarily hidden—for instance, a person walking behind a pillar—standard tracking algorithms frequently lose the detection, resulting in fragmented tracks and identity switches. Humans, by contrast, use intuitive physics to understand that the object still exists. This research bridges the gap between sub-symbolic visual processing and high-level semantic understanding to create a more robust and explainable tracking system.

Methodology: A Hybrid Architecture

The proposed system utilizes a tightly integrated two-part architecture designed to reconcile noisy visual data with logical constraints.

  1. Low-Level Visual Tracking ($\Sigma_{trk}$): This component follows a standard tracking-by-detection pipeline. It uses object detectors (e.g., Faster R-CNN) and Kalman filters to generate initial "space-time tracklets."
  2. High-Level Abductive Reasoning ($\Sigma_{abd}$): These noisy tracklets are fed into an Answer Set Programming (ASP) environment. ASP is a declarative logic language used here to encode commonsense rules about spatial and temporal continuity.
The hybrid architecture combining low-level visual tracking (Sigma_trk) with high-level abductive reasoning (Sigma_abd) using Answer Set Programming.

Rather than simple deduction, the system performs abduction—inference to the best explanation. When a track is interrupted, the ASP solver hypothesizes events like missing_detection or occludes to bridge the gap. An optimization layer then selects the most mathematically logical explanation, favoring smooth motion and penalizing physically unlikely scenarios.

Key Insights and Results

By translating spatial topologies into logical constraints, the system enforces physical reality on messy sensor data. This "top-down" logic actively fixes "bottom-up" errors, such as stitching together fragmented tracklets that logically belong to the same entity. Furthermore, the system provides explainability by design; its output includes a human-readable log of events (e.g., enters(from_left, trk9, 1500)).

Illustration of the system handling a long-term occlusion by abducing the missing segment of the object's trajectory.

Performance Evaluation

The abductive reasoning module demonstrated significant improvements across various datasets:

  • Cinematic Scenes: In a scene from The Bad Sleep Well, Multi-Object Tracking Accuracy (MOTA) improved from 58.5% to 100%, completely eliminating identity mismatches.
  • MOT16 Benchmark: On the complex "Venice-2" sequence, the ASP logic improved the baseline MOTA and reduced identity mismatches by 43.8% (from 153 to 86).

While the system significantly improves tracking consistency, the authors noted a slight decrease in tracking precision (MOTP) due to the use of linear interpolation for bounding boxes during occluded periods.

Conclusion and Future Work

The primary challenge for this approach is computational scalability, as the number of hypotheses grows exponentially with the number of tracks. Future work will focus on incremental computation strategies for real-time processing and the potential for Inductive Logic Programming to allow the system to learn physical rules directly from data rather than relying on hand-coded constraints.

Summary updated

Abstract

We propose a hybrid architecture for systematically computing robust visual explanation(s) encompassing hypothesis formation, belief revision, and default reasoning with video data. The architecture consists of two tightly integrated synergistic components: (1) (functional) answer set programming based abductive reasoning with space-time tracklets as native entities; and (2) a visual processing pipeline for detection based object tracking and motion analysis. We present the formal framework, its general implementation as a (declarative) method in answer set programming, and an example application and evaluation based on two diverse video datasets: the MOTChallenge benchmark developed by the vision community, and a recently developed Movie Dataset.
BibTeX
@inproceedings{suchan2018visual,
  author = {Jakob Suchan and Mehul Bhatt and Przemyslaw Andrzej Walega and Carl P. L. Schultz},
  title = {{Visual Explanation by High-Level Abduction: On Answer-Set Programming Driven Reasoning About Moving Objects}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11569},
}
Figure