Visual Explanation by High-Level Abduction: On Answer-Set Programming Driven Reasoning About Moving Objects

Jakob Suchan, Mehul Bhatt, Przemyslaw Andrzej Walega, Carl Schultz

Introduction

Modern computer vision has achieved remarkable success in object detection and tracking through deep learning. However, these systems often lack "common sense" reasoning, struggling with scenarios where objects are occluded, enter/exit scenes, or when detectors fail intermittently. Humans, by contrast, use abductive reasoning to fill these gaps—inferring that an object hidden behind a pillar still exists. This paper, presented at AAAI-18, introduces a hybrid framework that bridges low-level visual tracking with high-level symbolic reasoning to provide logical, human-understandable explanations for object dynamics.

A Hybrid Architecture for Visual Reasoning

The proposed system integrates two distinct layers to move beyond purely statistical tracking:

  1. Low-Level Visual Pipeline ($\Sigma_{trk}$): Utilizing standard tools like Faster R-CNN and Kalman filters, this layer generates "tracklets"—short, fragmented segments of an object's trajectory.
  2. High-Level Abductive Reasoning ($\Sigma_{abd}$): This core component uses Answer-Set Programming (ASP) to evaluate tracklets against spatio-temporal rules based on Region Connection Calculus (RCC8).
The hybrid architecture combining low-level tracking ($\Sigma_{trk}$) with high-level abductive reasoning ($\Sigma_{abd}$) to generate visual explanations ($VXP$).

The system operates through three levels of Visual Explanation ($VXP$):

  • $VXP_1$ (Hypothesizing): Proposing theories about events such as scene entries or occlusions.
  • $VXP_2$ (Belief Revision): Resolving logical contradictions, such as ensuring a single object does not occupy two distinct locations simultaneously.
  • $VXP_3$ (Default Assumptions): Maintaining object persistence across frames unless evidence suggests otherwise.

Reasoning as Global Optimization

Unlike traditional trackers that operate frame-by-frame, this framework treats perception as a global optimization problem. It seeks the "cheapest" explanation by minimizing a cost function where, for instance, assuming a temporary occlusion is preferred over the spontaneous creation of a new object. This allows the system to maintain a consistent "story" across an entire video sequence by filling in the blanks of raw visual data.

Examples of abduced events, including missed detections and scene entries, which allow the system to maintain track continuity through logical inference.

Experimental Evaluation

The framework was tested on the MOTChallenge (MOT16) benchmark and a specialized "Movie Dataset" featuring complex movements from films by directors like Wes Anderson and Akira Kurosawa. These scenes were chosen specifically for their long-term occlusions that test the limits of logical consistency.

Key findings include:

  • Significant Accuracy Gains: In a complex scene from Kurosawa’s The Bad Sleep Well, the reasoning engine improved Multi-Object Tracking Accuracy (MOTA) from 58.5% to 100% by resolving intricate occlusions.
  • Reduced Identity Switches: On the Venice-2 dataset (MOT16), the system achieved a 43.8% reduction in "miss-matches" compared to the base tracker.
  • Computational Efficiency: Despite the symbolic complexity, the ASP engine is highly performant, abducing hundreds of optimal models for 11 tracks in under one second (0.973s).

Conclusion and Future Directions

While the system demonstrates the power of combining qualitative reasoning with quantitative tracking, challenges remain regarding scalability as the hypothesis space grows. Future work aims to implement incremental reasoning for real-time applications and explore Inductive Logic Programming (ILP) to automatically learn scene-specific rules, reducing the reliance on hand-coded spatio-temporal logic.

Summary updated

Abstract

We propose a hybrid architecture for systematically computing robust visual explanation(s) encompassing hypothesis formation, belief revision, and default reasoning with video data. The architecture consists of two tightly integrated synergistic components: (1) (functional) answer set programming based abductive reasoning with space-time tracklets as native entities; and (2) a visual processing pipeline for detection based object tracking and motion analysis. We present the formal framework, its general implementation as a (declarative) method in answer set programming, and an example application and evaluation based on two diverse video datasets: the MOTChallenge benchmark developed by the vision community, and a recently developed Movie Dataset.
BibTeX
@inproceedings{suchan2018visual,
  author = {Jakob Suchan and Mehul Bhatt and Przemyslaw Andrzej Walega and Carl Schultz},
  title = {{Visual Explanation by High-Level Abduction: On Answer-Set Programming Driven Reasoning About Moving Objects}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11569},
}
Figure