Out of Sight But Not Out of Mind: An Answer Set Programming Based Online Abduction Framework for Visual Sensemaking in Autonomous Driving

Jakob Suchan, Mehul Bhatt, Srikrishna VaradarajanIJCAI 2019 · DOI

TL;DR

This paper introduces a novel framework for "visual sensemaking" in autonomous driving, using Answer Set Programming (ASP) to integrate low-level visual perception with high-level semantic reasoning. The system can interpret complex dynamic scenes, predict future events like occlusions, and provide explanations, crucial for safety-critical applications.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on computer vision to perceive the world. While deep learning has made significant strides in tasks like object detection and tracking, a critical gap remains in understanding the meaning and implications of these perceptions, especially in dynamic and uncertain situations. This includes explaining why an object is behaving a certain way, predicting events that aren't directly visible (like an occluded cyclist reappearing), and ensuring the system's reasoning is transparent and human-understandable. Current systems often focus on "how fast to drive" and "which way to steer," neglecting the deeper semantic understanding needed for ethical, explainable, and robust autonomous operation. This paper aims to bridge this gap by developing a system that can perform online, incremental "visual sensemaking" – essentially, understanding the "story" of a scene in real-time.

Methods

The core of the proposed solution is an Answer Set Programming (ASP) based framework for online abduction. ASP is a declarative programming paradigm well-suited for non-monotonic reasoning and finding explanations for observed data.

Here's a breakdown of the key methodological components:

  • Hybrid Architecture: The framework integrates state-of-the-art deep learning-based visual computing (for tasks like object detection and semantic segmentation) with high-level ASP-based reasoning.
  • Online Abduction: Instead of processing data in large batches, the system operates incrementally, processing information frame-by-frame in real-time. Abduction is used to find the "best explanation" for the observed visual data.
  • Declarative Model of Scene Dynamics: The system uses ontological primitives to represent objects, their spatial properties, motion, and temporal relationships. This includes concepts like "occlusion," "entering/leaving the scene," and "missing detections."
  • Tracking as Abduction: The process of associating detected objects with existing tracks is framed as an abductive problem. The system generates hypotheses for matching observations to tracks and high-level events that explain these associations.
  • Event Calculus: A variant of event calculus is used to reason about dynamic properties and events occurring in the scene.
  • Optimization: ASP's optimization capabilities are used to find the "optimal hypothesis" by maximizing matching likelihood between detections and tracks, and minimizing the cost of inferred events.

Datasets

The framework is evaluated using two widely recognized datasets:

  • KITTI Object Tracking Dataset: This dataset is a standard benchmark for autonomous driving, providing annotated data for object detection and tracking in real-world scenarios.
  • MOT (Multi-Object Tracking) Dataset: This dataset offers more challenging, unconstrained videos with both static and moving cameras, providing a broader evaluation of tracking performance.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Bridging Vision and Semantics is Crucial: The paper strongly argues that simply having good visual perception isn't enough. Integrating semantic reasoning is essential for safety-critical applications like autonomous driving, enabling explainability and human-centric AI.
  • Abduction for Explanation: Framing visual sensemaking as abduction allows the system to not just detect objects but also infer why they are behaving as they are, especially in ambiguous situations like occlusions.
  • Online Processing is Feasible: The framework demonstrates that complex abductive reasoning can be performed in real-time, making it suitable for dynamic environments.
  • Hidden Entities are Predictable: The system can hypothesize about and predict the reappearance of occluded objects, which is vital for proactive safety measures.
  • Joint Reasoning Improves Tracking: By jointly abducing object dynamics and high-level events, the system achieves improved accuracy in object tracking compared to methods that only rely on detection and prediction.

Key Results

  • Improved Tracking Accuracy: The framework shows a consistent improvement in tracking accuracy (MOTA) by approximately 5% for cars and 3.86% for pedestrians on the KITTI dataset, and 4.8% on the MOT dataset, when compared to tracking without abduction.
  • Real-time Performance: The system can achieve real-time performance, processing up to 30 frames per second with up to 10 highly overlapping object tracks. For realistic scenes like KITTI, it runs at 33.9fps with YOLOv3 detectors.
  • Qualitative Demonstrations: The paper demonstrates the system's ability to handle occlusion scenarios, correctly inferring events like "hides_behind" and predicting the reappearance of occluded objects.

Limitations and Future Work

  • Tracking Accuracy Improvement: While the framework improves tracking, the authors acknowledge that the core tracking mechanism relies on a naive IoU measure and doesn't use visual information for prediction or association. They suggest that improving the underlying prediction methods (e.g., using particle filters or optical flow) could further enhance tracking performance.
  • Focus on Specific Events: The current abducible events are primarily focused on object persistence and visibility (occlusion, entering/leaving scene). Future work could incorporate more complex commonsense theories and specialized theories for multi-sensory integration, multi-agent belief merging, and contextual knowledge.
  • Dataset Development: The authors plan to develop a novel dataset that emphasizes semantics and commonsense explainability, driven by mixed-methods research in AI, Psychology, and HCI. This dataset will focus on understanding driving behavior in complex urban environments with unstructured traffic and natural interactions.

Summary updated

BibTeX
@inproceedings{suchan2019out,
  author = {Jakob Suchan and Mehul Bhatt and Srikrishna Varadarajan},
  title = {{Out of Sight But Not Out of Mind: An Answer Set Programming Based Online Abduction Framework for Visual Sensemaking in Autonomous Driving}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/260},
}
Figure