There and Back Again: Combining Non-monotonic Logical Reasoning and Deep Learning on an Assistive Robot

Mohan Sridharan, Chloé Benz, Arthur Findelair, Kévin GloaguenNMR 2022 · OpenReview

Introduction

The field of robotics is increasingly moving toward assistive roles in human-centric environments, such as restaurants or hospitals. However, creating robots that can interact naturally and reliably remains a significant challenge. This paper presents a hybrid architecture that combines the strengths of two traditionally separate AI paradigms: non-monotonic logical reasoning (symbolic AI) and deep learning (neural networks). By creating a bidirectional feedback loop between these components, the researchers developed a system that is more accurate, efficient, and explainable than standalone models.

The Challenge of Hybrid Intelligence

Robots in dynamic environments face two primary hurdles. First, they must operate with incomplete commonsense knowledge. While they may know general rules, they frequently encounter exceptions that require them to retract previous conclusions—a process known as non-monotonic reasoning. Second, while deep learning excels at processing noisy sensor data, such as human hand gestures, it often functions as a "black box." This lack of transparency makes it difficult for humans to understand the robot's decisions and requires massive amounts of data for training.

The core contribution of this work is the tight integration of these two paradigms. Instead of treating reasoning and learning as isolated modules, the architecture allows the robot’s logical "brain" to guide its perceptual "eyes," and vice versa.

Methods: A Bidirectional Architecture

The proposed architecture bridges the gap between the virtual and physical worlds through three interconnected components:

  1. Knowledge Representation & Reasoning: The system utilizes Answer Set Prolog (ASP), specifically the CR-Prolog extension. This allows the robot to handle defaults and exceptions gracefully. For instance, if a robot assumes a table is free but then detects a personal item, it can immediately update its belief state.
  2. Deep Learning for Perception: To interpret the physical world, the team employed Artificial Neural Networks (ANNs) integrated with the OpenPose system. OpenPose tracks 21 keypoints on the human hand, providing a structured input for the ANNs to recognize specific gestures.
  3. The Feedback Loop:
    • Reasoning to Learning: The robot uses its current task context to filter incoming data. If the robot is in the middle of a delivery, it can ignore gestures unrelated to its current goal, effectively "zooming in" on relevant signals and reducing the complexity of the learning task.
  4. Learning to Reasoning: When the robot encounters an unfamiliar gesture, it can solicit human feedback. This feedback is then translated into new logical axioms and integrated into the robot's knowledge base.
The hybrid architecture integrates logical reasoning with interactive learning.

Datasets and Perception

Rather than using standard large-scale datasets, the researchers created a custom dataset of 30 hand gestures performed in a physical setting. Using OpenPose, they extracted keypoint sequences to train various ANN configurations. The system was evaluated in a simulated restaurant environment using PyBullet, where a Pepper robot interacted with virtual customers based on real-world human inputs.

Hand gesture skeletons used for training the neural network models.

Key Results and Insights

The integration of reasoning and learning yielded substantial improvements across several metrics:

  • Accuracy: Standalone neural networks achieved approximately 85% accuracy in gesture recognition. However, when context-aware reasoning was applied to filter the input, accuracy improved to nearly 100%.
  • Efficiency: The models reached optimal performance significantly faster when guided by reasoning, demonstrating high learning efficiency.
  • Explainability: One of the most significant advantages is the ability to generate human-readable explanations. By tracing the "proof tree" of its logical deductions, the robot can justify its actions (e.g., "I seated the customers at Table 2 because it was the closest available table"). In testing, the robot provided correct relational descriptions in 95% of cases.

Conclusion and Future Work

This research demonstrates that combining symbolic reasoning with deep learning creates more robust and transparent robotic systems. By allowing logic to provide context for learning and learning to update logical rules, the robot becomes more resilient to noise and better at communicating with humans. Future work will focus on moving beyond simulation to full physical deployment in complex environments and making the human-robot feedback loop more natural and conversational.

Summary updated

BibTeX
@inproceedings{sridharan2022there,
  author = {Mohan Sridharan and Chloé Benz and Arthur Findelair and Kévin Gloaguen},
  title = {{There and Back Again: Combining Non-monotonic Logical Reasoning and Deep Learning on an Assistive Robot}},
  booktitle = {NMR},
  year = {2022},
  url = {https://openreview.net/forum?id=T4TY2WEftI},
}
Figure