Learning Visually Grounded Domain Ontologies via Embodied Conversation and Explanation

Jonghyuk Park, Alex Lascarides, Subramanian RamamoorthyAAAI 2025 · DOI

Abstract

Traditional Explainable AI (XAI) systems often function as "read-only" interfaces, providing justifications for decisions without a mechanism to incorporate human feedback into the learning process. This paper introduces a neurosymbolic framework designed for Explanatory Interactive Learning (XIL). By engaging in natural language dialogue and asking "why" an agent made a specific prediction, human teachers can provide targeted corrections to both logical rules and visual recognition. This approach significantly improves data efficiency in low-resource, novel environments.

Introduction

Deep learning models typically require vast amounts of data and struggle to adapt to niche domains where pre-trained datasets are unavailable. Furthermore, when these models fail, it is often unclear whether the error stems from faulty visual perception or incorrect logical reasoning. To address this, the authors propose an agent capable of building a domain ontology from scratch through embodied conversation.

Neurosymbolic Architecture

The proposed system utilizes a modular architecture to bridge the gap between raw visual data and symbolic logic:

  1. Vision Processing Module: Employs few-shot learning to segment and classify object parts (e.g., ladders, dumpers) based on current knowledge.
  2. Language Processing Module: Translates natural language into formal logic for reasoning.
  3. Long-term Memory: Maintains a visual exemplar base and a symbolic knowledge base of rules (e.g., "Dump trucks have dumpers").
  4. Symbolic Reasoning Module: Uses Probabilistic Factor Graphs and abductive reasoning to connect visual features to object categories.

The Explanatory Dialogue Loop

The core of the framework is the interaction strategy. When an agent misclassifies an object, the teacher probes the agent's reasoning. This allows the teacher to identify if the error is a "deictic" visual error (misidentifying a part) or a "generic" logical error (having the wrong rule for a category).

The agent explains its reasoning by pointing to a specific part, allowing the human to provide a targeted correction.

By explaining which part it believes justifies its classification, the agent enables the teacher to fix underlying visual recognition errors that would otherwise remain hidden. This "Aha!" moment for vision allows the agent to filter out irrelevant features and focus on the parts that define a category.

Experimental Evaluation

The framework was tested using 3D models of toy trucks in two tasks:

  • single_4way: Trucks distinguishable by one unique part (e.g., a ladder vs. a rocket).
  • double_5way: Trucks requiring identification of two dimensions (e.g., cabin type and load type).
An example of a toy truck from the 3D simulated environment used for training and evaluation.

The agent began with zero prior knowledge of these categories, simulating a low-resource environment where pre-trained models are ineffective.

Key Results and Insights

The study measured performance via cumulative regret over 120 training episodes. The full explanatory approach (Vis+Genr+Expl) consistently outperformed all baselines.

  • Superior Data Efficiency: The integration of generic rules acts as a visual filter, directing the agent's attention to relevant features and drastically reducing the number of examples needed to learn a concept. In the "single_4way" task, the performance gap was statistically significant ($p < 10^{-6}$).
  • The Necessity of Explanation: In complex tasks, providing rules without explanations sometimes hindered performance. Without the "why" step, agents might confidently misidentify a part and then use a correct rule to reach a wrong conclusion. Explanations mitigate this "overconfidence" by exposing the perception error.
  • Robustness: The system proved resilient even with poor initial vision models (the "LQ" setting), as the interactive process allowed for rapid recovery through human-in-the-loop corrections.

Conclusion and Future Work

This research demonstrates that closing the loop between explanation and learning allows AI agents to acquire complex domain knowledge efficiently. Future work includes expanding the ontology to handle spatial relations and functional properties, as well as testing the system in real-world robotic assembly environments where exceptions (e.g., "Birds fly, but penguins don't") may be necessary.

Summary updated

BibTeX
@inproceedings{park2025learning,
  author = {Jonghyuk Park and Alex Lascarides and Subramanian Ramamoorthy},
  title = {{Learning Visually Grounded Domain Ontologies via Embodied Conversation and Explanation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i13.33573},
}
Figure