Hybrid Machine Learning/Knowledge Base Systems Learning through Natural Language Dialogue with Deep Learning Models

Sergei Nirenburg, Nikhil Krishnaswamy, Marjorie McShaneAAAI 2023

This paper introduces a neurosymbolic approach to developing "Language-Endowed Intelligent Agents" (LEIAs) capable of autonomous knowledge acquisition through dialogue. By utilizing Large Language Models (LLMs) as interactive partners, the researchers demonstrate a method for automating the construction of structured knowledge bases, effectively bridging the gap between the statistical pattern recognition of deep learning and the logical reasoning of symbolic AI.

Addressing the AI Knowledge Bottleneck

Symbolic AI systems—those that reason using logic and structured data—traditionally suffer from a "knowledge bottleneck." They require massive, hand-crafted ontologies and lexicons that are expensive and slow to build. Conversely, modern Deep Learning (DL) models are flexible but often lack true world understanding or explainability. This research proposes an "inverse neurosymbolic" integration: using the broad knowledge of DL models to automate the population of the structured knowledge required for symbolic reasoning.

System Architecture and Routine Operation

The core of the LEIA is the OntoSem (Ontological Semantics) architecture. In routine operation, the agent processes language and visual inputs using three primary resources: a Lexicon (word senses), an Ontology (world concepts), and an Opticon (visual-to-conceptual pairings).

Figure 1: The LEIA architecture for routine operation, illustrating how knowledge resources support interpretation and reasoning.

The Learning Loop: From Lacunae to Knowledge

Learning is triggered opportunistically when the agent encounters a "lacuna"—a word or concept missing from its internal resources. Instead of failing, the agent enters a learning mode characterized by several key steps:

  1. DL-Assisted Data Collection: The agent prompts an LLM (such as ChatGPT) to generate descriptive sentences about the unknown concept.
  2. Semantic Grounding: The agent extracts Text Meaning Representations (TMRs) from these sentences to build "skeleton" entries in its lexicon and ontology.
  3. The Inheritance Windfall: By situating a new concept within the existing ontological hierarchy (e.g., identifying "systemic sclerosis" as a type of "autoimmune disease"), the agent automatically inherits hundreds of properties from its parent concepts.
Figure 2: The learning configuration where perception failures trigger the collection of new data to modify the agent's knowledge.

Cross-Modal Mapping and Physical Grounding

For physical objects, the system uses BERT embeddings of descriptive text and maps them to physical feature vectors in the VoxWorld simulator using linear ridge regression. This allows the agent to link visual perception to ontological understanding via the Opticon. Experiments with geometric shapes showed that transformed word embeddings for objects like "blocks" successfully clustered with their physical counterparts in the simulator, while irrelevant word senses were correctly excluded.

Conclusion and Future Directions

The authors demonstrate a successful proof-of-concept for learning both complex medical terminology and physical object properties. While challenges such as LLM "hallucinations" remain, requiring some human validation, the system points toward a future where LEIAs and LLMs work in a bi-directional cycle to improve both structured reasoning and statistical breadth.

Summary updated

BibTeX
@inproceedings{nirenburg2023hybrid,
  author = {Sergei Nirenburg and Nikhil Krishnaswamy and Marjorie McShane},
  title = {{Hybrid Machine Learning/Knowledge Base Systems Learning through Natural Language Dialogue with Deep Learning Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
}
Figure