Aligning Artificial Neural Networks and Ontologies towards Explainable AI

Manuel de Sousa Ribeiro, João LeiteAAAI 2021 · DOI

Introduction

Deep neural networks (NNs) have achieved state-of-the-art performance across numerous domains, yet they remain notoriously opaque "black boxes." Their internal operations consist of high-dimensional vectors and matrices that lack inherent declarative meaning for humans. While techniques like saliency maps can highlight which pixels influenced a decision, they fail to explain the underlying conceptual reasoning. This paper proposes a framework to bridge the gap between connectionist AI and symbolic AI by mapping internal neural activations to human-defined concepts in an ontology.

Methodology: The Mapping Framework

The authors introduce a multi-step pipeline to generate symbolic, logic-based justifications for a model's decisions without requiring retraining or architectural modifications.

  1. Main Network: A standard Convolutional Neural Network (CNN) is trained on a specific classification task.
  2. Ontology: A formal Description Logic (DL) framework defines the domain, containing high-level concepts (e.g., PassengerCar, LongWagon) and logical axioms relating them to output classes.
  3. Mapping Networks: Tiny, shallow neural networks are trained to predict whether a specific ontological concept is "active" within the main network's internal state. These networks are computationally "cheap" and do not interfere with the original model.
  4. Input Reduce: A custom procedure pinpoints the specific neurons responsible for a concept, reducing noise and improving extraction efficiency. Experiments revealed that high-level concepts are often encoded in a very small subset (roughly 5%) of neurons.
  5. Justification Generation: Automated reasoning identifies the minimal set of ontological axioms that explain why the extracted concepts led to the final classification.

Experimental Setup: The Trains Dataset

To evaluate the framework, the study utilizes the Explainable Abstract Trains Dataset, a synthetic benchmark inspired by the classic Michalski train problems. It consists of 35,000 images of 2D trains with varying wagon shapes, wheel counts, and geometric loads.

Sample train configurations from the Explainable Abstract Trains Dataset, featuring various wagon types and loads used to test logical classification rules.

Trains are classified into types (A, B, or C) based on complex logical rules, such as "a train with a long wagon OR two wagons with at least one circle inside." This provides a ground truth for the logic the mapping networks aim to recover.

Key Results and Insights

The framework demonstrates that mapping networks perform significantly better when the concept is relevant to the main network's primary task; if the model didn't need a concept to solve the problem, it is not easily extractable from its internal state.

Concept extraction accuracy for model $NN_A$. The chart shows high accuracy (often >90%) across most relevant ontological concepts (a-k).

Key findings include:

  • High Extraction Accuracy: Mapping networks achieved high accuracy in identifying relevant concepts using as few as 800 labeled samples.
  • Effective Justifications: The system generated "All Correct" symbolic justifications for 85.5% to 94.2% of test cases across the three main models tested ($NN_A$, $NN_B$, $NN_C$).
  • Plausibility vs. Faithfulness: These justifications provide "plausible" explanations based on human concepts, creating a vital bridge for human-AI communication even if they are not a literal 1:1 trace of mathematical operations.

Conclusion and Future Work

While currently tested in a controlled synthetic environment, this approach offers a promising path toward explainable AI (XAI) in complex, real-world domains where explainability is a legal or ethical requirement. Future research will focus on handling noisier data, addressing the dependency on pre-defined ontologies, and using these justifications to detect learned biases in deep learning models.

Summary updated

BibTeX
@inproceedings{ribeiro2021aligning,
  author = {Manuel de Sousa Ribeiro and João Leite},
  title = {{Aligning Artificial Neural Networks and Ontologies towards Explainable AI}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i6.16626},
}
Figure