Exploring Alternative Approaches to Language Modeling for Learning from Data and Knowledge

Yuxin Zi, Kaushik Roy, Vignesh Narayanan, Amit P. ShethAAAI 2024 · DOI

Abstract

Large Language Models (LLMs) often function as "black boxes," leading to issues like hallucinations and poor alignment with structured logic. This paper introduces an interpretable read-out layer that reformulates self-attention as a kernel-based operation. By utilizing "support concepts" derived from Knowledge Graphs, the model provides an explicit mechanism for reasoning, significantly improving performance on discriminative tasks while offering transparent explanations for its outputs.

Introduction

Large Language Models have revolutionized natural language processing but remain difficult to interpret and control. Their reliance on implicit gradient-based adjustments often results in "hallucinations" and a failure to adhere to common-sense logic. To address these challenges, researchers propose moving from implicit embedding adjustments to an explicit, observable mechanism for concept modeling.

The Interpretable Read-Out Layer

The core innovation is the Interpretable Read-Out Layer ($\Phi(x)$). By mathematically demonstrating that standard self-attention in Transformers is essentially a Gaussian kernel operation, the authors introduce a non-parametric layer at the model's end. This layer characterizes a concept by comparing it to a set of "support concepts."

Architecture of the interpretable read-out layer and its underlying parametric function approximations.

Because the read-out layer is the final step, the underlying model ($g_\theta$) can utilize various backbones, including Feed-Forward Neural Networks (FFN), Polynomial Approximations, or Multi-headed Self-Attention (MHA). This flexibility allows the model to maintain performance while gaining transparency.

Integrating Structured Knowledge

To ground the model in factual reality, the researchers use Knowledge Graph (KG) paths (e.g., "Apple has Antioxidants") as support concepts. This process involves tokenizing input data, extracting concepts, and identifying relevant paths within structured graphs like WordNet or ConceptNet.

The pipeline for extracting support concepts from text and integrating them with Knowledge Graph embeddings.

By aligning internal vector spaces with structured human knowledge, the model moves beyond simple statistical patterns toward logical reasoning. This explicit alignment ensures that the model's understanding of a concept is tied to verifiable facts rather than just word co-occurrence probabilities.

Experimental Evaluation and Results

The models were evaluated on both generative (OpenWebText) and discriminative (GLUE) tasks. Key findings include:

  • Performance Gains: Adding KG support concepts led to significant improvements in discriminative tasks. On the Semantic Textual Similarity (STS) task, the Polynomial model with KG support achieved a score of 93.55, nearly matching state-of-the-art leaderboards.
  • Generative vs. Discriminative Split: Interestingly, KG integration increased loss in purely generative tasks (next-token prediction) while improving reasoning. This suggests that raw text prediction and logical reasoning require different types of "support."
  • Interpretability: The explicit nature of the read-out layer allows for the generation of "heat maps" showing which support concepts (specific KG relations) were most influential in a given prediction.

Conclusion and Future Work

This research provides a path toward more transparent and reliable AI. While scaling to the size of massive models like GPT-4 remains a challenge, the use of interpretable layers and structured support concepts offers a promising alternative to the black-box nature of current LLMs. Future work involves refining generative performance and exploring instruction-tuning datasets as additional sources of support concepts to better align models with human intent.

Summary updated

BibTeX
@inproceedings{zi2024exploring,
  author = {Yuxin Zi and Kaushik Roy and Vignesh Narayanan and Amit P. Sheth},
  title = {{Exploring Alternative Approaches to Language Modeling for Learning from Data and Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v3i1.31211},
}
Figure