Abstract
Large Language Models (LLMs) often function as "black boxes," leading to issues like hallucinations and poor alignment with structured logic. This paper introduces an interpretable read-out layer that reformulates self-attention as a kernel-based operation. By utilizing "support concepts" derived from Knowledge Graphs, the model provides an explicit mechanism for reasoning, significantly improving performance on discriminative tasks while offering transparent explanations for its outputs.
Introduction
Large Language Models have revolutionized natural language processing but remain difficult to interpret and control. Their reliance on implicit gradient-based adjustments often results in "hallucinations" and a failure to adhere to common-sense logic. To address these challenges, researchers propose moving from implicit embedding adjustments to an explicit, observable mechanism for concept modeling.
The Interpretable Read-Out Layer
The core innovation is the Interpretable Read-Out Layer ($\Phi(x)$). By mathematically demonstrating that standard self-attention in Transformers is essentially a Gaussian kernel operation, the authors introduce a non-parametric layer at the model's end. This layer characterizes a concept by comparing it to a set of "support concepts."
Because the read-out layer is the final step, the underlying model ($g_\theta$) can utilize various backbones, including Feed-Forward Neural Networks (FFN), Polynomial Approximations, or Multi-headed Self-Attention (MHA). This flexibility allows the model to maintain performance while gaining transparency.
Integrating Structured Knowledge
To ground the model in factual reality, the researchers use Knowledge Graph (KG) paths (e.g., "Apple has Antioxidants") as support concepts. This process involves tokenizing input data, extracting concepts, and identifying relevant paths within structured graphs like WordNet or ConceptNet.
By aligning internal vector spaces with structured human knowledge, the model moves beyond simple statistical patterns toward logical reasoning. This explicit alignment ensures that the model's understanding of a concept is tied to verifiable facts rather than just word co-occurrence probabilities.
Experimental Evaluation and Results
The models were evaluated on both generative (OpenWebText) and discriminative (GLUE) tasks. Key findings include:
- Performance Gains: Adding KG support concepts led to significant improvements in discriminative tasks. On the Semantic Textual Similarity (STS) task, the Polynomial model with KG support achieved a score of 93.55, nearly matching state-of-the-art leaderboards.
- Generative vs. Discriminative Split: Interestingly, KG integration increased loss in purely generative tasks (next-token prediction) while improving reasoning. This suggests that raw text prediction and logical reasoning require different types of "support."
- Interpretability: The explicit nature of the read-out layer allows for the generation of "heat maps" showing which support concepts (specific KG relations) were most influential in a given prediction.
Conclusion and Future Work
This research provides a path toward more transparent and reliable AI. While scaling to the size of massive models like GPT-4 remains a challenge, the use of interpretable layers and structured support concepts offers a promising alternative to the black-box nature of current LLMs. Future work involves refining generative performance and exploring instruction-tuning datasets as additional sources of support concepts to better align models with human intent.
Summary updated