Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach

Shuqi Liu, Han Wu, Guanzhi Deng, Jianshu Chen, Xiaoyang Wang, Linqi Song

Abstract

Large Language Models (LLMs) often struggle with factual consistency and interpretability when generating text from structured data. This paper introduces the Structured Knowledge Hunter (SKH), a task-agnostic module that identifies relevant knowledge triples through a hierarchical "macro-to-micro" selection process. By bridging the gap between raw data and text generation, SKH improves factual accuracy and provides an auditable "knowledge plan." Evaluations on NBA game summaries and multi-domain dialogues demonstrate significant improvements in BLEU scores and content selection metrics over state-of-the-art baselines.

Introduction

Large Language Models (LLMs) are notoriously "black boxes." While they generate fluent text, it is often difficult to determine why a model chose a specific fact or whether that fact is even grounded in the provided data. This lack of interpretability is a dealbreaker in high-stakes fields like finance, medicine, or sports analytics where reliability is paramount.

Existing solutions often rely on domain-specific retrievers that don't generalize well across different types of structured data (e.g., a tool built for NBA box scores might fail on a medical knowledge graph). Furthermore, many models struggle with "factual consistency"—hallucinating details that aren't in the source data. This paper tackles the challenge of creating a task-agnostic, interpretable retriever that works across various structured formats.

Methods

The authors introduce the Structured Knowledge Hunter (SKH), a module designed to "hunt" for the most relevant knowledge triples (e.g., <Brook Lopez, Points, 26>) to guide the LLM.

The architecture of the Structured Knowledge Hunter (SKH), featuring the Entity-Centric Local-Global Encoder and Hierarchical Knowledge Planner.

Key components of the SKH include:

  • Entity-Centric Local-Global Encoder: This module uses a hierarchical transformer to learn representations at two levels: local (details within a single entity/row) and global (relationships between different entities). An Attention Fusion Module then blends these views.
  • Hierarchical Knowledge Planner: Mirroring human cognitive processes, this uses a "macro-to-micro" strategy. It first identifies the important "macro" entities (e.g., which players to talk about) and then delves into the "micro" triples (e.g., which specific stats for those players).
  • Multi-task Learning: The model is trained using a hybrid loss function that optimizes for triple selection, entity selection, and a "matching loss" to ensure the chosen triples actually belong to the chosen entities.
  • Template Conversion: Once the "hunter" picks the best triples, they are converted into readable sentences using simple templates before being fed into a generator like BART or T5.

Datasets

The model was evaluated on two distinct types of knowledge-enhanced generation:

  1. RotoWire-FG: A dataset for generating NBA game summaries from numerical box score tables (internal structured knowledge).
  2. KdConv: A multi-domain Chinese dialogue dataset (film, music, travel) that requires retrieving information from knowledge graphs to answer user queries (external structured knowledge).

Key Results

The proposed HunterAug-BART (BART augmented with the SKH) set new standards on both benchmarks. On the RotoWire-FG dataset, it achieved a BLEU score of 28.69, outperforming the previous state-of-the-art (AuxEncoder) at 24.52.

Performance comparison on the RotoWire-FG dataset, highlighting the superior content selection and relation generation capabilities of HunterAug-BART.

As shown in the performance table, the model's Content Selection F1 score reached 55.78%, a significant jump over vanilla BART (50.12%) and AuxEncoder (49.04%). In the KdConv travel domain, the model saw a massive improvement in BLEU-4 (28.45 vs. 17.96 for the baseline) and Distinct-4 (37.32 vs. 34.35), indicating the responses were both more accurate and more diverse. In blind tests, humans rated the HunterAug-BART summaries significantly higher in "Information Accuracy" (4.03/5.0) compared to vanilla BART (3.37/5.0).

Key Insights and Ideas

  • Human-Readable Memory: Unlike models that store knowledge as abstract vector embeddings, the SKH produces a "knowledge plan" in raw text. This makes the model's "thought process" human-readable and even "human-writable"—an editor could manually update the knowledge base to change the model's output.
  • The Power of Hierarchy: The "macro-to-micro" approach is more effective than flat retrieval. By forcing the model to commit to an entity before picking a fact, the researchers reduced "knowledge tracing" errors and improved factual grounding.
  • Task Agnosticism: By treating tables and knowledge graphs as collections of triples, the SKH achieves high performance on two very different tasks (sports reporting and open-domain chat) without needing task-specific architecture changes.

Limitations and Future Work

The authors acknowledge a few hurdles. The "hunter" struggles when a conversation doesn't explicitly demand specific facts, sometimes resulting in an "empty" knowledge set and less informative responses. Additionally, because the model prioritizes factual grounding so heavily, it can sometimes produce less "creative" or diverse text when generating very long passages. Future directions include exploring how this hierarchical approach could be applied to even larger-scale, multi-modal structured data, such as combining tables with images.

Summary updated

Abstract

Knowledge-enhanced text generation aims to enhance the quality of generated text by utilizing internal or external knowledge sources. While language models have demonstrated impressive capabilities in generating coherent and fluent text, the lack of interpretability presents a substantial obstacle. The limited interpretability of generated text significantly impacts its practical usability, particularly in knowledge-enhanced text generation tasks that necessitate reliability and explainability. Existing methods often employ domain-specific knowledge retrievers that are tailored to specific data characteristics, limiting their generalizability to diverse data types and tasks. To overcome this limitation, we directly leverage the two-tier architecture of structured knowledge, consisting of high-level entities and low-level knowledge triples, to design our task-agnostic structured knowledge hunter. Specifically, we employ a local-global interaction scheme for structured knowledge representation learning and a hierarchical transformer-based pointer network as the backbone for selecting relevant knowledge triples and entities. By combining the strong generative ability of language models with the high faithfulness of the knowledge hunter, our model achieves high interpretability, enabling users to comprehend the model output generation process. Furthermore, we empirically demonstrate the effectiveness of our model in both internal knowledge-enhanced table-to-text generation on the RotoWireFG dataset and external knowledge-enhanced dialogue response generation on the KdConv dataset. Our task-agnostic model outperforms state-of-the-art methods and corresponding language models, setting new standards on the benchmark.
BibTeX
@article{liu2025towards,
  author = {Shuqi Liu and Han Wu and Guanzhi Deng and Jianshu Chen and Xiaoyang Wang and Linqi Song},
  title = {{Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach}},
  journal = {IEEE J. Sel. Top. Signal Process.},
  year = {2025},
}
Figure