Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach

Shuqi Liu, Han Wu, Guanzhi Deng, Jianshu Chen, Xiaoyang Wang, Linqi Song

TL;DR

This paper introduces the "Structured Knowledge Hunter," a task-agnostic framework designed to make knowledge-enhanced text generation highly interpretable. By explicitly extracting a human-readable plan of entities and facts before generating text, it pairs a hierarchical retriever with large language models (LLMs) to set new performance standards on table-to-text and dialogue benchmarks while giving users a clear audit trail of the model's generation process.

What Problem Does This Paper Address?

While modern language models excel at generating fluent text, they remain notorious "black boxes." In knowledge-intensive tasks—such as summarizing a sports statistics table or pulling facts from a knowledge graph for a chatbot—this lack of transparency is a major hurdle. If a model hallucinates or makes an error, it is incredibly difficult to trace why it made that decision.

Existing solutions often rely on domain-specific retrievers tailored to a single type of data, which limits their ability to generalize across different tasks. Furthermore, they frequently struggle with factual consistency when dealing with large-scale structured inputs. This paper tackles these limitations by introducing a generalized, task-agnostic architecture capable of handling diverse structured data types (like tables and graphs) while ensuring high interpretability and factual grounding.

Methods

The authors propose a two-stage pipeline that decouples knowledge selection from text generation. Instead of feeding raw structured data directly into an LLM, they route it through a Structured Knowledge Hunter to extract a precise factual plan.

The architecture relies on a few key innovations:

  1. Entity-Centric Local-Global Encoder: It processes structured inputs from two distinct perspectives: a local intra-entity view (focusing on attributes within a single entity or table row) and a global inter-entity view (capturing relationships across different entities).
  2. Local-Global Attention Fusion: A gated mechanism using a Sigmoid function dynamically weights and merges these local and global representations.
  3. Hierarchical Knowledge Planner: Modeled as a transformer-based pointer network, it mimics human "macro-to-micro" planning by first selecting high-level "macro" entities and then diving down to select specific low-level "micro" knowledge triples.
  4. Multi-task Learning Optimization: The retriever is jointly optimized using three losses: triple selection loss, entity selection loss, and a novel matching loss that ensures selected triples strictly align with their parent entities.

Once the human-readable knowledge plan is extracted, it is converted into simple text templates and fed into a fine-tuned LLM (such as BART or T5) to generate the final output.

Fig. 2. Overall pipeline paradigm of steering language model-based text generator with a Structured Knowledge Hunter. The Knowledge Hunter is designed as a hierarchical encoder-decoder system that searches for relevant knowledge triples from input knowledge graphs or numerical tables. These knowledge triples are then processed through a Template Conversion stage that transforms them into a format that can be readily fed into the language model. Finally, the language model generates text based on the processed knowledge triples.

Datasets

The framework was evaluated on two distinct tasks using standard benchmarks:

  • RotoWire-FG: A dataset for internal knowledge-enhanced table-to-text generation. Models must generate long, multi-paragraph NBA game summaries (averaging ~206 words) based on complex box-score and line-score tables containing hundreds of numerical records.
  • KdConv: A multi-domain Chinese dataset for external knowledge-graph-driven multi-turn dialogue. It spans film, music, and travel domains, featuring complex topic transitions and rich external knowledge graphs.

Key Insights and Ideas

  • Human-Readable and Human-Writable Memory: Instead of passing abstract vector embeddings to the text generator, the framework extracts explicit, raw-text knowledge triples. This makes the model's "thought process" fully auditable by humans and allows developers to dynamically update the model's knowledge base simply by editing the underlying text.
  • The Power of "Macro-to-Micro" Planning: Mirroring human cognitive behavior—where we first identify the main subjects of interest before analyzing specific details—significantly boosts retrieval accuracy. The introduction of the matching loss ensures that the low-level facts selected actually belong to the high-level entities being discussed.
  • Task-Agnostic Flexibility: By focusing on the universal two-tier hierarchy of structured data (Entities $\rightarrow$ Triples), the exact same architecture seamlessly adapts to both static numerical tables and dynamic relational knowledge graphs.

Key Results

The framework was evaluated on both knowledge selection accuracy and text generation quality, outperforming state-of-the-art baselines.

  • Table-to-Text (RotoWire-FG): HunterAug-BART achieved a Content Selection F1 score of 55.78%, a substantial improvement over the previous SOTA baseline AuxEncoder (49.04%) and vanilla BART (50.12%). It also achieved the highest text quality with a BLEU score of 28.69 (compared to vanilla BART's 24.51).
  • Multi-turn Dialogue (KdConv): HunterAug-BART showed massive improvements over the competitive KHRED baseline, raising the BLEU-4 score by 10.49 points in the travel domain, 5.00 in music, and 3.10 in film.

Ablation studies confirmed that the entity-aware design, local-global fusion layers, and multi-task loss combinations all progressively contributed to these performance gains.

Fig. 1. Generation examples of Structured Knowledge Hunter for numerical table summarization (top) and knowledge graph augmented dialogue (down).

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Handling Implicit Contexts: The Structured Knowledge Hunter can struggle when a dialogue context doesn't explicitly mention specific entities or relationships, occasionally leading to an empty extracted knowledge set and less informative responses.
  • Trade-off Between Factualness and Diversity: Because the model heavily prioritizes factual grounding and strictly adheres to the extracted knowledge plan, it can suffer from reduced text diversity, particularly when generating longer narratives.

For future work, the authors suggest exploring better ways to handle incomplete or noisy structured data and investigating methods to boost stylistic diversity without sacrificing factual accuracy.

Summary updated

Abstract

Knowledge-enhanced text generation aims to enhance the quality of generated text by utilizing internal or external knowledge sources. While language models have demonstrated impressive capabilities in generating coherent and fluent text, the lack of interpretability presents a substantial obstacle. The limited interpretability of generated text significantly impacts its practical usability, particularly in knowledge-enhanced text generation tasks that necessitate reliability and explainability. Existing methods often employ domain-specific knowledge retrievers that are tailored to specific data characteristics, limiting their generalizability to diverse data types and tasks. To overcome this limitation, we directly leverage the two-tier architecture of structured knowledge, consisting of high-level entities and low-level knowledge triples, to design our task-agnostic structured knowledge hunter. Specifically, we employ a local-global interaction scheme for structured knowledge representation learning and a hierarchical transformer-based pointer network as the backbone for selecting relevant knowledge triples and entities. By combining the strong generative ability of language models with the high faithfulness of the knowledge hunter, our model achieves high interpretability, enabling users to comprehend the model output generation process. Furthermore, we empirically demonstrate the effectiveness of our model in both internal knowledge-enhanced table-to-text generation on the RotoWireFG dataset and external knowledge-enhanced dialogue response generation on the KdConv dataset. Our task-agnostic model outperforms state-of-the-art methods and corresponding language models, setting new standards on the benchmark.
BibTeX
@article{liu2025towards,
  author = {Shuqi Liu and Han Wu and Guanzhi Deng and Jianshu Chen and Xiaoyang Wang and Linqi Song},
  title = {{Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach}},
  journal = {IEEE J. Sel. Top. Signal Process.},
  year = {2025},
}
Figure