Abstract
Large Language Models (LLMs) often struggle with factual consistency and interpretability when generating text from structured data. This paper introduces the Structured Knowledge Hunter (SKH), a task-agnostic module that identifies relevant knowledge triples through a hierarchical "macro-to-micro" selection process. By bridging the gap between raw data and text generation, SKH improves factual accuracy and provides an auditable "knowledge plan." Evaluations on NBA game summaries and multi-domain dialogues demonstrate significant improvements in BLEU scores and content selection metrics over state-of-the-art baselines.
Introduction
Large Language Models (LLMs) are notoriously "black boxes." While they generate fluent text, it is often difficult to determine why a model chose a specific fact or whether that fact is even grounded in the provided data. This lack of interpretability is a dealbreaker in high-stakes fields like finance, medicine, or sports analytics where reliability is paramount.
Existing solutions often rely on domain-specific retrievers that don't generalize well across different types of structured data (e.g., a tool built for NBA box scores might fail on a medical knowledge graph). Furthermore, many models struggle with "factual consistency"—hallucinating details that aren't in the source data. This paper tackles the challenge of creating a task-agnostic, interpretable retriever that works across various structured formats.
Methods
The authors introduce the Structured Knowledge Hunter (SKH), a module designed to "hunt" for the most relevant knowledge triples (e.g., <Brook Lopez, Points, 26>) to guide the LLM.
Key components of the SKH include:
- Entity-Centric Local-Global Encoder: This module uses a hierarchical transformer to learn representations at two levels: local (details within a single entity/row) and global (relationships between different entities). An Attention Fusion Module then blends these views.
- Hierarchical Knowledge Planner: Mirroring human cognitive processes, this uses a "macro-to-micro" strategy. It first identifies the important "macro" entities (e.g., which players to talk about) and then delves into the "micro" triples (e.g., which specific stats for those players).
- Multi-task Learning: The model is trained using a hybrid loss function that optimizes for triple selection, entity selection, and a "matching loss" to ensure the chosen triples actually belong to the chosen entities.
- Template Conversion: Once the "hunter" picks the best triples, they are converted into readable sentences using simple templates before being fed into a generator like BART or T5.
Datasets
The model was evaluated on two distinct types of knowledge-enhanced generation:
- RotoWire-FG: A dataset for generating NBA game summaries from numerical box score tables (internal structured knowledge).
- KdConv: A multi-domain Chinese dialogue dataset (film, music, travel) that requires retrieving information from knowledge graphs to answer user queries (external structured knowledge).
Key Results
The proposed HunterAug-BART (BART augmented with the SKH) set new standards on both benchmarks. On the RotoWire-FG dataset, it achieved a BLEU score of 28.69, outperforming the previous state-of-the-art (AuxEncoder) at 24.52.
As shown in the performance table, the model's Content Selection F1 score reached 55.78%, a significant jump over vanilla BART (50.12%) and AuxEncoder (49.04%). In the KdConv travel domain, the model saw a massive improvement in BLEU-4 (28.45 vs. 17.96 for the baseline) and Distinct-4 (37.32 vs. 34.35), indicating the responses were both more accurate and more diverse. In blind tests, humans rated the HunterAug-BART summaries significantly higher in "Information Accuracy" (4.03/5.0) compared to vanilla BART (3.37/5.0).
Key Insights and Ideas
- Human-Readable Memory: Unlike models that store knowledge as abstract vector embeddings, the SKH produces a "knowledge plan" in raw text. This makes the model's "thought process" human-readable and even "human-writable"—an editor could manually update the knowledge base to change the model's output.
- The Power of Hierarchy: The "macro-to-micro" approach is more effective than flat retrieval. By forcing the model to commit to an entity before picking a fact, the researchers reduced "knowledge tracing" errors and improved factual grounding.
- Task Agnosticism: By treating tables and knowledge graphs as collections of triples, the SKH achieves high performance on two very different tasks (sports reporting and open-domain chat) without needing task-specific architecture changes.
Limitations and Future Work
The authors acknowledge a few hurdles. The "hunter" struggles when a conversation doesn't explicitly demand specific facts, sometimes resulting in an "empty" knowledge set and less informative responses. Additionally, because the model prioritizes factual grounding so heavily, it can sometimes produce less "creative" or diverse text when generating very long passages. Future directions include exploring how this hierarchical approach could be applied to even larger-scale, multi-modal structured data, such as combining tables with images.
Summary updated