Knowledge-based Consistency Testing of Large Language Models

Sai Sathiesh Rajan, Ezekiel Olamide Soremekun, Sudipta ChattopadhyayEMNLP 2024 · DOI

Abstract

Large Language Models (LLMs) often exhibit logical inconsistencies, providing contradictory answers to semantically equivalent prompts. This paper introduces KONTEST, a framework that leverages external Knowledge Graphs to systematically identify these "knowledge gaps." By generating atomic, sequential, and ontological test cases, KONTEST exposes the fragility of LLM reasoning. Furthermore, the authors propose a weighted ensemble method that significantly improves consistency and reduces the gap between model beliefs and ground-truth facts.

Introduction

While Large Language Models (LLMs) have demonstrated remarkable capabilities, their reliability remains a significant hurdle for deployment in critical fields such as navigation, medicine, or legal technology. A particularly pervasive failure mode is inconsistency: a model may correctly state a fact in one context but deny it when the question is phrased differently or asked within a longer conversation. This "knowledge gap"—the discrepancy between real-world truth and a model's internal representation—highlights that LLMs often rely on pattern matching rather than robust logical reasoning.

The KONTEST Framework

To address these challenges, the authors developed KONTEST (Knowledge-based CONsistency TESTing), an automated pipeline designed to measure and mitigate LLM inconsistencies.

The KONTEST framework architecture, illustrating the pipeline from Knowledge Graph construction to test generation and oracle evaluation.

The framework operates in three primary stages:

  1. Knowledge Graph Construction: KONTEST extracts ground-truth entities and relationships from structured databases like Wikidata (e.g., the hierarchy Kinawley → Ulster → Ireland).
  2. Test Generation: These relationships are transformed into semantically equivalent "Yes/No" queries using predefined templates.
  3. Test Oracles: The system employs two types of logical "judges":
  4. Metamorphic Oracle: Ensures the model provides identical answers to different phrasings of the same fact.
  5. Ontological Oracle: Validates logical transitivity. For instance, if a model agrees that "A is in B" and "B is in C," the oracle checks if it also agrees that "A is in C."

Experimental Methodology

The study evaluated four prominent LLMs—Falcon-7B, Gemini-Pro, GPT-3.5-Turbo, and Llama-2-7B—across two domains: Places (spatial/administrative data) and Music (artist/album relationships). Testing was conducted in two modes: Atomic, where queries are isolated in fresh conversations, and Sequential, where queries are asked consecutively to observe how previous context influences subsequent answers.

Examples of atomic, sequential, and ontological queries used to identify inconsistencies and construct the LLM's perceived knowledge graph.

Key Insights and Results

The application of KONTEST revealed several critical weaknesses in current LLMs:

  • High Inconsistency Rates: Approximately 19.2% of valid test executions resulted in consistency errors. The average "knowledge gap" across all models was measured at 16.5%.
  • The Context Trap: Models frequently failed "sequential-inter" tests. Interestingly, providing a correct answer to an initial question often made the model more likely to contradict itself in a related follow-up question within the same session.
  • Size vs. Reliability: Model scale does not guarantee consistency. GPT-3.5-Turbo exhibited a significantly higher knowledge gap (29.6%) in the spatial domain compared to the much smaller Falcon-7B (3.1%).
  • LLMs are not Knowledge Bases: In an ablation study, GPT-3.5 was used to construct the knowledge graph itself but only recovered 68% of the ground-truth relations, underscoring that LLMs remain poor substitutes for structured databases.
  • Template Sensitivity: LLMs are highly sensitive to phrasing. A model's knowledge gap could fluctuate from 12.9% to 47.3% based solely on slight variations in the question template.

Mitigation via Weighted Ensembles

To improve reliability, the authors proposed a Weighted Model Ensemble. By identifying which models are most consistent on specific types of facts during the testing phase, the system assigns higher weights to their outputs. This approach reduced the knowledge gap by 32.48%, outperforming standard majority voting and providing a more stable "consensus" answer.

Conclusion and Future Work

KONTEST provides a systematic method for auditing the factual consistency of LLMs. While currently limited to binary queries and dependent on the completeness of external Knowledge Graphs, it offers a path toward more reliable AI. Future research will focus on expanding the framework to handle open-ended queries and exploring how these discovered inconsistencies can be used to "repair" models through targeted fine-tuning.

Summary updated

BibTeX
@inproceedings{rajan2024knowledge,
  author = {Sai Sathiesh Rajan and Ezekiel Olamide Soremekun and Sudipta Chattopadhyay},
  title = {{Knowledge-based Consistency Testing of Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.596},
}
Figure