CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models

Xiaqiang Tang, Jian Li, Keyu Hu, Du Nan, Xiaolong Li, Xi Zhang, Weigao Sun, Sihong Xie

Introduction

As Large Language Models (LLMs) are increasingly integrated into high-stakes sectors such as medicine, law, and finance, the requirement for "faithfulness"—the degree to which a model's output is supported by its input context—has become paramount. However, current benchmarks primarily focus on factual faithfulness, evaluating whether a model can accurately summarize or retrieve explicit information. This leaves a critical gap: the assessment of "cognitive faithfulness."

Cognitive faithfulness refers to an LLM's ability to make logical inferences, provide explanations, and evaluate situations based on provided evidence. Unlike factual statements, cognitive statements are inherently subjective and involve reasoning leaps. The CogniBench framework addresses this by introducing a legal-inspired taxonomy and a robust dataset to measure how well models handle these complex cognitive tasks.

The Problem: The "Inference Tax"

Existing tools for detecting hallucinations often rely on simple textual overlap, which is insufficient for evaluating reasoning. When an AI agent moves beyond parroting facts to drawing conclusions—such as a medical AI assessing infection risk based on symptoms—the line between a "faithful inference" and a "hallucination" becomes blurred.

Research using the CogniBench framework reveals a significant "Inference Tax": LLMs are 4.6 times more likely to hallucinate when making cognitive leaps than when restating facts. As a conversation progresses, the proportion of cognitive statements typically grows from 15% to 50%, meaning the reliability of the model plummets exactly when the user needs its reasoning capabilities most.

The CogniBench Framework

To standardize the evaluation of subjective inferences, the authors borrow concepts from legal proceedings, treating the LLM's context as "evidence" and its response as "testimony." They categorize cognitive faithfulness into three increasingly rigorous tiers:

  1. Rational: The statement is a plausible speculation, even if not strictly proven by the text.
  2. Grounded: The statement is logically supported by the context or indirect evidence.
  3. Unequivocal: The statement is the only reasonable conclusion that can be drawn, free from subjective bias.

To ensure high-quality data, the researchers developed a sequential decision-making protocol for human annotators. By moving through these tiers step-by-step, the team achieved an inter-annotator agreement rate of over 96%, significantly reducing the subjectivity typically associated with reasoning benchmarks.

Dataset Construction and CogniDet

The study introduces two primary datasets:

  • CogniBench: A high-quality, human-annotated set of 179 multi-turn, knowledge-grounded conversations (2,516 labeled sentences).
  • CogniBench-L: A large-scale extension containing over 24,000 dialogues with sentence-level hallucination annotations.

To generate CogniBench-L, the authors utilized an automated pipeline powered by Contrastive and Formative Prompting (CFP). This method provides an evaluator LLM (such as GPT-4) with specific examples of common reasoning errors, allowing it to distinguish between valid inferences and hallucinations more effectively than standard prompting.

Using this data, the researchers fine-tuned an 8B Llama-3 model, dubbed CogniDet. This specialized detector achieved a 70.3% F1 score, significantly outperforming general-purpose hallucination detectors and NLI-based baselines, which often see their performance drop by over 30% when faced with cognitive rather than factual statements.

Key Findings

The evaluation of state-of-the-art models under the CogniBench framework highlights a sobering reality regarding current AI reasoning:

  • High Hallucination Rates: Even top-tier models struggle with cognitive faithfulness. GPT-4 exhibited a 60.1% hallucination rate on cognitive statements, while Gemini-Pro reached a 79.9% error rate.
  • The Subjectivity Trap: Many statements previously dismissed as "baseless" by other benchmarks were found to be "Reliable" under CogniBench because they followed a rational, albeit non-verbatim, path from the source text.
  • Scaling Potential: The researchers observed a log-linear relationship between training data size and detector performance, suggesting that further expansion of the CogniBench-L dataset could lead to even more robust detectors.

Limitations and Future Directions

While CogniBench provides a rigorous foundation, it currently relies on Wikipedia-based data. High-stakes domains like medicine or law will eventually require domain-specific experts to refine these taxonomies further. Additionally, because legal standards vary by jurisdiction, the framework serves as a systematic inspiration for reasoning rather than a universal normative standard.

Future work will focus on extending the protocol to specialized domains and exploring real-time implementation of cognitive detectors to prevent LLMs from delivering ungrounded or dangerous advice during live interactions.

Summary updated

BibTeX
@inproceedings{tang2025cognibench,
  author = {Xiaqiang Tang and Jian Li and Keyu Hu and Du Nan and Xiaolong Li and Xi Zhang and Weigao Sun and Sihong Xie},
  title = {{CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure