Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models

Philipp Mondorf, Barbara Plank

Introduction

While Large Language Models (LLMs) have shown remarkable progress in standard deductive reasoning tasks, they often falter when faced with scenarios requiring suppositional reasoning. This form of reasoning involves entertaining a hypothesis, exploring its logical consequences, and checking for contradictions—a process essential for real-world problem-solving where premises are not always guaranteed to be true. To address this gap, researchers have introduced TruthQuest, a benchmark centered on the classic "Knights and Knaves" logic puzzles.

The TruthQuest Benchmark

The TruthQuest dataset consists of 2,400 unique logic puzzles designed to test an agent's ability to navigate a "logical mire." In these puzzles, characters are either Knights (who always tell the truth) or Knaves (who always lie). Solving these requires multi-step conditional logic: "If Person A is a Knight, then their statement is true; does this lead to a contradiction?"

An example of a Knights and Knaves puzzle from the TruthQuest benchmark, detailing the rules and character statements.

The dataset is categorized by two primary complexity axes:

  1. Character Count: Puzzles scale from 3 to 6 characters.
  2. Statement Complexity: Puzzles are divided into Simple (S), Implication (I), and Equivalence (E) sets, utilizing various logical operators like conjunctions and "if-then" statements.

Experimental Methodology

The study evaluated several open-access model families, including Llama 2, Llama 3, and Mixtral. Researchers employed various prompting strategies, such as Zero-shot, Few-shot, and Chain-of-Thought (CoT). To ensure rigorous evaluation, a two-step framework was used: initial regular expression parsing followed by a secondary "evaluator" model (Llama 3-8B) to standardize outputs for exact-match accuracy checking.

Key Results and Scaling Challenges

The findings highlight a significant performance gap in current LLMs:

  • Baseline Performance: Under zero-shot conditions, most models performed near random chance.
  • Llama 3-70B Superiority: Llama 3-70B emerged as the top performer, achieving 60% accuracy on the simplest 3-character puzzles when using 8-shot CoT prompting.
  • The Complexity Wall: Performance dropped precipitously as complexity increased. Accuracy for the best models fell to 10-20% when faced with 6-character puzzles or complex logical operators like Equivalence.
  • Terminology Independence: To rule out memorization, researchers replaced "Knights and Knaves" with nonsense terms. Performance remained stable, suggesting models are attempting genuine logical reasoning rather than recalling training data.

Analysis of Error Types: The Proficiency Shift

A critical insight from the study is the "proficiency shift" in errors. Lower-performing models (e.g., Llama 2-7B) often fail to grasp basic rules, such as the fact that a Knave must lie. In contrast, higher-performing models (e.g., Llama 3-70B) understand the rules but struggle with the logical implications of a lie. For instance, if a Knave asserts a conjunction ("A and B are Knights"), a sophisticated model might fail to realize this only implies that at least one of them is a Knave, not necessarily both.

Conclusion

TruthQuest demonstrates that while Chain-of-Thought prompting is beneficial, it does not fully resolve the scaling issues inherent in suppositional reasoning. As puzzles grow in complexity, LLMs lose the logical thread. Future research may need to explore more advanced architectures or prompting techniques, such as Tree-of-Thoughts, to better navigate the branching paths of complex logical deduction.

Summary updated

Abstract

Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie. The objective is to logically deduce each character's identity based on their statements. The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement. Solving these puzzles requires not only direct deductions from individual statements, but the ability to assess the truthfulness of statements by reasoning through various hypothetical scenarios. As such, knights and knaves puzzles serve as compelling examples of suppositional reasoning. In this paper, we introduce $\textit{TruthQuest}$, a benchmark for suppositional reasoning based on the principles of knights and knaves puzzles. Our benchmark presents problems of varying complexity, considering both the number of characters and the types of logical statements involved. Evaluations on $\textit{TruthQuest}$ show that large language models like Llama 3 and Mixtral-8x7B exhibit significant difficulties solving these tasks. A detailed error analysis of the models' output reveals that lower-performing models exhibit a diverse range of reasoning errors, frequently failing to grasp the concept of truth and lies. In comparison, more proficient models primarily struggle with accurately inferring the logical implications of potentially false statements.
BibTeX
@inproceedings{mondorf2024liar,
  author = {Philipp Mondorf and Barbara Plank},
  title = {{Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.404},
}
Figure