Introduction
While Large Language Models (LLMs) have shown remarkable progress in standard deductive reasoning tasks, they often falter when faced with scenarios requiring suppositional reasoning. This form of reasoning involves entertaining a hypothesis, exploring its logical consequences, and checking for contradictions—a process essential for real-world problem-solving where premises are not always guaranteed to be true. To address this gap, researchers have introduced TruthQuest, a benchmark centered on the classic "Knights and Knaves" logic puzzles.
The TruthQuest Benchmark
The TruthQuest dataset consists of 2,400 unique logic puzzles designed to test an agent's ability to navigate a "logical mire." In these puzzles, characters are either Knights (who always tell the truth) or Knaves (who always lie). Solving these requires multi-step conditional logic: "If Person A is a Knight, then their statement is true; does this lead to a contradiction?"
The dataset is categorized by two primary complexity axes:
- Character Count: Puzzles scale from 3 to 6 characters.
- Statement Complexity: Puzzles are divided into Simple (S), Implication (I), and Equivalence (E) sets, utilizing various logical operators like conjunctions and "if-then" statements.
Experimental Methodology
The study evaluated several open-access model families, including Llama 2, Llama 3, and Mixtral. Researchers employed various prompting strategies, such as Zero-shot, Few-shot, and Chain-of-Thought (CoT). To ensure rigorous evaluation, a two-step framework was used: initial regular expression parsing followed by a secondary "evaluator" model (Llama 3-8B) to standardize outputs for exact-match accuracy checking.
Key Results and Scaling Challenges
The findings highlight a significant performance gap in current LLMs:
- Baseline Performance: Under zero-shot conditions, most models performed near random chance.
- Llama 3-70B Superiority: Llama 3-70B emerged as the top performer, achieving 60% accuracy on the simplest 3-character puzzles when using 8-shot CoT prompting.
- The Complexity Wall: Performance dropped precipitously as complexity increased. Accuracy for the best models fell to 10-20% when faced with 6-character puzzles or complex logical operators like Equivalence.
- Terminology Independence: To rule out memorization, researchers replaced "Knights and Knaves" with nonsense terms. Performance remained stable, suggesting models are attempting genuine logical reasoning rather than recalling training data.
Analysis of Error Types: The Proficiency Shift
A critical insight from the study is the "proficiency shift" in errors. Lower-performing models (e.g., Llama 2-7B) often fail to grasp basic rules, such as the fact that a Knave must lie. In contrast, higher-performing models (e.g., Llama 3-70B) understand the rules but struggle with the logical implications of a lie. For instance, if a Knave asserts a conjunction ("A and B are Knights"), a sophisticated model might fail to realize this only implies that at least one of them is a Knave, not necessarily both.
Conclusion
TruthQuest demonstrates that while Chain-of-Thought prompting is beneficial, it does not fully resolve the scaling issues inherent in suppositional reasoning. As puzzles grow in complexity, LLMs lose the logical thread. Future research may need to explore more advanced architectures or prompting techniques, such as Tree-of-Thoughts, to better navigate the branching paths of complex logical deduction.
Summary updated