Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting

Rylan Schaeffer, Kateryna Pistunova, Samar Khanna, Sarthak Consul, Sanmi KoyejoarXiv 2023 · OpenReview

Abstract

Language models can be prompted to reason through problems in a manner that significantly improves performance. However, why such prompting improves performance is unclear. Recent work showed that using logically invalid Chain-of-Thought (CoT) prompting improves performance almost as much as logically valid CoT prompting, and that editing CoT prompts to replace problem-specific information with abstract information or out-of-distribution information typically doesn't harm performance. Critics have responded that these findings are based on too few and too easily solved tasks to draw meaningful conclusions. To resolve this dispute, we test whether logically invalid CoT prompts offer the same level of performance gains as logically valid prompts on the hardest tasks in the BIG-Bench benchmark, termed BIG-Bench Hard (BBH). We find that the logically invalid reasoning prompts do indeed achieve similar performance gains on BBH tasks as logically valid reasoning prompts. We also discover that some CoT prompts used by previous works contain logical errors. This suggests that covariates beyond logically valid reasoning are responsible for performance improvements.

Introduction

Since the introduction of Chain-of-Thought (CoT) prompting—where a model is shown step-by-step reasoning before an answer—researchers have debated why it works. The prevailing theory was that these prompts "unlock" emergent human-like reasoning abilities. However, this paper suggests the answer is not necessarily tied to the logic itself. By testing on the challenging BIG-Bench Hard (BBH) benchmark, we found that prompts containing completely nonsensical or logically invalid reasoning steps still yield performance gains nearly identical to those of logically sound prompts.

Methodology

To investigate the role of logical validity, we compared three distinct prompting strategies using the Codex model:

  1. Answer-Only (AO): A baseline where the model is asked for a direct answer without intermediate steps.
  2. Chain-of-Thought (CoT): The standard approach using high-quality, human-written, logically valid step-by-step reasoning examples.
  3. Invalid Chain-of-Thought (Invalid CoT): A "bizarre" version where the reasoning steps in the examples were modified to be nonsensical or logically incorrect (e.g., claiming "not not True" evaluates to "not True"), while keeping the final answer in the example correct.

The study utilized BIG-Bench Hard (BBH), a subset of 23 tasks from the broader BIG-Bench collection. These tasks were specifically chosen because they are extremely difficult for language models and typically require multiple steps of reasoning, such as Boolean expressions, date understanding, and multi-step arithmetic.

Key Results

On average across all BBH tasks, CoT performed best, but Invalid CoT was a close second. Both significantly outperformed the Answer-Only baseline. This suggests that the model's performance does not collapse when the "logic" it is supposed to emulate is fundamentally broken.

Heatmap showing the percentage of answers found across BBH tasks for Answer-Only (AO), Chain-of-Thought (CoT), and Invalid CoT prompts.

In many tasks, the difference between valid and invalid logic was negligible. For example, in the "Boolean Expressions" task, the model was remarkably robust to the nonsensical logic provided in the prompt. We also confirmed our results by showing that our Answer-Only and CoT accuracies closely matched previously published benchmarks, ensuring the findings weren't due to a weak baseline.

Discussion: Structure over Substance

The results suggest that the performance boost from CoT might come from "covariates"—factors like the length of the response, the presence of intermediate steps, or specific formatting—rather than the actual logical sequence.

Accuracy gains of Chain-of-Thought (CoT) over Answer-Only (AO) across various BIG-Bench Hard tasks for three major language models.

Interestingly, while preparing these experiments, we discovered that the "valid" CoT prompts used in previous landmark papers actually contained several logical and arithmetic errors. Despite these errors, those prompts had been successfully used to demonstrate "reasoning" for years. This further supports the idea that the model is attending to structural features rather than strict logical validity.

Conclusion and Future Work

The primary limitation of this study is the focus on the Codex model. While Codex benefited significantly from the CoT format (likely due to its code-heavy training), it remains to be seen if newer, more instruction-tuned models like GPT-4 are more sensitive to logical consistency.

Future research should focus on quantifying the degree of incorrectness required to degrade performance and using mechanistic interpretability to understand what the model is actually attending to in a CoT prompt. This work suggests we might be over-optimizing the logic of our prompts when we should perhaps be focusing on the structural features that actually drive performance.

Summary updated

Abstract

Language models can be prompted to reason through problems in a manner that significantly improves performance. However, \textit{why} such prompting improves performance is unclear. Recent work showed that using logically \textit{invalid} Chain-of-Thought (CoT) prompting improves performance almost as much as logically \textit{valid} CoT prompting, and that editing CoT prompts to replace problem-specific information with abstract information or out-of-distribution information typically doesn't harm performance. Critics have responded that these findings are based on too few and too easily solved tasks to draw meaningful conclusions. To resolve this dispute, we test whether logically invalid CoT prompts offer the same level of performance gains as logically valid prompts on the hardest tasks in the BIG-Bench benchmark, termed BIG-Bench Hard (BBH). We find that the logically \textit{invalid} reasoning prompts do indeed achieve similar performance gains on BBH tasks as logically valid reasoning prompts. We also discover that some CoT prompts used by previous works contain logical errors. This suggests that covariates beyond logically valid reasoning are responsible for performance improvements.
BibTeX
@misc{schaeffer2023invalid,
  author = {Rylan Schaeffer and Kateryna Pistunova and Samar Khanna and Sarthak Consul and Sanmi Koyejo},
  title = {{Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting}},
  year = {2023},
  eprint = {2307.10573},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.10573},
}
Figure