Abstract
Large language models (LLMs) often struggle with zero-shot formal reasoning, frequently relying on shallow syntactic heuristics rather than logical deduction. This paper investigates "Dynamic Problem Elaboration," a strategy where a model generates its own intermediate reasoning—or "self-talk"—before arriving at a final answer. Using GPT-2 (1.5B), we demonstrate that this "thinking aloud" process can improve zero-shot accuracy by up to 24% on complex logic puzzles. Our findings suggest that elaboration works by increasing the "epistemic luck" of the prompt, allowing simple heuristics to latch onto correct conclusions repeated within the generated context.
Introduction
Despite the remarkable capabilities of transformer-based models, their performance on formal reasoning tasks in zero-shot settings remains inconsistent. Research indicates that models like GPT-2 often employ a syntactic heuristic: they prioritize answers based on the frequency of terms in the "then" clauses of provided rules rather than following the logical chain. This leads to systematic failures in the presence of "distractors" (irrelevant rules) or when tasks require complex operations like contraposition (reasoning from "not-B" to "not-A").
This paper addresses the challenge of improving zero-shot reasoning without the need for computationally expensive fine-tuning. We propose a meta-cognitive approach inspired by human "thinking aloud" strategies, enabling the model to expand its own context dynamically.
Methodology: Dynamic Problem Elaboration
The core contribution of this work is the Dynamic Problem Elaboration framework. Instead of a direct prompt-to-answer pipeline, we introduce an intermediate generation step.
The process consists of four stages:
- Prompting for Elaboration: The model receives the problem description and is prompted to "Explain!" or "Therefore...".
- Generating Context: The model generates several sentences of "self-talk" analyzing the premises.
- Feedback Loop: This generated elaboration is appended to the original problem context.
- Final Prediction: The model makes its final prediction based on the augmented context.
We explored several elaboration variants, including Free Elaboration (zero-shot), Few-shot Elaboration (using proof chains or intermediary conclusions as guides), and Piecemeal approaches (Structured or Recursive) designed to maintain focus on the problem's subject.
Experimental Setup: The ChainRuler Dataset
To evaluate these methods, we developed ChainRuler, a synthetic benchmark for logical reasoning. ChainRuler allows for precise control over the depth (number of reasoning steps), breadth (number of distractors), and logic type (standard deductive chains versus contraposition tasks involving negations).
A typical task includes a "fact" (e.g., "Jill is green"), a "rule chain" (e.g., "If green, then loud; if loud, then guilty"), and distractors. The model must determine the validity of a conclusion (e.g., "Jill is guilty") among several alternatives.
Results and Analysis
Our experiments with GPT-2 (1.5B) revealed significant performance gains across various problem configurations.
Key Findings
- Accuracy Improvements: Dynamic elaboration improved zero-shot performance by an average of 9%. In optimal configurations—specifically using "piecemeal" elaborations—accuracy jumped by 16% to 24% over the baseline.
- The Role of Redundancy: Interestingly, elaboration does not necessarily make the model "smarter" at logic. Instead, it often leads the model to repeat the correct conclusion. This redundancy increases the "total epistemic luck," making it easier for the model's inherent syntactic heuristics to identify the correct answer.
- De-biasing Effect: In high-distraction scenarios where the model typically performs worse than random guessing, elaboration acts as a de-biasing mechanism, restoring performance to levels above chance.
- The Negation Barrier: While effective for standard chains, elaboration struggles with contraposition. The presence of "not" significantly hampers the model's ability to generate faithful and useful self-talk.
Discussion and Future Work
While Dynamic Problem Elaboration provides a clear path to better zero-shot reasoning, it highlights a fundamental limitation: the final prediction step remains "shallow," relying on word-matching heuristics even if the context is improved.
Future research should explore scaling these dynamics to larger models (e.g., GPT-4) and investigating whether models can be fine-tuned specifically to generate high-quality reasoning chains. Additionally, reinforcement learning could be used to train models to identify which types of "self-talk" are most effective for specific logical structures.
Summary updated