Abstract
Chain-of-thought (CoT) prompting can guide language models to engage in complex multi-step reasoning. The quality of provided demonstrations significantly impacts the success of downstream inference tasks. While existing automated methods prioritize accuracy and semantics in these demonstrations, we show that the underlying reasoning patterns play a more crucial role in such tasks. In this paper, we propose Pattern-Aware CoT (PA-CoT), a prompting method that considers the diversity of demonstration patterns. By incorporating patterns such as step length and reasoning process within intermediate steps, PA-CoT effectively mitigates the issue of bias induced by demonstrations and enables better generalization to diverse scenarios. We conduct experiments on nine reasoning benchmark tasks using two open-source LLMs. The results show that our method substantially enhances reasoning performance and exhibits robustness to errors.
Introduction: The Pitfalls of Semantic Selection
Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, the effectiveness of few-shot CoT relies heavily on the quality of the demonstrations provided. Current state-of-the-art unsupervised methods, such as Auto-CoT, typically select demonstrations based on semantic similarity—grouping questions by their topical content (e.g., "shopping" or "apples").
The authors identify two critical flaws in this approach. First, semantic similarity is often "noisy": two questions about the same topic may require vastly different logical operations. Second, relying on a narrow set of semantically similar examples can induce a "monotonous reasoning" bias. If all demonstrations in a prompt use the same mathematical operation, the model may blindly apply that operation to a new problem, even if it is logically inappropriate.
As shown in Figure 1, when a prompt contains a multiplication pattern (Prompt 1), the model incorrectly applies multiplication to a division problem. Conversely, a prompt with a division pattern (Prompt 2) guides the model to the correct solution. This demonstrates that the logical pattern of the demonstration is more influential than its semantic topic.
The PA-CoT Framework
To address these issues, the researchers propose Pattern-Aware CoT (PA-CoT). This method shifts the selection criteria from semantic topics to underlying logical reasoning patterns—the specific sequence of operations required to solve a problem.
The PA-CoT workflow (Figure 2) consists of four primary stages:
- Seed Collection: Initial rationales are generated for a pool of questions using Zero-Shot CoT ("Let's think step by step").
- Pattern Discovery: A set of "operation tokens" is identified. For arithmetic tasks, these include operators like
+,-,*,/, and keywords liketwice. For logical tasks, tokens might include temporal markers likeyesterday. - Pattern Extraction: The system extracts the sequence of these tokens from each seed rationale to define its "reasoning pattern" (e.g.,
[+, *, -]). - Pattern-Wise Selection: These patterns are converted into embeddings and clustered using k-means. By selecting one representative example from each cluster, PA-CoT ensures the final prompt contains a diverse array of logical strategies.
A key technical contribution is the Adaptive $k$ formula, which determines the optimal number of demonstrations based on the complexity of the operation space, preventing the model from becoming biased toward any single reasoning path.
Experimental Evaluation and Key Insights
The method was evaluated across nine reasoning benchmarks, including arithmetic (GSM8K, AQuA, SVAMP), symbolic (Coin-Flip), and logical reasoning (Date Understanding).
Patterns Over Correctness
One of the most striking findings is that LLMs are more influenced by the logical template of a demonstration than by its factual accuracy. On the AQuA dataset, PA-CoT improved performance even when the selected demonstrations were mathematically incorrect, as the model successfully learned the method of solution rather than the specific numbers.
Mitigating Semantic Noise
In domains like "Date Understanding," where all questions are semantically similar (all about dates), traditional clustering fails to provide diversity. PA-CoT excels by identifying the distinct logical shifts required for different date-related queries. Using perturbation analysis, the authors demonstrated that a diverse set of patterns in the prompt more reliably "activates" the correct reasoning pathways in the model, whereas "dense" (repetitive) patterns lead to systematic errors.
Results and Conclusion
PA-CoT consistently outperformed Zero-Shot CoT, Random-CoT, and Auto-CoT across various models, including LLaMA-2 (7B/13B), GPT-3.5, and Qwen-7B. For instance, on the MultiArith dataset with LLaMA-2-7B, PA-CoT achieved 79.66% accuracy, compared to Auto-CoT’s 76.00%. On the "Date" understanding task, it reached 60.97% accuracy (13B model), a significant improvement over Auto-CoT's 53.93%.
In conclusion, PA-CoT demonstrates that the diversity of reasoning patterns is a fundamental driver of LLM performance in complex tasks. By prioritizing logical structure over semantic topic, PA-CoT provides a more robust and interpretable framework for few-shot prompting.
Summary updated