TL;DR
This paper introduces DDPrompt, a novel prompting strategy that generates diverse reasoning paths tailored to specific question types, significantly improving Large Language Model (LLM) performance on complex reasoning tasks. By clustering questions and selecting optimal trigger sentences for each cluster, DDPrompt moves beyond uniform prompting to unlock more robust and accurate LLM reasoning.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive capabilities, but their performance on complex reasoning tasks is highly sensitive to the prompts used. Traditional methods like Chain-of-Thought (CoT) prompting, while effective, often require laborious manual design of demonstrations. More recent approaches like Zero-Shot-CoT simplify this by using a single, generic trigger sentence (e.g., "Let's think step by step") for all questions. However, this paper argues that different question types benefit from different reasoning strategies. The core problem addressed is the lack of a method that automatically generates diverse and appropriate reasoning paths for LLMs, moving beyond a one-size-fits-all prompting approach to better leverage the LLM's potential for complex problem-solving.
Methods
DDPrompt employs a two-stage approach:
- Generating Optimal Trigger Sentence Set (GOTSS):
- Question Clustering: Questions are first embedded using Sentence-BERT and then clustered using K-Means based on their similarity. This groups questions of similar types.
- Generating Optimal Collection: For each cluster, a set of manually constructed trigger sentences is evaluated. The trigger sentences that yield the highest accuracy for questions within that cluster are selected to form an "optimal trigger sentence set" for that specific question type.
- Inference:
- Given a new question, its embedding is computed, and it's assigned to the most similar cluster.
- The optimal trigger sentence set for that cluster is retrieved.
- The LLM is prompted with the question and each trigger sentence from the optimal set, generating multiple reasoning paths and answers.
- A majority voting mechanism is used to determine the final answer from these multiple generated responses.
This contrasts with Zero-Shot-CoT, which uses a single trigger sentence for all questions, whereas DDPrompt dynamically selects the best trigger sentences based on question characteristics.
Datasets
The paper evaluates DDPrompt on twelve reasoning benchmarks across four categories:
- Arithmetic: SingleEq, AddSub, MultiArith, AQUA-RAT, GSM8K, SVAMP
- Commonsense: CSQA, StrategyQA
- Symbolic: Last Letter Concatenation, Coin Flip
- Logical: Date Understanding, Tracking Shuffled Objects
The experiments utilize a training dataset for the GOTSS phase and a test dataset for evaluation.
Key Insights and Ideas
- Question Type Matters: The central idea is that different types of questions require different reasoning approaches, and a single trigger sentence is suboptimal for diverse problem sets.
- Diversity Enhances Reasoning: Generating multiple reasoning paths, even with the same question, can lead to more robust and accurate answers, especially for complex problems.
- Automated Prompt Optimization: DDPrompt automates the selection of effective prompts by clustering questions and identifying the best-performing trigger sentences for each cluster, reducing manual effort compared to traditional CoT.
- Clustering as a Foundation: The effectiveness of DDPrompt relies on accurately clustering questions to identify distinct types that benefit from specialized prompting.
Key Results
DDPrompt demonstrates significant performance improvements across various reasoning benchmarks compared to baselines like Zero-Shot-CoT.
- GSM8K: Accuracy improved from 75% to 84%.
- AQUA-RAT: Accuracy improved from 50% to 63%.
- Last Letter Concatenation: Accuracy improved from 64% to 89.8%.
- Tracking Shuffled Objects: Accuracy improved from 68.8% to 83.9%.
DDPrompt outperformed Few-Shot-CoT (which requires manual rationale design) on eight out of twelve reasoning tasks. The paper highlights that DDPrompt is particularly effective on more intricate arithmetic problems requiring multi-step reasoning.
Limitations and Future Work
- Computational Cost: DDPrompt is slower than other prompting methods because it requires feeding multiple trigger sentences and questions into the LLM to generate multiple answers.
- LLM Generalization: The current evaluation was performed using only GPT3.5-turbo. The authors plan to evaluate DDPrompt on other LLMs in the future.
- Future Work: The paper suggests exploring DDPrompt on other LLMs and potentially optimizing the inference stage for speed.
Summary updated