PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine

Chenrui Zhang, Lin Liu, Jinpeng Wang, Chuyuan Wang, Xiao Sun, Hongyu Wang, Mingchen Cai

Abstract

As an effective tool for eliciting the power of Large Language Models (LLMs), prompting has recently demonstrated unprecedented abilities across a variety of complex tasks. To further improve the performance, prompt ensemble has attracted substantial interest for tackling the hallucination and instability of LLMs. However, existing methods usually adopt a two-stage paradigm, which requires a pre-prepared set of prompts with substantial manual effort, and is unable to perform directed optimization for different weak learners. In this paper, we propose a simple, universal, and automatic method named PREFER (Pompt Ensemble learning via Feedback-Reflect-Refine) to address the stated limitations. Specifically, given the fact that weak learners are supposed to focus on hard examples during boosting, PREFER builds a feedback mechanism for reflecting on the inadequacies of existing weak learners. Based on this, the LLM is required to automatically synthesize new prompts for iterative refinement. Moreover, to enhance stability of the prompt effect evaluation, we propose a novel prompt bagging method involving forward and backward thinking, which is superior to majority voting and is beneficial for both feedback and weight calculation in boosting. Extensive experiments demonstrate that our PREFER achieves state-of-the-art performance in multiple types of tasks by a significant margin.

Introduction: The "Black Art" of Prompt Engineering

Prompt engineering is essential for maximizing Large Language Model (LLM) performance but remains a manual, trial-and-error process. While prompt ensembling—combining multiple prompts to vote on a result—mitigates instability and hallucinations, current methods are often static. They rely on pre-defined prompt sets and lack "directed optimization," failing to adapt based on specific errors the model makes on "hard" examples.

Methodology: The PREFER Framework

PREFER introduces an automated, adaptive approach to prompt ensembling through two primary components: the Boosting Branch and the Stability Branch.

1. Feedback-Reflect-Refine (The Boosting Branch)

Inspired by the classical AdaBoost algorithm, PREFER treats each prompt as a "weak learner." The process follows an iterative cycle:

  • Feedback: The system identifies "hard examples" where the current prompt fails.
  • Reflect: The LLM analyzes these errors to explain why the prompt was insufficient.
  • Refine: The LLM generates a new, optimized prompt targeting these specific failures. This creates a "textual gradient" that guides the optimization process.

2. Bilateral Bagging (The Stability Branch)

To improve individual prompt reliability, the authors propose "bilateral bagging." Instead of simple majority voting, it employs:

  • Forward Thinking: Assessing the confidence that a candidate answer is correct.
  • Backward Thinking: Evaluating the confidence that an answer should be excluded (process of elimination). This dual assessment counteracts LLM overconfidence and improves calibration.
The PREFER framework architecture, illustrating the iterative Feedback-Reflect-Refine cycle and Bilateral Bagging.

Experimental Results

The researchers evaluated PREFER across seven benchmarks, including Natural Language Inference (SNLI, MNLI, QNLI, RTE) and classification tasks (Ethos, Liar, ArSarcasm).

PREFER consistently outperformed single-prompt baselines, Chain-of-Thought (CoT) prompting, and existing state-of-the-art methods like PromptBoosting and APO. Notably, on the Liar dataset for fake news classification, PREFER improved performance by 13.1% over the next best method.

Performance comparison of PREFER against various baselines across seven diverse datasets.

Key Insights

  • Text as a Gradient: Natural language feedback serves as a directional signal to "update" prompts, analogous to numerical gradients in traditional neural networks.
  • Targeting the "Hard" Space: By focusing on previous failures, each new prompt acts as a specialist for different sub-problems, expanding the ensemble's overall solving capacity.
  • Calibration via Elimination: Implementing "backward thinking" mirrors human decision-making and significantly reduces variance in model predictions.

Conclusion and Future Work

PREFER demonstrates that automated, iterative refinement can significantly boost LLM performance without manual intervention. While highly effective, the ensemble approach increases inference latency and cost. Future research will focus on distilling ensemble knowledge into a single "super-prompt" and exploring other classical machine learning algorithms re-imagined for LLMs.

Summary updated

Abstract

As an effective tool for eliciting the power of Large Language Models (LLMs), prompting has recently demonstrated unprecedented abilities across a variety of complex tasks. To further improve the performance, prompt ensemble has attracted substantial interest for tackling the hallucination and instability of LLMs. However, existing methods usually adopt a two-stage paradigm, which requires a pre-prepared set of prompts with substantial manual effort, and is unable to perform directed optimization for different weak learners. In this paper, we propose a simple, universal, and automatic method named PREFER (Pompt Ensemble learning via Feedback-Reflect-Refine) to address the stated limitations. Specifically, given the fact that weak learners are supposed to focus on hard examples during boosting, PREFER builds a feedback mechanism for reflecting on the inadequacies of existing weak learners. Based on this, the LLM is required to automatically synthesize new prompts for iterative refinement. Moreover, to enhance stability of the prompt effect evaluation, we propose a novel prompt bagging method involving forward and backward thinking, which is superior to majority voting and is beneficial for both feedback and weight calculation in boosting. Extensive experiments demonstrate that our PREFER achieves state-of-the-art performance in multiple types of tasks by a significant margin. We have made our code publicly available.
BibTeX
@inproceedings{zhang2024prefer,
  author = {Chenrui Zhang and Lin Liu and Jinpeng Wang and Chuyuan Wang and Xiao Sun and Hongyu Wang and Mingchen Cai},
  title = {{PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29924},
}
Figure