Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers
Abstract Optimization by PROmpting (OPRO) has emerged as a promising technique for leveraging Large Language Models (LLMs) to automatically refine instructions. While frontier models like GPT-4 demonstrate significant self-optimization capabilities, this study investigates whether such benefits extend to smaller, open-source models (7B to 70B parameters). Our findings reveal a "capability floor" for meta-reasoning; smaller models often fail to improve prompts iteratively, frequently underperforming simple manual baselines like Chain-of-Thought (CoT) while incurring substantial computational costs. We conclude that for small-scale LLMs, manual prompt engineering remains more effective and efficient than automated OPRO loops.
Introduction
The efficacy of Large Language Models (LLMs) is highly sensitive to the specific phrasing of prompts. This has led to the development of automated prompt engineering techniques, most notably Optimization by PROmpting (OPRO). In the OPRO framework, an LLM acts as an "optimizer," analyzing a history of previously attempted prompts and their associated accuracy scores to generate improved instructions. While successful on massive models, the applicability of OPRO to the smaller models typically deployed in production—such as LLaMa-2 or Mistral—remains under-explored. This paper evaluates whether self-optimization is an emergent property of scale or a viable tool for the broader open-source community.
Methods
The researchers replicated the OPRO framework across a spectrum of model sizes. The methodology involved:
- Motivational Study: A simplified linear regression task (optimizing $w$ and $b$) to test basic numerical optimization capabilities via natural language.
- The OPRO Loop: A full iterative cycle where an "optimizer" LLM generates instructions for a "scorer" LLM, which evaluates them on a target task.
- Comparative Baselines: Performance was measured against standard Zero-shot and Few-shot Chain-of-Thought (CoT) prompting.
- Meta-Instruction Sensitivity: Evaluation of how different human-written starting instructions influenced the automated optimization trajectory.
Key Insights: The Capability Floor
A primary finding is the existence of a "capability floor" for the meta-reasoning required by OPRO. Small-scale models (7B-70B) struggle to synthesize feedback from previous iterations into meaningful improvements.
As shown in Figure 1, when tasked with a simple optimization problem, LLaMa-2-13B failed to grasp the underlying logic, even hallucinating technical difficulties with "gradient descent" algorithms it was not actually running. This lack of meta-awareness translates to the prompt optimization task, where smaller models often generate generic or repetitive instructions (e.g., "Let's think about") rather than task-specific refinements.
Key Results: Performance and Efficiency
The disparity between large and small models becomes evident when tracking accuracy over optimization steps. While large models like Gemini-Pro show a clear, albeit noisy, upward trend in performance, smaller models exhibit high variance with no sustained improvement.
On the GSM8K benchmark, Gemini-Pro achieved a performance boost through OPRO (76.92% vs. 71.29% for Zero-shot CoT). Conversely, for Mistral 7B, LLaMa-2-13B, and LLaMa-2-70B, OPRO failed to surpass the Few-shot CoT baseline. In the case of LLaMa-2-70B, OPRO-generated prompts were significantly worse (27.98%) than Few-shot CoT (48.67%).
Furthermore, OPRO is remarkably inefficient for small models. Reaching an "optimal" prompt with Gemini-Pro required approximately 266,000 tokens and 21 hours of compute. For smaller models, this massive investment yielded results inferior to a simple 130-token Few-shot CoT prompt that took only 5 hours to evaluate.
Conclusion
Automated prompt optimization via OPRO is currently a "luxury" of scale. For models in the 7B to 70B parameter range, the "optimizer" becomes a bottleneck. These models lack the inherent reasoning capacity to effectively navigate the prompt space based on historical performance. Future research should focus on developing more efficient, scale-aware optimization methods or prioritizing the refinement of robust manual baselines for smaller LLMs.
Summary updated