Introduction
Prompt engineering is a critical yet labor-intensive process for maximizing Large Language Model (LLM) performance. Optimization by PROmpting (OPRO) was introduced to automate this by using an LLM as an optimizer to iteratively refine instructions based on past performance. While OPRO has demonstrated success with frontier models like GPT-4, its efficacy on smaller, open-source models remains under-explored. This study investigates whether the "self-optimization" capability scales down to models in the 7B to 70B parameter range.
Methodology
The researchers evaluated the OPRO framework using the GSM8K dataset, a benchmark for grade school math reasoning. The core of OPRO involves a "meta-prompt" containing a trajectory of previous instructions and their accuracy scores. The optimizer model is tasked with generating a new instruction to improve results.
To understand the baseline reasoning capabilities required, the authors conducted a motivational study using a simple linear regression task. They asked Llama-2-13B to find optimal weights ($w, b$) for a dataset via prompting, providing previous pairs and their function values.
The Optimization Gap
The findings reveal a significant "optimization gap" between large and small models. While Gemini-Pro successfully navigated the OPRO loop to improve its prompts, smaller models like Llama-2-70B and Mistral 7B often saw performance degradation. For instance, Llama-2-70B's accuracy dropped from 39.35% (using a simple manual prompt) to 27.98% when using OPRO-generated instructions.
Smaller models proved to be "bad bosses." In the linear regression test, Llama-2-13B frequently reported an inability to find better solutions, citing the non-linearity or complexity of the task, even when simple gradient descent logic was applicable. This suggests that the bottleneck is not just in evaluating results, but in the fundamental ability to reason about optimization trajectories.
Comparative Results and Efficiency
The study compared OPRO against standard manual baselines: Zero-shot Chain-of-Thought (CoT) ("Let’s think step by step") and Few-shot CoT. For small models, the simple "Let’s think step by step" prompt remained a robust and often superior baseline.
Beyond accuracy, the computational cost of OPRO is a major concern. Reaching an optimal prompt with Gemini-Pro required 21 hours and over 266,000 tokens. For smaller models, this massive overhead resulted in a negative return on investment, as they failed to surpass the performance of a zero-cost manual prompt.
Conclusion
Self-optimization via prompting appears to be an emergent property of scale. For developers deploying models in the 7B-70B range, automated prompt engineering frameworks like OPRO may not yet be viable. The authors recommend prioritizing high-quality manual instructions and traditional CoT baselines, as the reasoning limitations of smaller models prevent them from effectively navigating complex optimization trajectories. Future work should focus on making automated optimization more computationally efficient and accessible for smaller-scale architectures.
Summary updated