Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers

Tuo Zhang, Jinyue Yuan, Salman Avestimehr

Introduction

Prompt engineering is a critical yet labor-intensive process for maximizing Large Language Model (LLM) performance. Optimization by PROmpting (OPRO) was introduced to automate this by using an LLM as an optimizer to iteratively refine instructions based on past performance. While OPRO has demonstrated success with frontier models like GPT-4, its efficacy on smaller, open-source models remains under-explored. This study investigates whether the "self-optimization" capability scales down to models in the 7B to 70B parameter range.

Methodology

The researchers evaluated the OPRO framework using the GSM8K dataset, a benchmark for grade school math reasoning. The core of OPRO involves a "meta-prompt" containing a trajectory of previous instructions and their accuracy scores. The optimizer model is tasked with generating a new instruction to improve results.

To understand the baseline reasoning capabilities required, the authors conducted a motivational study using a simple linear regression task. They asked Llama-2-13B to find optimal weights ($w, b$) for a dataset via prompting, providing previous pairs and their function values.

Llama-2-13B failing to optimize a simple linear regression task, illustrating the reasoning bottleneck in small models.

The Optimization Gap

The findings reveal a significant "optimization gap" between large and small models. While Gemini-Pro successfully navigated the OPRO loop to improve its prompts, smaller models like Llama-2-70B and Mistral 7B often saw performance degradation. For instance, Llama-2-70B's accuracy dropped from 39.35% (using a simple manual prompt) to 27.98% when using OPRO-generated instructions.

Smaller models proved to be "bad bosses." In the linear regression test, Llama-2-13B frequently reported an inability to find better solutions, citing the non-linearity or complexity of the task, even when simple gradient descent logic was applicable. This suggests that the bottleneck is not just in evaluating results, but in the fundamental ability to reason about optimization trajectories.

Comparative Results and Efficiency

The study compared OPRO against standard manual baselines: Zero-shot Chain-of-Thought (CoT) ("Let’s think step by step") and Few-shot CoT. For small models, the simple "Let’s think step by step" prompt remained a robust and often superior baseline.

Comparison of optimization trajectories on GSM8K. (a) Gemini-Pro shows a clear upward trend in accuracy, whereas (b) and (c) demonstrate the erratic and unsuccessful optimization attempts of Llama-2-13B regardless of the scorer model used.

Beyond accuracy, the computational cost of OPRO is a major concern. Reaching an optimal prompt with Gemini-Pro required 21 hours and over 266,000 tokens. For smaller models, this massive overhead resulted in a negative return on investment, as they failed to surpass the performance of a zero-cost manual prompt.

Conclusion

Self-optimization via prompting appears to be an emergent property of scale. For developers deploying models in the 7B-70B range, automated prompt engineering frameworks like OPRO may not yet be viable. The authors recommend prioritizing high-quality manual instructions and traditional CoT baselines, as the reasoning limitations of smaller models prevent them from effectively navigating complex optimization trajectories. Future work should focus on making automated optimization more computationally efficient and accessible for smaller-scale architectures.

Summary updated

BibTeX
@inproceedings{zhang2024revisiting,
  author = {Tuo Zhang and Jinyue Yuan and Salman Avestimehr},
  title = {{Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.100},
}
Figure