Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extraction

Saurabh Srivastava, Ziyu Yao

Abstract

Large Reasoning Models (LRMs) such as DeepSeek-R1 and OpenAI o1 have demonstrated remarkable capabilities in various reasoning tasks. Their strong capability to generate and reason over intermediate thoughts has also led to arguments that they may no longer require extensive prompt engineering or optimization to interpret human instructions and produce accurate outputs. In this work, we aim to systematically study this open question, using the structured task of event extraction for a case study. We experimented with two LRMs (DeepSeek-R1 and o1) and two general-purpose Large Language Models (LLMs) (GPT-4o and GPT-4.5), when they were used as task models or prompt optimizers. Our results show that on tasks as complicated as event extraction, LRMs as task models still benefit from prompt optimization, and that using LRMs as prompt optimizers yields more effective prompts. Finally, we provide an error analysis of common errors made by LRMs and highlight the stability and consistency of LRMs in refining task instructions and event guidelines.

Introduction

The emergence of Large Reasoning Models (LRMs) like OpenAI’s o1 and DeepSeek-R1 has sparked a debate: does their internal Chain-of-Thought (CoT) capability render prompt engineering obsolete? While these models excel at following complex instructions, this study investigates whether systematic prompt optimization still provides value, particularly for structured tasks like Event Extraction (EE).

EE is a complex NLP task requiring models to identify "triggers" (what happened), "arguments" (who was involved), and "roles" (what part they played), all while adhering to a strict schema. This paper tests if LRMs still benefit from iterative prompt optimization and if they are superior at the "meta-task" of optimizing prompts for other models.

Methodology: Automated Prompt Optimization via MCTS

To test the limits of LRMs, the researchers employed a Monte Carlo Tree Search (MCTS) framework for discrete prompt optimization. This approach moves beyond manual trial-and-error by using an automated loop to refine instructions based on objective performance feedback.

The MCTS framework for prompt optimization involves a task model, an error-identifying interpreter, and an optimizer model that generates refined instructions.

The framework consists of four key components:

  1. The Task Model ($M_{task}$): Executes the EE task using the current prompt.
  2. The Interpreter: A Python-based script that compares outputs to ground truth, identifying specific errors like missing arguments or hallucinations.
  3. The Optimizer ($M_{opt}$): An LLM or LRM that analyzes errors and generates structured feedback to refine the prompt.
  4. The Search: MCTS explores the prompt space, branching into different instruction versions and retaining those with the highest F1 scores on a development set.

The study evaluated standard LLMs (GPT-4o, GPT-4.5) and LRMs (DeepSeek-R1, o1) in both task and optimizer roles across datasets like ACE05 (Event Extraction), Geometric Shapes, and NCBI Disease NER.

Key Results: The Compounded Advantage of Reasoning

The findings challenge the notion that LRMs are "prompt-proof." In fact, LRMs demonstrated significantly larger gains from optimization than standard LLMs.

Average performance metrics show that LRMs as optimizers yield the most significant improvements and that LRMs as task models benefit the most from optimization.
  • Massive Performance Gains: On the $ACE_{med}$ dataset, DeepSeek-R1’s performance jumped from a baseline of 16.45 to 40.00 AC F1—a staggering relative improvement. This indicates that even models with strong internal reasoning benefit from external instruction refinement.
  • LRMs as Superior Optimizers: When acting as $M_{opt}$, LRMs generated more precise, actionable rules. While LLMs often provided vague advice, LRMs like DeepSeek-R1 produced specific heuristics, such as removing specific articles and possessive pronouns unless they are part of official names.
  • Efficiency and Stability: DeepSeek-R1 achieved peak performance with shorter, more concise prompts (~1750 tokens) compared to the more verbose instructions favored by GPT-4.5 and o1. Furthermore, LRM-guided optimization paths were more stable and converged faster, better distinguishing between guideline errors and model limitations.
  • Cross-Model Transfer: Prompts optimized by DeepSeek-R1 for other models (like GPT-4o) often outperformed prompts those models generated for themselves, highlighting the "expert" nature of LRM-driven optimization.

Discussion and Conclusion

This research demonstrates that prompt optimization remains a critical tool even in the era of reasoning models. LRMs do not just follow instructions better; they are also more capable of refining those instructions when provided with structured feedback. The ability of LRMs to distill complex task requirements into concise, high-performing guidelines suggests that the future of prompt engineering lies in LRM-driven automated optimization rather than manual intervention. Future work will explore scaling these methods to even more complex schemas and real-time self-correction during inference.

Summary updated

Abstract

Large Reasoning Models (LRMs) such as DeepSeek-R1 and OpenAI o1 have demonstrated remarkable capabilities in various reasoning tasks. Their strong capability to generate and reason over intermediate thoughts has also led to arguments that they may no longer require extensive prompt engineering or optimization to interpret human instructions and produce accurate outputs. In this work, we aim to systematically study this open question, using the structured task of event extraction for a case study. We experimented with two LRMs (DeepSeek-R1 and o1) and two general-purpose Large Language Models (LLMs) (GPT-4o and GPT-4.5), when they were used as task models or prompt optimizers. Our results show that on tasks as complicated as event extraction, LRMs as task models still benefit from prompt optimization, and that using LRMs as prompt optimizers yields more effective prompts. Finally, we provide an error analysis of common errors made by LRMs and highlight the stability and consistency of LRMs in refining task instructions and event guidelines.
BibTeX
@misc{srivastava2025revisiting,
  author = {Saurabh Srivastava and Ziyu Yao},
  title = {{Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extraction}},
  year = {2025},
  eprint = {2504.07357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.07357},
}
Figure