ZERA: Zero-init Instruction Evolving Refinement Agent – From Zero Instructions to Structured Prompts via Principle-based Optimization

Seungyoun Yi, Minsoo Khang, Sungrae ParkEMNLP 2025

TL;DR

This paper introduces ZERA, a novel approach to automatically optimize prompts for Large Language Models (LLMs). ZERA refines both system and user prompts by using a principled, low-overhead method that leverages eight evaluation principles, enabling faster convergence and better generalization compared to existing methods.

What Problem Does This Paper Address?

Crafting effective prompts for LLMs is crucial for their performance, especially in complex agent-based systems. However, current prompt optimization methods often rely on unstructured feedback, require large datasets, and are prone to overfitting to specific tasks or models. This makes them costly, brittle, and difficult to generalize. The authors aim to address these limitations by developing a more robust, efficient, and generalizable prompt optimization framework.

Methods

ZERA employs a two-stage iterative, training-free framework:

  1. Principle-based Critique Generation (PCG): This stage evaluates LLM outputs based on eight general principles (Meaning Accuracy, Completeness, Expression Style, Faithfulness, Conciseness, Correctness, Structural Alignment, Reasoning Quality). It assigns importance weights to these principles for a given task, scores the output against each principle, and generates natural language feedback and suggestions.
  2. Meta-cognitive Prompt Refinement (MPR): This stage uses the critiques from PCG, along with historical prompt information (previous prompts, best-performing prompts, and exemplar task samples), to iteratively refine the task description and the system/user prompts.

A key novelty of ZERA is its "zero-init" approach, meaning it starts with minimal, generic prompts and no task-specific instructions or metrics, relying solely on the task samples and the principled evaluation framework.

Datasets

The paper evaluates ZERA across nine diverse benchmark datasets:

  • Reasoning: MMLU, MMLU-Pro, GSM8K, MBPP, HumanEval, BBH, HellaSwag
  • Summarization: CNN/DailyMail, SAMSum

These datasets cover a range of tasks including reasoning, summarization, and code generation.

Key Insights and Ideas

  • Principle-based Optimization: Using a set of eight general principles for evaluation and refinement is more robust and generalizable than relying solely on task-specific metrics or unstructured feedback. This approach helps prevent overfitting.
  • Joint Optimization: ZERA jointly optimizes system prompts, user prompts, and task descriptions, which is a novel contribution.
  • Zero-Initialization: The ability to start from scratch with minimal prompts and still achieve high performance is a significant advantage, reducing the need for manual prompt engineering.
  • Efficiency: ZERA requires fewer API calls and task samples compared to other state-of-the-art methods, making it more practical.
  • Task-Adaptive Weighting: The importance weights assigned to the eight principles are dynamically inferred based on the task, allowing ZERA to prioritize criteria that are most relevant for optimal performance. For example, "reasoning quality" is weighted higher for reasoning tasks.

Key Results

  • Performance Gains: ZERA consistently improves performance across various LLMs (GPT-3.5-turbo, GPT-40, LLaMA-3.1, Qwen2.5, Mistral-7B) and tasks, outperforming baseline prompts and state-of-the-art APO methods like PromptAgent, OPRO, and CriSPO.
  • Specific Improvements: ZERA achieved a +6.0 ROUGE-L improvement on SAMSum and outperformed PromptAgent in 5 out of 6 BBH sub-tasks, with significant gains in epistemic reasoning (+20.0).
  • Efficiency: ZERA demonstrated the lowest number of API calls and required significantly fewer task samples (20 vs. 200 for OPRO/CriSPO) to achieve comparable or better performance.
  • Prompt Evolution: The paper shows how ZERA's prompts evolve from generic instructions to highly structured, task-specific formats, often incorporating self-generated reasoning exemplars.

Limitations and Future Work

  • Summarization Metrics: The evaluation of summarization tasks relies on automatic metrics (ROUGE-L) without human judgment, which might miss nuances like coherence or factuality.
  • Sample Requirement: While ZERA uses fewer samples (5-20 per iteration), it still requires some training data, leaving fully zero-shot refinement as an open challenge.
  • Prompt Length: Optimized prompts can become longer, potentially increasing inference latency or causing context overflow, though the authors note that prompt length typically stabilizes.
  • Internal LLM Dependency: ZERA relies on an internal LLM for feedback, and its reliability under ambiguous or adversarial outputs needs further analysis.

Future work could explore fully zero-shot refinement, address prompt length efficiency, and further analyze the robustness of the internal LLM's feedback mechanism.

Summary updated

BibTeX
@inproceedings{yi2025zera,
  author = {Seungyoun Yi and Minsoo Khang and Sungrae Park},
  title = {{ZERA: Zero-init Instruction Evolving Refinement Agent – From Zero Instructions to Structured Prompts via Principle-based Optimization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure