Abstract
Current research on long-form context in Large Language Models (LLMs) primarily focuses on the understanding of long-contexts, while Open-ended Long Text Generation (Open-LTG) remains insufficiently explored. Training a long text generation model requires curation of gold-standard reference data, which is typically nonexistent for informative Open-LTG tasks. However, previous methods only utilize general assessments as reward signals, which limits accuracy. To bridge this gap, we introduce ProxyReward, an innovative reinforcement learning (RL) based framework, which includes a data synthesis method and a novel reward signal. Firstly, ProxyReward Dataset synthesis is accomplished through simple prompts that enables the model to create automatically, obviating extensive labeled data or significant manual effort. Secondly, ProxyReward Signal offers a targeted evaluation of information comprehensiveness and accuracy for specific questions. The experimental results indicate that our method ProxyReward surpasses even GPT-4-Turbo. It can significantly enhance performance by 20% on the Open-LTG task when training widely used open-source models, while also surpassing the LLM-as-a-Judge approach. Our work presents effective methods to enhance the ability of LLMs to address complex open-ended questions posed by humans.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in processing long contexts, yet generating high-quality, long-form content remains a significant challenge. This task, known as Open-ended Long Text Generation (Open-LTG), suffers from a fundamental evaluation bottleneck: there is no single "gold standard" for complex, subjective prompts like writing a comprehensive report on a scientific topic.
Traditional evaluation methods rely on "LLM-as-a-Judge," where a powerful model provides a general score. However, these general rewards are often noisy, inconsistent, and fail to capture the specific factual density required for high-quality long-form output. ProxyReward addresses this by shifting from general assessments to targeted, objective rewards.
The ProxyReward Framework
The ProxyReward framework decomposes the evaluation of complex text into a series of objective, verifiable components. The process is divided into three distinct stages:
1. ProxyReward Dataset Generation
Instead of relying on human experts, the framework utilizes an LLM to generate a massive training set. This involves creating "meta-questions" (complex, open-ended prompts) and a corresponding set of ~15 "proxy" question-answer pairs for each. These proxy pairs are simple, boolean (True/False) questions that serve as a checklist for what a high-quality response must contain.
2. Active Exploration
The model under training generates multiple diverse responses to the same meta-question, providing a variety of candidates for evaluation.
3. Automatic Assessment and Training
In this stage, a reward model (e.g., GPT-4o-mini) acts as a reader. It is tasked with answering the 15+ proxy questions using only the generated response as its source material. The final reward signal is calculated as the percentage of proxy questions correctly addressed by the response. This transforms a subjective quality judgment into an objective reading comprehension task. Finally, Direct Preference Optimization (DPO) is used to fine-tune the model toward maximizing this Proxy Accuracy score.
Experimental Results
ProxyReward was evaluated using the ProxyQA benchmark and the Web-based Question Answering (WQA) benchmark. The results demonstrate significant improvements over existing baselines:
- Performance Gains: Models trained with ProxyReward showed a relative performance increase of approximately 20% across both Qwen and Llama model families.
- Surpassing Proprietary Models: A Qwen2.5-7B model fine-tuned with ProxyReward achieved a score of 35.07% on the ProxyQA benchmark, notably outperforming GPT-4-Turbo (33.94%).
- Superiority over LLM-as-a-Judge: The targeted reward signal proved far more effective than general LLM-based judging, which often introduced noise that hindered model performance.
- Domain Sensitivity: The framework showed particularly strong results in scientific and medical domains, where factual accuracy is more easily verified through discrete proxy questions compared to more subjective humanities topics.
Conclusion and Future Work
ProxyReward provides a scalable, automated solution for the Open-LTG evaluation problem by converting subjective quality into objective, fine-grained checklists. While highly effective, the framework's reliance on a "teacher" LLM for proxy generation introduces potential teacher bias. Future research will focus on increasing the diversity of proxy questions and extending this targeted reward logic to creative and non-informational generation tasks.
Summary updated