TL;DR
This paper introduces RAE, a novel framework for evaluating automatically generated public opinion reports. RAE addresses the challenge of assessing subjective qualities like "innovation" by using dynamically generated roles to simulate diverse stakeholder perspectives, leading to more robust and human-aligned evaluations.
What Problem Does This Paper Address?
Evaluating automatically generated public opinion reports, especially those used for crisis management, is crucial but challenging. Traditional metrics fall short, and current LLM-based evaluators often suffer from a single, static perspective. This is problematic because subjective qualities like "innovation" or "feasibility" are judged differently by various stakeholders (e.g., a government official vs. a public health expert). This discrepancy leads to evaluations that don't align with human expert judgment, potentially delaying critical decision-making during crises.
Methods
The core of RAE is its Role-based Adaptive Evaluation (RAE) framework. It tackles the evaluation problem through two main mechanisms:
- Adaptive Role-Play Mechanism:
- For dimensions with factual reference information (like Event Title or Causes), RAE uses a set of predefined, representative roles (e.g., "Netizen," "Journalist") to ensure consistent, multi-perspective evaluation.
- For highly open-ended dimensions (like "Innovation" or "Feasibility"), RAE dynamically generates context-specific roles. This is achieved by using an LLM to create roles that capture the unique concerns and priorities of relevant stakeholders for a given crisis context. To avoid redundancy, these generated roles undergo a deduplication process using MinHash and semantic clustering.
- Multi-Role Reasoning Aggregation:
- After roles evaluate the report, their judgments are aggregated. The paper explores three methods:
- Voting Aggregation: A simple majority vote based on the scores assigned by each role.
- Process Aggregation: An LLM judge synthesizes only the textual justifications from each role.
- Comprehensive Aggregation: An LLM judge considers both the scores and justifications from all roles.
The framework aims to maximize alignment with human expert consensus.
Datasets
The primary dataset used for evaluation is OPOR-Bench. This is an event-centric dataset containing 463 crisis events from 2012-2025, covering various crisis types (natural disasters, human-caused incidents). Each event includes multi-source documents, a structured reference summary, and LLM-generated reports.
Key Insights and Ideas
- The "Wisdom of Crowds" for Evaluation: The study strongly suggests that aggregating diverse, independent judgments (like in Voting Aggregation) is superior to relying on a single, centralized judge (like in Comprehensive Aggregation). This "wisdom of crowds" effect helps cancel out individual biases and random errors, leading to more robust evaluations.
- Dynamic Roles are Crucial for Subjectivity: For open-ended dimensions, dynamically generated roles are essential. Predefined roles or simply evaluating without roles fail to capture the nuanced, context-dependent perspectives needed for accurate assessment.
- Five is the Magic Number for Roles: The research found an optimal balance with approximately five dynamically generated roles. Too few roles miss crucial perspectives, while too many can introduce irrelevant ones, diluting the evaluation quality.
- Adaptive Strategy Balances Performance and Efficiency: Using predefined roles for factual dimensions and dynamically generated roles for subjective ones offers a good balance between comprehensive evaluation and computational efficiency.
Key Results
- Strong Human-Agent Agreement: RAE consistently achieved high alignment with human expert judgments, as measured by Spearman's ρ and Kendall's τ correlations, often surpassing established "strong" thresholds.
- Voting Aggregation Outperforms Others: The Voting Aggregation method consistently showed the best performance across various metrics and base models, outperforming Process and Comprehensive aggregation.
- RAE Beats Baselines: RAE significantly outperformed the OPOR-Eval baseline, demonstrating its effectiveness in capturing human evaluation patterns.
- Dynamic Roles are Key: Ablation studies showed substantial performance drops when dynamic roles were removed for open-ended dimensions, highlighting their critical importance.
- Case Study Illustrates Nuance: A case study on the "Innovation" dimension for the "Black Sea incident" showed how RAE's diverse roles identified that suggestions considered innovative by some stakeholders were conventional to others, leading to a more grounded score than a vanilla LLM evaluation.
Limitations and Future Work
- Computational Cost: The multi-role aggregation process is computationally more expensive than single-agent systems, which could be a barrier for resource-constrained or real-time applications.
- LLM Dependency: The performance of RAE is inherently tied to the capabilities of the underlying LLMs used for role simulation and evaluation. Biases or limitations in these LLMs could affect the evaluation quality.
- Scope: The current framework is primarily focused on text-based, English-language reports.
Future work aims to:
- Develop more parameter-efficient aggregation mechanisms and explore model distillation to reduce computational overhead.
- Enhance role simulation and calibration, including domain-aware constraints and consistency checks.
- Extend RAE to handle multimodal data (images, videos) and other languages and domains.
Summary updated