RAE: A Role-Based Adaptive Framework for Evaluating Automatically Generated Public Opinion Reports

Jinzheng Yu, Yang Xu, Yifan Feng, Ligu Zhu, Hao Shen, Lei ShiElectronics 2026 · DOI

TL;DR

This paper introduces RAE, a novel framework for evaluating automatically generated public opinion reports. RAE addresses the challenge of assessing subjective qualities like "innovation" by using dynamically generated roles to simulate diverse stakeholder perspectives, leading to more robust and human-aligned evaluations.

What Problem Does This Paper Address?

Evaluating automatically generated public opinion reports, especially those used for crisis management, is crucial but challenging. Traditional metrics fall short, and current LLM-based evaluators often suffer from a single, static perspective. This is problematic because subjective qualities like "innovation" or "feasibility" are judged differently by various stakeholders (e.g., a government official vs. a public health expert). This discrepancy leads to evaluations that don't align with human expert judgment, potentially delaying critical decision-making during crises.

Methods

The core of RAE is its Role-based Adaptive Evaluation (RAE) framework. It tackles the evaluation problem through two main mechanisms:

  1. Adaptive Role-Play Mechanism:
    • For dimensions with factual reference information (like Event Title or Causes), RAE uses a set of predefined, representative roles (e.g., "Netizen," "Journalist") to ensure consistent, multi-perspective evaluation.
  2. For highly open-ended dimensions (like "Innovation" or "Feasibility"), RAE dynamically generates context-specific roles. This is achieved by using an LLM to create roles that capture the unique concerns and priorities of relevant stakeholders for a given crisis context. To avoid redundancy, these generated roles undergo a deduplication process using MinHash and semantic clustering.
  3. Multi-Role Reasoning Aggregation:
    • After roles evaluate the report, their judgments are aggregated. The paper explores three methods:
  4. Voting Aggregation: A simple majority vote based on the scores assigned by each role.
  5. Process Aggregation: An LLM judge synthesizes only the textual justifications from each role.
  6. Comprehensive Aggregation: An LLM judge considers both the scores and justifications from all roles.

The framework aims to maximize alignment with human expert consensus.

Datasets

The primary dataset used for evaluation is OPOR-Bench. This is an event-centric dataset containing 463 crisis events from 2012-2025, covering various crisis types (natural disasters, human-caused incidents). Each event includes multi-source documents, a structured reference summary, and LLM-generated reports.

Key Insights and Ideas

  • The "Wisdom of Crowds" for Evaluation: The study strongly suggests that aggregating diverse, independent judgments (like in Voting Aggregation) is superior to relying on a single, centralized judge (like in Comprehensive Aggregation). This "wisdom of crowds" effect helps cancel out individual biases and random errors, leading to more robust evaluations.
  • Dynamic Roles are Crucial for Subjectivity: For open-ended dimensions, dynamically generated roles are essential. Predefined roles or simply evaluating without roles fail to capture the nuanced, context-dependent perspectives needed for accurate assessment.
  • Five is the Magic Number for Roles: The research found an optimal balance with approximately five dynamically generated roles. Too few roles miss crucial perspectives, while too many can introduce irrelevant ones, diluting the evaluation quality.
  • Adaptive Strategy Balances Performance and Efficiency: Using predefined roles for factual dimensions and dynamically generated roles for subjective ones offers a good balance between comprehensive evaluation and computational efficiency.

Key Results

  • Strong Human-Agent Agreement: RAE consistently achieved high alignment with human expert judgments, as measured by Spearman's ρ and Kendall's τ correlations, often surpassing established "strong" thresholds.
  • Voting Aggregation Outperforms Others: The Voting Aggregation method consistently showed the best performance across various metrics and base models, outperforming Process and Comprehensive aggregation.
  • RAE Beats Baselines: RAE significantly outperformed the OPOR-Eval baseline, demonstrating its effectiveness in capturing human evaluation patterns.
  • Dynamic Roles are Key: Ablation studies showed substantial performance drops when dynamic roles were removed for open-ended dimensions, highlighting their critical importance.
  • Case Study Illustrates Nuance: A case study on the "Innovation" dimension for the "Black Sea incident" showed how RAE's diverse roles identified that suggestions considered innovative by some stakeholders were conventional to others, leading to a more grounded score than a vanilla LLM evaluation.

Limitations and Future Work

  • Computational Cost: The multi-role aggregation process is computationally more expensive than single-agent systems, which could be a barrier for resource-constrained or real-time applications.
  • LLM Dependency: The performance of RAE is inherently tied to the capabilities of the underlying LLMs used for role simulation and evaluation. Biases or limitations in these LLMs could affect the evaluation quality.
  • Scope: The current framework is primarily focused on text-based, English-language reports.

Future work aims to:

  • Develop more parameter-efficient aggregation mechanisms and explore model distillation to reduce computational overhead.
  • Enhance role simulation and calibration, including domain-aware constraints and consistency checks.
  • Extend RAE to handle multimodal data (images, videos) and other languages and domains.

Summary updated

Abstract

Public Opinion Reports are essential tools for crisis management, yet their evaluation remains a critical bottleneck that often delays response actions. Recently, dominant Large Language Model (LLM)-based evaluators often overlook a critical challenge: highly open-ended dimensions such as “innovation” and “feasibility” require synthesizing diverse stakeholder perspectives, as different groups judge these qualities from fundamentally different perspectives. Motivated by this, we propose the Role-based Adaptive Evaluation (RAE) framework. This framework employs an adaptive mechanism leveraging multi-perspective evaluation insights through role-based analysis, and further introduces dynamically generated roles tailored to specific contexts for these dimensions. RAE further incorporates multi-role reasoning aggregation to minimize individual biases and enhance evaluation robustness. Extensive experiments demonstrate that RAE significantly improves alignment with human expert judgments, especially on challenging highly open-ended dimensions.
BibTeX
@inproceedings{yu2026rae,
  author = {Jinzheng Yu and Yang Xu and Yifan Feng and Ligu Zhu and Hao Shen and Lei Shi},
  title = {{RAE: A Role-Based Adaptive Framework for Evaluating Automatically Generated Public Opinion Reports}},
  booktitle = {Electronics},
  year = {2026},
  doi = {10.3390/electronics15020380},
}
Figure