A Sentiment Consolidation Framework for Meta-Review Generation

Miao Li, Jey Han Lau, Eduard Hovy

This paper explores how Large Language Models (LLMs) handle the complex task of "information consolidation"—specifically in the context of generating scientific meta-reviews. While LLMs are excellent at summarizing text, they often struggle to accurately aggregate conflicting opinions or follow the nuanced logic human experts use when weighing multiple viewpoints.

TL;DR

The authors propose a three-layer "Sentiment Consolidation Framework" that mimics how human meta-reviewers aggregate scientific reviews and discussions. By explicitly prompting LLMs to follow this hierarchical logic—identifying specific judgments, clustering them by facets like "Novelty" or "Soundness," and then fusing them—the researchers generated meta-reviews that were more sentiment-accurate and actually preferred by human experts over original human-written meta-reviews.

What Problem Does This Paper Address?

Summarizing a single document is a solved problem for modern LLMs. However, multi-document consolidation of opinionated information is much harder. In scientific peer review, a meta-reviewer must digest multiple reviews (which often conflict) and author rebuttals to reach a final judgment.

Current AI summarization often takes "shortcuts," such as simple majority voting or surface-level pattern matching, which fails in the scientific domain where one highly convincing negative point can outweigh three weak positive ones. The authors aim to move beyond simple "tl;dr" prompts toward a model that understands the underlying logic of expert judgment.

Methods

The researchers introduced a three-layer framework to guide the generation process:

  1. Input Layer: The raw source documents (official reviews and multi-turn discussions).
  2. Consolidation Layer: This is the core innovation. The model is prompted to:
  3. Extract specific judgments (e.g., "The method is novel").
  4. Categorize these into facets (Novelty, Soundness, Clarity, Advancement, Compliance, Overall).
  5. Assess Sentiment Level (from strongly negative to strongly positive) and Convincingness.
  6. Cluster and fuse these sentiments to form a final opinion for each facet.
  7. Generation Layer: The model writes the final meta-review based on the consolidated opinions from the previous layer.
Figure 1: The three-layer framework of the underlying information consolidation logic in meta-reviewing (P: Positive, P +: Strongly positive, N: Negative, N +: Strongly negative).

The authors tested two main prompting strategies: Prompt-Ours (a single complex prompt describing the steps) and Pipeline-Ours (a multi-step process where the output of one prompt serves as the input for the next).

Datasets

  • PeerSum: A large-scale dataset of scientific reviews, discussions, and meta-reviews.
  • Human-Annotated Benchmark: The authors recruited senior PhD students to annotate 30 samples (comprising over 1,800 individual judgments) to create a high-quality ground truth for how humans extract and categorize sentiments.

Key Insights and Ideas

  • Beyond Majority Voting: The study found that in nearly 24% of cases in the PeerSum dataset, the final meta-review decision did not follow the majority sentiment of the individual review ratings. This highlights why simple summarization fails—meta-reviewing requires weighing the strength of arguments, not just counting votes.
  • Facet Importance: "Soundness" and "Advancement" were identified as the most critical facets for meta-reviewers, while "Compliance" was rarely a deciding factor.
  • The Subjectivity of Convincingness: Interestingly, human agreement was lowest when judging how "convincing" a review was (see Figure 2 below). LLMs also struggled significantly with this, suggesting that "convincingness" is a highly subjective expert assessment that is difficult to automate.
Figure 2: Inter-annotator agreement on meta-reviews, official reviews and discussions in terms of Krippen- dorff’s α for different judgement components including Content Expression (CE), Sentiment Expression (SE), Review Facet (RF), Sentiment Level (SL), and Convinc- ingness Level (CL).

Key Results

The researchers evaluated their framework using both automated metrics and human preference:

  • Sentiment Accuracy: The proposed methods (Prompt-Ours and Pipeline-Ours) significantly outperformed standard "naive" prompting across all LLMs tested (GPT-4, GPT-3.5, LLaMA2-70B). The Pipeline-Ours approach was particularly effective at getting the overall sentiment correct.
  • Human Preference: In a head-to-head comparison, human annotators preferred GPT-4 meta-reviews generated with the new framework over human-written ones 73.3% to 26.7%.
  • Why AI Won: The authors noted that AI-generated meta-reviews tended to be more consistent and detailed across all review facets, whereas human meta-reviewers often had high variance in the amount of detail they provided.

Limitations and Future Work

  • Domain Bias: The data was pulled primarily from AI and machine learning conferences; the framework might need adjustment for other fields like medicine or social sciences.
  • Language: The study was limited to English-language documents.
  • Argumentation Logic: While the framework handles sentiment consolidation well, it does not yet fully capture the complex "argumentation" (the why behind a judgment) that is central to high-level peer review.
  • Prompting vs. Fine-Tuning: The current work relies on prompting existing models. The authors suggest that future work should involve fine-tuning models specifically on this consolidation logic.

Summary updated

Abstract

Modern natural language generation systems with Large Language Models (LLMs) exhibit the capability to generate a plausible summary of multiple documents; however, it is uncertain if they truly possess the capability of information consolidation to generate summaries, especially on documents with opinionated information. We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. To make scientific sentiment summarization more grounded, we hypothesize that human meta-reviewers follow a three-layer framework of sentiment consolidation to write meta-reviews. Based on the framework, we propose novel prompting methods for LLMs to generate meta-reviews and evaluation metrics to assess the quality of generated meta-reviews. Our framework is validated empirically as we find that prompting LLMs based on the framework -- compared with prompting them with simple instructions -- generates better meta-reviews.
BibTeX
@inproceedings{li2024sentiment,
  author = {Miao Li and Jey Han Lau and Eduard Hovy},
  title = {{A Sentiment Consolidation Framework for Meta-Review Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.547},
}
Figure