Abstract
The rapid growth in academic submissions has placed an immense burden on Area Chairs (ACs) to synthesize diverse reviewer feedback. Standard multi-document summarization (MDS) techniques often prioritize consensus, potentially overlooking critical unique critiques. We present GLIMPSE, a novel extractive summarization tool based on the Rational Speech Act (RSA) framework. GLIMPSE identifies both shared agreements and divergent opinions, providing highly discriminative yet concise summaries that outperform large language models in information density.
Introduction
The academic peer review system is facing a scalability crisis. Conferences like ICLR and ACL have seen submission volumes increase fivefold since 2017. Area Chairs must navigate multiple lengthy reviews to reach informed decisions. While standard MDS tools exist, they are typically optimized for finding the "centroid" or consensus of a document set. In peer review, however, a single reviewer's unique observation—such as a specific technical flaw—can be more important than points of general agreement. GLIMPSE addresses this by focusing on "discriminative summarization," ensuring that the full spectrum of feedback is captured.
Methodology: Pragmatic Summarization via RSA
GLIMPSE frames the summarization task as a "reference game" within the Rational Speech Act (RSA) framework. It models a speaker attempting to describe a specific review such that a listener can uniquely identify it from a set. To achieve this, GLIMPSE calculates two primary metrics for sentence extraction:
- Pragmatic-speaker-based score: Quantifies how informative a sentence is for identifying its specific source document.
- Uniqueness score: Measures the degree to which a sentence belongs to only one review, calculated as the divergence from a uniform distribution across all documents.
By remaining strictly extractive, GLIMPSE ensures high attributability. Unlike abstractive models (e.g., LLMs), which may hallucinate or conflate viewpoints, GLIMPSE pulls exact sentences, allowing ACs to trace feedback directly to its source.
Results and Evaluation
The model was evaluated on a dataset of over 28,000 ICLR reviews. The primary metric, discriminativeness, measures how effectively a summary allows a listener model to identify the source review.
As shown in Figure 3, while the Llama 7B Instruct model achieves the highest raw discriminativeness (0.76), it does so at the cost of extreme verbosity, averaging nearly 2,000 characters. In contrast, GLIMPSE-Unique achieves a high discriminativeness of 0.66 using only 111 characters on average. When normalized by length, GLIMPSE-Speaker and GLIMPSE-Unique significantly outperform all baselines, including LLMs, demonstrating superior information density.
Figure 4 further illustrates the trade-off between discriminativeness and language model perplexity. GLIMPSE variants occupy a favorable position on the Pareto front, balancing informativeness with the constraints of the source text. Human evaluations further validated these findings: expert annotators were able to correctly identify source reviews from GLIMPSE summaries with 93.75% accuracy, surpassing the 85.18% accuracy achieved with Llama-generated summaries.
Discussion and Limitations
While GLIMPSE offers a powerful tool for efficient review synthesis, it is not without limitations. As an extractive method, it can occasionally select "unique" but uninformative filler sentences (e.g., "Minor comments:"). Furthermore, the use of meta-reviews as a gold standard for ROUGE evaluation is imperfect, as meta-reviews often serve to justify a decision rather than provide a neutral summary. Future research will focus on task-based evaluations to measure GLIMPSE's impact on actual Area Chair decision-making workflows.
Conclusion
GLIMPSE demonstrates that pragmatic, RSA-based extractive summarization can provide a highly efficient alternative to generative models for specialized domains like peer review. By prioritizing discriminativeness and conciseness, it offers a scalable solution for managing the increasing volume of scholarly communication.
Summary updated