Uncertainty Aware Review Hallucination for Science Article Classification

Korbinian Friedl, Georgios Rizos, Lukas Stappen, Madina Hasan, Lucia Specia, Thomas Hain, Björn SchullerACL 2021

TL;DR

This paper tackles the challenge of building AI systems to assist in scientific peer review by proposing a novel framework that accounts for reviewer disagreement and models uncertainty. It introduces a "hallucination" technique to leverage review information during training while still making predictions without explicit reviews at test time, improving classification performance.

What Problem Does This Paper Address?

The process of scientific peer review is notoriously subjective and time-consuming. Significant disagreement among reviewers can make the area chair's job difficult and even lead to questioning of the final decision. While machine learning can offer decision support, existing approaches often fail to:

  • Effectively use both reviewer and area chair recommendations, which can sometimes conflict.
  • Explicitly model the inherent subjectivity of individual submissions.
  • Operate realistically, as they often assume review texts are available at test time, when in practice, this is precisely the information that might be missing.

This paper aims to bridge this gap by developing a more robust and realistic AI assistant for scientific article classification.

Methods

The authors propose a multi-faceted approach:

  • Modeling Disagreement and Subjectivity: They acknowledge that both area chair and reviewer recommendations contain valuable, complementary information. Instead of simply averaging them, they propose an interpolation between "hard" (area chair) and "soft" (reviewer average) labels to capture this. Crucially, they introduce a method to explicitly model aleatory uncertainty – the inherent randomness or subjectivity in a submission's review – as an auxiliary prediction task. This helps the model avoid being overly penalized for misclassifying subjective papers with conflicting reviews.
  • Modality Hallucination: To address the realistic scenario where review texts might not be available at test time, they employ a "hallucination" technique. Inspired by methods that impute missing modalities in other domains, their model learns to predict review representations based on the abstract text during training. This allows the system to utilize review information for learning but then generate these representations "on the fly" at test time, effectively simulating the presence of reviews.
  • Two-Stage Training: The model is trained in two stages. First, it learns meaningful review representations using only the review texts. Second, it incorporates the hallucination loss and the classification loss, using the true review representations as an auxiliary target to train the hallucinated ones. This avoids "teacher forcing" and allows the model to learn to predict based on hallucinated representations.

Datasets

The study utilizes the Interspeech 2019 submission corpus, which is described as the largest available dataset of its kind. It comprises 2179 preprocessed academic submissions, 5842 reviews, and corresponding acceptance decisions and reviewer scores. The dataset is balanced in terms of acceptances and rejections (50.2% acceptances, 49.8% rejections) and is split into 80-10-10 train-validation-test sets.

Key Insights and Ideas

  • Complementary Signals: Area chair and reviewer recommendations are not redundant; they offer distinct and valuable information for classification.
  • Uncertainty Matters: Simply averaging soft labels isn't enough to handle subjective papers. Explicitly modeling aleatory uncertainty is crucial for better performance.
  • Hallucination for Realism: The "modality hallucination" approach is a clever way to leverage rich review data during training while maintaining a realistic test-time scenario where reviews might be absent.
  • Uncertainty and Hallucination Synergy: The uncertainty modeling techniques (especially GTU and HLA) benefit significantly from the hallucinated review representations, suggesting a strong interplay between these two components.

Key Results

  • The proposed model, particularly when using the GTU (Ground Truth Uncertainty) loss and a soft-hard interpolation weight of 0.8, significantly outperforms a baseline that only uses abstract information.
  • The uncertainty-aware methods (GTU and HLA) show statistically significant improvements over hard labels when using abstracts and reviews, and also over abstract-only approaches when combined with hallucinated reviews.
  • The hallucination approach, especially with the HLA method, demonstrates a notable improvement over using only the abstract, indicating that imputed review representations can effectively bridge the performance gap.
  • The Pearson Correlation Coefficient between predicted uncertainty and the standard deviation of reviewer recommendations is higher when reviews are used (or hallucinated) compared to abstract-only models, suggesting the model is learning meaningful uncertainty signals.

Limitations and Future Work

  • Performance Gap: Despite improvements, there remains a performance gap compared to using true review representations at test time.
  • Self-Supervision: The authors plan to explore self-supervision methods to further close this gap.
  • Active Learning: Investigating the use of predicted uncertainty to request additional reviewers for borderline cases is a promising future direction.
  • Full Paper Modeling: Exploring the impact of using the full paper text (instead of just the abstract) for modeling and hallucination is also suggested.
  • Analysis of Uncertainties: A deeper analysis of the predicted uncertainties across different loss functions is planned.

Summary updated

BibTeX
@inproceedings{friedl2021uncertainty,
  author = {Korbinian Friedl and Georgios Rizos and Lukas Stappen and Madina Hasan and Lucia Specia and Thomas Hain and Björn Schuller},
  title = {{Uncertainty Aware Review Hallucination for Science Article Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure