Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction

Panitan Muangkammuen, Fumiyo Fukumoto, Jiyi Li, Yoshimi SuzukiEMNLP 2022 · DOI

Introduction

The peer-review process is fundamental to scientific progress, ensuring the quality and integrity of published research. However, the exponential growth in paper submissions has placed an immense burden on reviewers and editors. To mitigate this, researchers have developed Predicting Aspect Scores (PASP) systems. These automated tools estimate scores for specific quality dimensions, such as "clarity," "originality," or "impact," providing valuable feedback to authors and assisting editors in decision-making.

A significant bottleneck for PASP is the scarcity of labeled data. While millions of scientific papers are publicly accessible, datasets containing papers paired with their official review scores (like PeerRead) are small due to confidentiality. Standard supervised learning often fails in this low-data regime, leading to overfitting.

The $\Gamma$-Trans Approach

To address data sparsity, researchers from the University of Yamanashi introduced $\Gamma$-Trans, a semi-supervised learning (SSL) framework designed to leverage both limited labeled reviews and abundant unlabeled scientific literature.

Architecture and Methodology

$\Gamma$-Trans is built upon the $\Gamma$-model, a variant of the Ladder Network architecture, integrated with a pre-trained Transformer backbone (e.g., BERT or SciBERT). The model operates through two parallel processing paths:

  1. Clean Path: Processes the input text normally to generate hidden representations.
  2. Corrupted Path: Introduces Gaussian noise to the input embeddings, simulating a "noisy" version of the data.

The model is trained to perform a dual task. It must predict the correct review scores for labeled data (Supervised Cost) while simultaneously learning to "denoise" the corrupted representations by reconstructing the clean versions (Unsupervised Denoising Cost).

The $\Gamma$-Trans architecture, featuring parallel clean and corrupted paths for semi-supervised fine-tuning.

Unlike previous SSL methods that used fixed Transformer features, $\Gamma$-Trans fine-tunes the Transformer's internal weights end-to-end. This allows the model to adapt its contextual understanding specifically for the domain of scientific peer review.

Experimental Setup

The researchers evaluated $\Gamma$-Trans using the PeerRead dataset (ACL subset) for labeled data, focusing on binary classification of aspect scores (Positive $\ge 4$, Negative $< 4$). For unlabeled data, they utilized 1,000 papers from the ScisummNet Corpus to provide raw text for the unsupervised denoising task.

Results and Analysis

$\Gamma$-Trans demonstrated superior performance across all evaluated metrics and aspects compared to standard supervised baselines and other SSL techniques like Virtual Adversarial Training (VAT).

Comparison of Accuracy and F1 scores across different aspects for supervised and semi-supervised methods.

As shown in the results, $\Gamma$-Trans achieved an average accuracy of 0.714 and an F1 score of 0.767, significantly outperforming the supervised BERT baseline (0.559 Accuracy, 0.611 F1).

Key Findings

  • Domain-Specific Backbones: Utilizing SciBERT as the underlying Transformer provided the best results, highlighting the importance of domain-specific pre-training in scientific NLP tasks.
  • Aspect-Specific Performance: The model was particularly effective at identifying "Impact," likely due to distinct linguistic markers associated with high-impact research.
  • The 512-Token Limit: Performance was lower for "Meaningful Comparison." This is attributed to the standard Transformer's 512-token limit, which often excludes the "Related Work" section where comparisons are typically discussed.

Conclusion and Future Work

$\Gamma$-Trans proves that semi-supervised fine-tuning of Transformers is a highly effective strategy for peer-review score prediction when labeled data is limited. By learning from the structure of unlabeled scientific text, the model achieves more robust and accurate predictions. Future research will focus on handling longer document sequences more efficiently and moving toward generating explainable textual reviews alongside numerical scores.

Reference: Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2022). Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction. Findings of the Association for Computational Linguistics: EMNLP 2022.

Summary updated

BibTeX
@inproceedings{muangkammuen2022exploiting,
  author = {Panitan Muangkammuen and Fumiyo Fukumoto and Jiyi Li and Yoshimi Suzuki},
  title = {{Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.164},
}
Figure