Causal inference with textual data: A quasi-experimental design assessing the association between author metadata and acceptance among ICLR submissions from 2017 to 2022

Chang Chen, Jiayao Zhang, Ting Ye, Dan Roth, Bo ZhangJournal of Causal Inference 2024 · DOI

Abstract: Many recent studies have probed status bias in the peer-review process of academic journals and conferences. In this article, we investigated the association between author metadata and area chairs’ final decisions (Accept/Reject) using our compiled database of 5,313 borderline submissions to the International Conference on Learning Representations from 2017 to 2022 under a matched observational study framework. We carefully defined elements in a cause-and-effect analysis, including the treatment and its timing, pre-treatment variables, potential outcomes (POs) and causal null hypothesis of interest, all in the context of study units being textual data and under Neyman and Rubin’s PO framework. We found some weak evidence that author metadata was associated with articles’ final decisions. We also found that, under an additional stability assumption, borderline articles from high-ranking institutions (top-30% or top-20%) were less favored by area chairs compared to their matched counterparts. The results were consistent in two different matched designs (odds ratio = 0.82 [95% confidence interval (CI): 0.67 to 1.00] in a first design and 0.83 [95% CI: 0.64 to 1.07] in a strengthened design) and most pronounced in the subgroup of articles with low ratings.

Introduction

Status bias—the idea that reviewers or editors favor authors from prestigious institutions—is a persistent concern in academic publishing. While many conferences use double-blind review, "leakage" often occurs through pre-prints on platforms like arXiv or through name recognition in niche sub-fields. This study investigates whether institutional prestige tips the scales for "borderline" papers at the International Conference on Learning Representations (ICLR), specifically focusing on the final decision stage managed by Area Chairs (ACs).

Methodology and Data Characterization

The researchers treated this problem as a quasi-experimental design using a matched observational study framework under the Neyman-Rubin Potential Outcomes (PO) model. To isolate the effect of institutional prestige, they sought to find "twins"—pairs of papers that were nearly identical in content, quality, and reviewer sentiment, but differed in the authors' institutional rankings.

Key to this approach was the use of advanced NLP techniques to control for high-dimensional covariates:

  • SPECTER Embeddings: Captured the semantic "topic" and meaning of paper abstracts.
  • RoBERTa Sentiment Analysis: Quantified the "vibe" of textual reviews beyond simple numerical scores.
  • Complexity Scores: A fluency-based metric used to proxy the technical or mathematical "heaviness" of the submission.
Distribution of review ratings (a) and semantic clustering of paper abstracts via SPECTER embeddings (b).

The study utilized a database of 10,289 ICLR submissions from 2017 to 2022. The analysis focused on a "borderline" subset of 5,313 submissions that received average reviewer ratings between 5 and 7. Institutional prestige was defined by the top 20% or 30% of institutions based on historical ICLR acceptance rates.

Review sentiment analysis (c) and complexity scores across different research categories (d).

Key Results: The "Reverse" Status Bias

Contrary to common expectations of prestige-based favoritism, the study found evidence of a "reverse" status bias for borderline papers. While the overall association between metadata and decisions was weak (Odds Ratios of 0.82–0.83), the effect was stark in the subgroup of papers with low ratings (average score < 6).

In this low-rating subgroup, borderline papers from prestigious institutions had roughly 40% lower odds of being accepted (Odds Ratios of 0.57–0.61) compared to matched papers from less prestigious institutions. However, for papers with average scores of 6 or higher, this institutional effect vanished, suggesting that once a paper reaches a certain quality threshold, prestige ceases to be a factor for Area Chairs.

Discussion and Implications

The authors propose the "Higher Bar" hypothesis to explain these findings. Area Chairs may hold authors from top-tier labs to a higher standard; a mediocre score for a famous lab might be viewed as a "disappointment," whereas the same score for an unknown lab might be seen as an "over-achievement."

While this suggests that the final decision stage at ICLR is relatively resilient to traditional status bias for borderline papers, the researchers emphasize that bias may still occur earlier in the process—at the reviewer stage—before the Area Chair's intervention. Future research is needed to apply this causal framework to the interaction between authors and reviewers to see if bias is more pronounced before the double-blind "veil" is potentially lifted.

Summary updated

Abstract

Abstract Many recent studies have probed status bias in the peer-review process of academic journals and conferences. In this article, we investigated the association between author metadata and area chairs’ final decisions (Accept/Reject) using our compiled database of 5,313 borderline submissions to the International Conference on Learning Representations from 2017 to 2022 under a matched observational study framework. We carefully defined elements in a cause-and-effect analysis, including the treatment and its timing, pre-treatment variables, potential outcomes (POs) and causal null hypothesis of interest, all in the context of study units being textual data and under Neyman and Rubin’s PO framework. We found some weak evidence that author metadata was associated with articles’ final decisions. We also found that, under an additional stability assumption, borderline articles from high-ranking institutions (top-30% or top-20%) were less favored by area chairs compared to their matched counterparts. The results were consistent in two different matched designs (odds ratio = 0.82 [95% confidence interval (CI): 0.67 to 1.00] in a first design and 0.83 [95% CI: 0.64 to 1.07] in a strengthened design) and most pronounced in the subgroup of articles with low ratings. We discussed how to interpret these results in the context of multiple interactions between a study unit and different agents (reviewers and area chairs) in the peer-review system.
BibTeX
@article{chen2024causal,
  author = {Chang Chen and Jiayao Zhang and Ting Ye and Dan Roth and Bo Zhang},
  title = {{Causal inference with textual data: A quasi-experimental design assessing the association between author metadata and acceptance among ICLR submissions from 2017 to 2022}},
  journal = {Journal of Causal Inference},
  year = {2024},
  doi = {10.1515/jci-2023-0052},
}
Figure