Geographic Citation Gaps in NLP Research

Mukund Rungta, Janvijay Singh, Saif M. Mohammad, Diyi Yang

Abstract

In a fair world, people have equitable opportunities to education, to conduct scientific research, to publish, and to get credit for their work, regardless of where they live. However, it is common knowledge among researchers that a vast number of papers accepted at top NLP venues come from a handful of western countries and (lately) China; whereas, very few papers from Africa and South America get published. Similar disparities are also believed to exist for paper citation counts. In the spirit of "what we do not measure, we cannot improve", this work asks a series of questions on the relationship between geographical location and publication success (acceptance in top NLP venues and citation impact). We first created a dataset of 70,000 papers from the ACL Anthology, extracted their meta-information, and generated their citation network. We then show that not only are there substantial geographical disparities in paper acceptance and citation but also that these disparities persist even when controlling for a number of variables such as venue of publication and sub-field of NLP. Further, despite some steps taken by the NLP community to improve geographical diversity, we show that the disparity in publication metrics across locations is still on an increasing trend since the early 2000s.

Introduction

In an ideal scientific ecosystem, the merit of an idea should be independent of its geographic origin. However, the Natural Language Processing (NLP) community has long observed an intuitive "geographic bias," where top-tier conferences are dominated by institutions in North America, Europe, and China. While these observations are common, quantitative data on how location correlates with long-term citation impact and publication success has been scarce. Guided by the principle that "what we do not measure, we cannot improve," this research quantifies the extent of these disparities and examines whether they are narrowing over time.

Methodology

To analyze these trends, a multi-step pipeline was developed to process the ACL Anthology (AA):

  1. Metadata Extraction: The authors utilized the SciPDF Parser to extract author affiliations from paper headers.
  2. Geographic Mapping: A heuristic-based approach was employed to map affiliations to countries, matching organization names against a curated database of 18,000 universities and global country lists.
  3. Citation Network Construction: The AA papers were linked with the Semantic Scholar API to build a comprehensive citation graph, allowing for the tracking of long-term impact.
  4. Inequality Metrics: The study utilized the Gini coefficient—a standard economic measure of inequality—to track the distribution of citation "wealth" and shifting collaboration patterns over two decades.

The AA Citation Corpus

The resulting dataset, the AA Citation Corpus, includes approximately 70,000 papers published up to January 2022. It contains paper titles, author IDs, venues, publication dates, and inferred countries of affiliation. Manual annotation of a 1,000-paper subset verified the geographic mapping with an 84.5% accuracy rate.

Key Insights and Results

The Widening Citation Gap

The data reveals a "rich get richer" phenomenon. While the gap in mean citations between the top-10 most and least cited countries is relatively small (~3.5) in the first two years post-publication, it balloons to ~23 after 15 years. This suggests that research from dominant regions (US, UK, Canada) enjoys significantly longer-lasting influence and visibility than research from other regions. Over the last 21 years, the overall gap in mean citation metrics across the top-10 countries has more than doubled, increasing from 6.31 to 15.11.

The Venue and Subfield Paradox

A critical finding is that publication venue does not act as a total equalizer. Even when controlling for prestigious venues like ACL or EMNLP, papers from the US consistently garner higher median citations than those from India or China. Similarly, the disparity persists across subfields; in areas like Sentiment Analysis, US-based papers maintain higher median citations despite similar publication volumes from other regions. This suggests that the geographic gap is not merely a result of different regions focusing on "niche" or less popular topics.

Geographic and Linguistic Disparities

The concentration of research output is stark: North America and Western Europe account for 65% of all papers in the corpus, while Africa, South America, and Southeast Asia combined account for only 3.7%. Furthermore, there is a tragic inverse correlation between a region's linguistic diversity and its NLP research output. Regions with the highest number of spoken languages, such as Africa, remain the most underrepresented in the global research conversation.

Collaboration Silos

While global collaboration is nominally increasing, it remains highly selective. North American researchers primarily collaborate with European counterparts. In contrast, the Gini coefficient for collaboration in Africa and South America has increased, indicating that these regions are becoming more isolated or selective in their partnership patterns rather than more integrated into the global network.

Discussion and Limitations

While this study provides a rigorous quantitative baseline, it faces certain limitations. The automated country extraction has a ~15% error rate, and the focus on the ACL Anthology introduces an English-language bias, potentially overlooking significant research published in local, non-English venues. Furthermore, the data quantifies the existence of the gap but cannot definitively prove its causes—whether they stem from systemic reviewer bias, funding inequities, or institutional prestige.

To address these systemic inequities, the authors call for the NLP community to adopt transparent reporting of geographic diversity statistics and urge the ACL to make location-associated metadata publicly available for future research.

Summary updated

Abstract

In a fair world, people have equitable opportunities to education, to conduct scientific research, to publish, and to get credit for their work, regardless of where they live. However, it is common knowledge among researchers that a vast number of papers accepted at top NLP venues come from a handful of western countries and (lately) China; whereas, very few papers from Africa and South America get published. Similar disparities are also believed to exist for paper citation counts. In the spirit of "what we do not measure, we cannot improve", this work asks a series of questions on the relationship between geographical location and publication success (acceptance in top NLP venues and citation impact). We first created a dataset of 70,000 papers from the ACL Anthology, extracted their meta-information, and generated their citation network. We then show that not only are there substantial geographical disparities in paper acceptance and citation but also that these disparities persist even when controlling for a number of variables such as venue of publication and sub-field of NLP. Further, despite some steps taken by the NLP community to improve geographical diversity, we show that the disparity in publication metrics across locations is still on an increasing trend since the early 2000s. We release our code and dataset here: https://github.com/iamjanvijay/acl-cite-net
BibTeX
@inproceedings{rungta2022geographic,
  author = {Mukund Rungta and Janvijay Singh and Saif M. Mohammad and Diyi Yang},
  title = {{Geographic Citation Gaps in NLP Research}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=CtfhU11nj5m},
}
Figure