EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Weicheng Ma, Samiha Datta, Lili Wang, Soroush Vosoughi

Introduction

In the current landscape of Natural Language Processing (NLP), language is often treated as a monolith. Models trained on English are typically expected to perform uniformly across all English-speaking populations. When cultural context is considered, it is usually at a very coarse level—categorizing users by official language or broad geopolitical groupings (e.g., "Western" vs. "Eastern"). However, this approach overlooks the nuanced "linguistic fingerprints" that distinguish speakers within the same language. A journalist in Texas, for instance, employs different stylistic choices, sentence structures, and tones than a journalist in London or New Delhi.

To address this, researchers at Dartmouth have introduced EnCBP (English Cultural Background Prediction), a new benchmark dataset designed to predict cultural backgrounds at a finer grain, such as distinguishing between specific US states or different English-speaking countries. By modeling these subtle cultural variations, the researchers demonstrate that NLP models can achieve higher accuracy and sensitivity across a range of standard tasks.

The EnCBP Dataset

The EnCBP dataset was constructed to capture stylistic cultural markers rather than simple keyword associations. It consists of two primary subsets:

  1. EnCBP-country: Covers five countries (United States, United Kingdom, Canada, Australia, and India).
  2. EnCBP-district: Focuses on four US states (New York, California, Georgia, and Texas).

Each category contains 2,000 paragraph-level instances. To ensure the models learned cultural style rather than just identifying place names, the researchers replaced specific location mentions with generic tokens (e.g., [country]). Furthermore, the data collection focused on five consistent, frequently discussed topics to ensure that the subject matter remained a constant variable across different regions.

Human Validation

To verify that these cultural distinctions are indeed detectable in writing style, the researchers employed Amazon Mechanical Turk to have local residents identify text from their own regions. The results confirmed that locals could identify their "home" text based purely on stylistic cues, validating the premise that distinct cultural linguistic patterns exist and are perceptible.

Methodology

The study explored how this cultural knowledge could be integrated into existing transformer-based models like BERT and RoBERTa. Two primary methods of "feature augmentation" were tested:

  • Two-Stage Training: The model is first fine-tuned to predict the cultural background of a text using EnCBP. Subsequently, it is fine-tuned on a target downstream task (e.g., sentiment analysis).
  • Multi-Task Learning (MTL): The model is trained simultaneously on both the cultural background prediction and the target NLP task.

Experimental Results

Benchmark Performance

RoBERTa-base emerged as the most effective model for identifying cultural backgrounds. It achieved an F1-macro score of 82.96 for country-level prediction and 73.96 for US state-level prediction. Perplexity analysis further supported these results; models trained on a specific country's text showed significantly lower perplexity when reading new text from that same country compared to others, proving that "English" is not a singular domain.

Downstream Task Improvements

Integrating EnCBP data led to measurable gains across nine standard NLP benchmarks:

  • Named Entity Recognition (NER): On the CoNLL-2003 task, two-stage training improved BERT’s performance from 91.73 to 94.41.
  • Sentiment Analysis: On the SST-5 task, cultural augmentation increased performance from 52.41 to 54.54.
  • Semantic Tasks: Improvements were also noted in tasks such as PAWS-Wiki, QNLI, and STS-B.

Discussion and Insights

The Ideological Influence

The researchers observed that models found "liberal" states (New York, California) to have more consistent writing styles with each other, a pattern mirrored by "conservative" states (Georgia, Texas). This suggests that political and ideological cultures are deeply embedded in linguistic choices, influencing how information is framed and expressed.

Domain Sensitivity

While EnCBP provided significant boosts for news-based and formal text, it slightly hindered performance on Go-Emotions, which utilizes Reddit data. This suggests a "domain conflict"; the cultural markers of professional journalism do not perfectly align with the informal, noisy cultural expressions found on social media. This highlights that cultural knowledge is highly sensitive to the medium of communication.

Conclusion and Future Work

EnCBP demonstrates that accounting for fine-grained cultural backgrounds can significantly improve the performance of English NLP models. However, the current reliance on news articles limits the transferability of these models to more casual domains. Future research will focus on expanding these datasets to include social media and creative writing, which may reflect even more diverse and representative cultural distinctions. By moving beyond a monolithic view of language, NLP can become more inclusive and accurate for users across the global English-speaking community.

Summary updated

Abstract

While cultural backgrounds have been shown to affect linguistic expressions, existing natural language processing (NLP) research on culture modeling is overly coarse-grained and does not examine cultural differences among speakers of the same language. To address this problem and augment NLP models with cultural background features, we collect, annotate, manually validate, and benchmark EnCBP, a finer-grained news-based cultural background prediction dataset in English. Through language modeling (LM) evaluations and manual analyses, we confirm that there are noticeable differences in linguistic expressions among five English-speaking countries and across four states in the US. Additionally, our evaluations on nine syntactic (CoNLL-2003), semantic (PAWS-Wiki, QNLI, STS-B, and RTE), and psycholinguistic tasks (SST-5, SST-2, Emotion, and Go-Emotions) show that, while introducing cultural background information does not benefit the Go-Emotions task due to text domain conflicts, it noticeably improves deep learning (DL) model performance on other tasks. Our findings strongly support the importance of cultural background modeling to a wide variety of NLP tasks and demonstrate the applicability of EnCBP in culture-related research.
BibTeX
@inproceedings{ma2022encbp,
  author = {Weicheng Ma and Samiha Datta and Lili Wang and Soroush Vosoughi},
  title = {{EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.221},
}
Figure