EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Weicheng Ma, Samiha Datta, Lili Wang, Soroush Vosoughi

TL;DR

Researchers have introduced EnCBP, a new benchmark dataset designed to predict the cultural background of English speakers at both the country and US state levels. By incorporating these finer-grained cultural features, the authors demonstrate significant performance gains across a variety of downstream NLP tasks, including sentiment analysis and named entity recognition.

What Problem Does This Paper Address?

While we know that culture shapes how we express ourselves, most NLP research treats culture as a monolith—often grouping entire hemispheres together (e.g., "Western" vs. "Eastern") or assuming everyone speaking the same language belongs to the same cultural group. This coarse-grained approach ignores the nuanced linguistic differences between, say, a journalist in India versus one in the UK, or a resident of New York versus one in Texas.

The authors argue that this lack of cultural awareness limits model performance and ignores the rich diversity of mono-lingual contexts. To bridge this gap, they tackle two main challenges:

  1. Data Scarcity: Creating a dataset that isolates cultural writing style from the specific topics being discussed.
  2. Utility: Proving that modeling these subtle stylistic differences actually helps models perform better on standard NLP benchmarks.

Methods

The authors constructed the EnCBP (English Cultural Background Prediction) dataset by sampling news articles from major outlets across five countries (US, UK, Canada, Australia, India) and four US states (NY, CA, GA, TX).

To ensure the models learned style rather than just facts, they employed several clever controls:

  • Topic Control: They limited articles to five controversial topics (e.g., "global warming," "mandatory vaccination") to ensure the vocabulary was comparable across regions.
  • Entity Masking: They replaced specific mentions of countries and districts with generic tokens (e.g., [country]) and filtered out regional named entities using Inverse Document Frequency (IDF) scores to prevent the model from "cheating" by recognizing local names.
  • Human Validation: They used Amazon Mechanical Turk to have local residents identify if a paragraph "sounded" like it came from their home region.

For modeling, they benchmarked standard architectures like BiLSTM, BERT, and RoBERTa. They then tested two methods for "injecting" this cultural knowledge into downstream tasks: Two-Stage Training (fine-tuning on EnCBP before the target task) and Multi-Task Learning (MTL).

Datasets

The paper introduces two new subsets:

  • EnCBP-country: 10,000 paragraphs across 5 countries.
  • EnCBP-district: 8,000 paragraphs across 4 US states.
Table 1: Number of documents associated with each label and under each topic in EnCBP. For each country or district label, the documents under each topic are randomly sampled into the training, development, and test sets with a 80%/10%/10% split.

They evaluated the utility of these datasets on nine existing benchmarks, including CoNLL-2003 (NER), SST-5 (Sentiment Analysis), and various semantic tasks like PAWS-Wiki and QNLI.

Key Insights and Ideas

  • Identifiable Styles: Human validators were significantly better than random chance at identifying their own region's writing style, even with location names masked. This confirms that cultural background leaves a distinct "fingerprint" on syntax and word choice.
  • Linguistic Compatibility: Quantitative analysis showed that the Indian (IND) corpus is more compatible with the UK corpus than others, likely due to historical and educational ties. Similarly, "liberal" states (NY, CA) shared more stylistic similarities with each other than with "conservative" states (GA, TX).
  • Task-Specific Benefits: Cultural modeling is a "power-up" for tasks involving human subjectivity. Sentiment Analysis (SA) and Named Entity Recognition (NER) saw the biggest boosts. However, for purely semantic tasks like Paraphrase Identification, the stylistic variation sometimes acted as noise, slightly dampening the gains.
  • The Domain Gap: The authors found that cultural augmentation failed on the Go-Emotions dataset (Reddit comments). They attribute this to a "domain conflict"—models trained on formal news style (EnCBP) struggle to generalize to the highly informal, US-centric world of Reddit.

Key Results

The roberta-base model emerged as the top performer for the prediction task, achieving an F1-macro of 82.96 on EnCBP-country and 73.96 on EnCBP-district.

Table 3: Benchmark performance of BiLSTM, BERT, and RoBERTa.

When using EnCBP for feature augmentation (Table 5), the results were compelling:

  • NER (CoNLL-2003): Performance jumped from 91.73 to 94.41 F1-macro using two-stage training.
  • Sentiment Analysis (SST-5): Accuracy improved from 52.41 to 54.54.
  • Entailment (RTE): Accuracy rose from 64.69 to 68.45.
Table 5: The performance of BERT model without cultural feature augmentation (BERT-orig), and models with cultural feature augmentation via two-stage training and multi-task learning. EnCBP-country is used as the auxil- iary dataset. We report accuracy for QNLI, RTE, and SST-2, Pearson’s and Spearman’s correlations for STS-B, and F1-macro for the other tasks. The average score and standard deviation (in parentheses) in five runs with different random seeds are reported for each experiment

Limitations and Future Work

The primary limitation is domain sensitivity. Because EnCBP is built on news text, its benefits are most pronounced when the downstream task is also somewhat formal. The failure on the Go-Emotions dataset highlights that cultural markers in news don't perfectly translate to social media.

The authors suggest that future work should focus on social media writings, which likely contain even finer-grained cultural distinctions but are significantly harder to annotate and validate due to their noisy nature. Additionally, they note that the size of the auxiliary data matters; while EnCBP is sufficient, there is a point of diminishing returns where more data provides only marginal gains.

Summary updated

Abstract

While cultural backgrounds have been shown to affect linguistic expressions, existing natural language processing (NLP) research on culture modeling is overly coarse-grained and does not examine cultural differences among speakers of the same language. To address this problem and augment NLP models with cultural background features, we collect, annotate, manually validate, and benchmark EnCBP, a finer-grained news-based cultural background prediction dataset in English. Through language modeling (LM) evaluations and manual analyses, we confirm that there are noticeable differences in linguistic expressions among five English-speaking countries and across four states in the US. Additionally, our evaluations on nine syntactic (CoNLL-2003), semantic (PAWS-Wiki, QNLI, STS-B, and RTE), and psycholinguistic tasks (SST-5, SST-2, Emotion, and Go-Emotions) show that, while introducing cultural background information does not benefit the Go-Emotions task due to text domain conflicts, it noticeably improves deep learning (DL) model performance on other tasks. Our findings strongly support the importance of cultural background modeling to a wide variety of NLP tasks and demonstrate the applicability of EnCBP in culture-related research.
BibTeX
@inproceedings{ma2022encbp,
  author = {Weicheng Ma and Samiha Datta and Lili Wang and Soroush Vosoughi},
  title = {{EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.221},
}
Figure