TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages

Anirudh Srinivasan, Eunsol Choi

Introduction

Politeness is a fundamental "pragmatic" aspect of human communication. It transcends literal meaning, serving as a tool to navigate social relationships, manage "face," and establish social distance. Despite its importance, Natural Language Processing (NLP) research into politeness has historically been centered on English. This linguistic bias leaves a significant gap in our understanding of whether computational models can generalize politeness across diverse cultures and whether "politeness" is distinct from other stylistic markers like "formality."

To address this, researchers have introduced TyDiP, the first dataset for politeness classification spanning nine typologically diverse languages: Hindi (hi), Korean (ko), Spanish (es), Tamil (ta), French (fr), Vietnamese (vi), Russian (ru), Afrikaans (af), and Hungarian (hu).

The TyDiP Dataset and Methodology

Unlike many multilingual datasets that rely on machine translation—which often produces "translationese" and strips away subtle pragmatic markers—TyDiP was built using native data. The researchers scraped 4,500 requests (500 per language) from Wikipedia User Talk pages. This source provides a rich environment of natural, collaborative-professional interactions where politeness is frequently negotiated.

To ensure high-quality labels, the study employed a "model-in-the-loop" annotation process. Crowdworkers were vetted through qualification tasks that tested their ability to translate and their alignment with high-confidence model predictions. This rigorous approach ensured that the resulting labels reflected genuine native-speaker intuition.

Linguistic Strategies and Cultural Nuance

A central question in sociolinguistics is whether politeness strategies are universal. By using word-alignment tools to map English politeness markers (like "Please" or "Hedges") into target languages, the researchers analyzed how these strategies influence perceived politeness globally.

Correlation of linguistic strategies with politeness.

The findings suggest that while some markers, such as "Please," are almost universally associated with politeness, others are highly culture-specific. For instance, "indirect greetings" were found to have a significantly stronger positive impact on politeness scores in Korean and Tamil compared to English. Conversely, starting a request with a second-person pronoun often served as a marker of directness or rudeness in several languages.

Key Insights: Politeness vs. Formality

The research highlights several critical distinctions in how humans and machines perceive social language:

  1. The Fragility of Translation: Classifying politeness by first translating a foreign request into English (the "translate-test" approach) consistently decreased performance. Machine translation often fails to preserve honorifics or indirect phrasing, which are the very signals models need to identify respect.
  2. Formality is Not Politeness: Transfer experiments demonstrated that a model trained to detect formality is a poor substitute for a politeness classifier. While formal language is often perceived as polite, one can be formally rude or informally polite. Politeness is more deeply tied to the intent and content of a request, whereas formality is a matter of style.
  3. Native Consensus: Annotators showed much higher agreement when rating requests in their native language than when rating English requests, suggesting that our "politeness sensors" are finely tuned to our specific cultural contexts.

Key Results and the Human-Machine Gap

The study evaluated the zero-shot capabilities of modern multilingual models, specifically fine-tuning XLM-RoBERTa on the English Stanford Politeness Corpus and testing it on the nine TyDiP languages.

Human vs. XLM-R performance across languages.

While XLM-RoBERTa achieved a respectable average accuracy of 80.8%, it still lags significantly behind human performance. Native speakers achieved an estimated accuracy of approximately 90%, leaving a performance gap of over 10% in the majority of the languages studied. Large Language Models like GPT-3 (Davinci-002) also struggled in comparison, reaching only 70.6% accuracy in a zero-shot prompting setup.

Conclusion and Future Directions

TyDiP represents a major step toward culturally aware NLP, but challenges remain. The current dataset is limited to the professional-collaborative domain of Wikipedia. Future research should expand into more diverse social settings, such as retail or family environments, and focus on building "politeness-aware" machine translation systems that can preserve social nuances across linguistic boundaries.

Summary updated

BibTeX
@inproceedings{srinivasan2022tydip,
  author = {Anirudh Srinivasan and Eunsol Choi},
  title = {{TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.420},
}
Figure