Who Is Missing? Characterizing the Participation of Different Demographic Groups in a Korean Nationwide Daily Conversation Corpus

Haewoon Kwak, Jisun An, Kunwoo ParkarXiv 2022 · arXiv

TL;DR

This paper investigates demographic biases in the Daily Conversation Corpus 2020 (DCC2020), a large-scale Korean dataset. The researchers found that participants strongly prefer talking to their own demographic peers, and that social dynamics—particularly age-based honorifics—significantly dampen participation and vocabulary richness when people speak with strangers or different age groups.

What Problem Does This Paper Address?

To build conversational AI that doesn't just "work" but feels natural and fair, we need representative data. However, most dialogue datasets are "black boxes" regarding who is actually speaking. If a model is trained on data where young men mostly talk to other young men, it will likely fail to navigate the complex social nuances of speaking to an elderly woman.

This is especially critical in languages like Korean, which feature a rich, mandatory system of honorifics based on the relative age and social status of the speakers. The authors tackle the "who is missing" problem by analyzing how age and sex influence who people choose to talk to, what they talk about, and how much they actually engage in the conversation.

Methods

The authors analyzed the DCC2020 corpus using several statistical frameworks:

  • Random Rewiring & Z-scores: To detect "partner preference" (homophily), they compared actual conversation pairings against 10,000 "null models" where partners were randomly swapped. A high Z-score indicates a pairing occurs much more often than chance.
  • Chi-square Tests: Used to determine if topic preferences (e.g., sports vs. family) were statistically different across demographic groups.
  • One-sided t-tests: Employed to measure differences in speaking time (participation) and the number of unique tokens (vocabulary richness) across different social settings (strangers vs. non-strangers).

Datasets

The study uses the Daily Conversation Corpus 2020 (DCC2020 v1.0), constructed by the National Institute of Korean Language (NIKL).

  • Scale: 500 hours of transcribed audio, 2,232 conversations, and 2,739 participants.
  • Verification: Unlike many datasets that rely on self-reporting, DCC2020 verified demographics (sex and birth year) via Korean social security numbers, ensuring perfect accuracy.
  • Structure: It includes both "stranger" pairs (matched by NIKL) and "non-stranger" pairs (friends/family who applied together).

Key Insights and Ideas

  • The Homophily Trap: People have a massive bias toward talking to their own kind. As seen in Figure 1, the highest Z-scores (indicating over-representation) are strictly along the diagonal. We prefer our own age and sex, followed by our own age but the opposite sex.
Figure 1: Z-score of non-stranger pairs (label: age+gender).
  • Strangers Silence Women: While men’s speaking time remained relatively stable, females across almost all age groups showed a significant drop in participation when talking to strangers compared to friends or family (see Figure 4).
  • The "Age Gap" Tax on Data: In Korean culture, age is the primary social filter. The study found that people are significantly more interactive and use a more diverse vocabulary when talking to their own age group. When the age gap increases, the conversation often becomes more rigid and less "rich" for the dataset.
  • Topic Bias: Demographics dictate the "what" as much as the "who." For instance, teenage males talk about "dreams," while older groups shift toward "health" and "family." If a dataset lacks certain age groups, it doesn't just lose their voices; it loses entire subject matters.

Key Results

  • Participation: Conversations between non-strangers are significantly more interactive ($p=0.012$) than those between strangers.
  • The 30s Group Case Study: Participants in their 30s showed a dramatic spike in speaking time when talking to peers of the same age and sex, but their engagement dropped when talking to older strangers (see Figure 5/6).
Figure 5: Speaking time of the 30s group according to the listeners (when the listeners are strangers)
  • Vocabulary: Groups like 10s males and females use significantly more unique tokens when talking to their own demographic group. There were no cases where talking to a different demographic group resulted in higher vocabulary diversity.

Limitations and Future Work

The authors acknowledge that these findings are deeply rooted in Korean social structures and the honorific system. While they expect similar trends in other East Asian cultures (like Japan), the results might not translate directly to Western contexts where age-based social hierarchy is less formalized.

Furthermore, this was an observational study. Future work could investigate "interventions"—specific prompts or moderator actions—that might encourage more active and diverse participation even when strangers of different ages are paired together. This would be a vital step toward "de-biasing" the data collection process itself.

Summary updated

Abstract

A conversation corpus is essential to build interactive AI applications. However, the demographic information of the participants in such corpora is largely underexplored mainly due to the lack of individual data in many corpora. In this work, we analyze a Korean nationwide daily conversation corpus constructed by the National Institute of Korean Language (NIKL) to characterize the participation of different demographic (age and sex) groups in the corpus.
BibTeX
@misc{kwak2022who,
  author = {Haewoon Kwak and Jisun An and Kunwoo Park},
  title = {{Who Is Missing? Characterizing the Participation of Different Demographic Groups in a Korean Nationwide Daily Conversation Corpus}},
  year = {2022},
  eprint = {2204.09209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.09209},
}
Figure