Who are CUIs Really For? Representation and Accessibility in the Conversational User Interface Literature

William Seymour, Xiao Zhan, Mark Coté, Jose M. Such

TL;DR

This paper investigates the representation and accessibility of participants in Conversational User Interface (CUI) research, finding a significant overrepresentation of participants from North America and Europe. It argues that this lack of diversity limits the inclusivity of CUI design and calls for a broader community effort to address these systemic issues.

What Problem Does This Paper Address?

Conversational User Interfaces (CUIs), like voice assistants and chatbots, are becoming increasingly prevalent. However, the field of computer science, and technology more broadly, has a well-documented diversity problem. This paper asks: who are CUIs really designed for? It highlights that if research participants are predominantly from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies, the resulting technologies may not be designed for the needs and contexts of the global majority. This lack of diverse representation can perpetuate existing societal biases and lead to technologies that are not inclusive or accessible to everyone. The paper aims to understand how this issue manifests in the CUI research community specifically.

Methods

The authors conducted two main analyses:

  1. Systematic Literature Review of Voice Assistants: They performed a broad review of literature on ethical concerns with voice assistants, analyzing 117 papers published since 2012. This review focused on themes like privacy, social interaction, accessibility, performance of gender, and social order. They also coded papers for participant demographics and country of residence.
  2. Sift of CUI Conference Papers: To specifically address the CUI community, they conducted a less rigorous sift of the 46 full-length research papers published at the CUI conference from its inception in 2019 to 2022. They analyzed abstracts, introductions, and methodologies to identify papers engaging with themes of accessibility, social interaction, and the performance of gender. They also examined participant demographics and geographic locations.

Datasets

Not discussed in the paper. The paper focuses on analyzing existing published research papers rather than training or evaluating models on specific datasets.

Key Insights and Ideas

  • WEIRD Dominance: The most striking finding is the overwhelming representation of participants from North America and Europe in both voice assistant literature (94%) and CUI research (90% of CUI papers with user studies). This suggests a significant gap between the global user base and the populations studied in research.
  • CUI's Ethical Awareness, Limited Scope: While the CUI community shows willingness to engage with ethical aspects of CUIs, this engagement often remains within a narrow geographic and cultural context.
  • Accessibility is a Double-Edged Sword: CUIs can be accessible to visually impaired users due to their text/language-based nature. However, they can exclude individuals with speech, language, or hearing difficulties, or those who struggle with the required speed and formulation of requests.
  • Gender Performance Perpetuates Bias: The default persona of many CUIs is often a white, middle-class American woman, justified by user acceptability studies. This can reinforce gender stereotypes, and attempts to create "genderless" interfaces can sometimes erase non-binary individuals.
  • Author Location Influences Participant Location: The research suggests a strong correlation between the geographic location of paper authors and the location of their study participants, indicating that researchers tend to recruit from their immediate surroundings.
  • Systemic Issues Beyond Individual Control: The paper acknowledges that addressing diversity is a complex, systemic issue influenced by profit motives, data protection regulations, design team homogeneity, and the academic peer-review process, which can favor WEIRD methodologies.

Key Results

  • Voice Assistant Literature: 94% of participants in papers investigating ethics were from Europe or North America.
  • CUI Conference Papers (2019-2022):
    • Out of 46 full-length papers, 8 papers (17%) explored topics related to accessibility, social interaction, and performance of gender.
  • 90% of CUI papers with user studies recruited participants solely from Europe and North America, or did not specify the location.
  • The two papers that studied participants from South America shared common authors, representing the only studies focusing on locations outside of Europe and North America.
  • Global voice assistants exist in diverse languages and regions, yet remain "dramatically unstudied by CUI papers."

Limitations and Future Work

Limitations:

  • "Less Rigorous" CUI Sift: The analysis of CUI papers was described as less rigorous than the systematic review of voice assistant literature.
  • Data Limitations: The paper acknowledges that information on author affiliations and participant demographics in research papers is often limited and may not fully capture the nuances of diversity (e.g., culture, background, ethnicity). Using country of residence as a proxy for diversity is a simplification.
  • Systemic Complexity: The authors admit that their suggestions for improvement in Section 4 are not perfect answers and that the issue is deeply complex, involving political and bureaucratic barriers.
  • Peer Review Bias: The peer-review process itself can inherently make it harder to publish work that deviates from WEIRD agendas and methodologies.

Future Work:

  • Broader Community Discussion: The paper explicitly aims to "form the beginning of a wider discussion at the conference" and encourages suggestions on how to work against the current status quo.
  • Increased Financial Support: Suggests increased professional registration fees to support additional financial aid for registration and travel for attendees from outside WEIRD countries.
  • Parallel Submission Tracks: Proposes parallel submission tracks for diverse/underrepresented research.
  • Encourage Replication: Advocates for encouraging the replication of previously published studies across different communities and cultures.
  • Address Incentives: Aims to better align researcher incentives (publish or perish) with the conference's and community's value of diverse research.
  • Further Research on Non-WEIRD CUIs: The paper highlights the need to study the vast array of non-WEIRD CUIs that are currently underrepresented in research.

Summary updated

Abstract

The theme for CUI 2023 is 'designing for inclusive conversation', but who are CUIs really designed for? The field has its roots in computer science, which has a long acknowledged diversity problem. Inspired by studies mapping out the diversity of the CHI and voice assistant literature, we set out to investigate how these issues have (or have not) shaped the CUI literature. To do this we reviewed the 46 full-length research papers that have been published at CUI since its inception in 2019. After detailing the eight papers that engage with accessibility, social interaction, and performance of gender, we show that 90% of papers published at CUI with user studies recruit participants from Europe and North America (or do not specify). To complement existing work in the community towards diversity we discuss the factors that have contributed to the current status quo, and offer some initial suggestions as to how we as a CUI community can continue to improve. We hope that this will form the beginning of a wider discussion at the conference.
BibTeX
@inproceedings{seymour2023who,
  author = {William Seymour and Xiao Zhan and Mark Coté and Jose M. Such},
  title = {{Who are CUIs Really For? Representation and Accessibility in the Conversational User Interface Literature}},
  booktitle = {CUI},
  year = {2023},
  url = {https://openreview.net/forum?id=xyVJ9iFTxi},
}
Figure