Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas

Aashaka Desai, Maartje De Meulder, Julie A. Hochgesang, Annemarie Kocab, Alex X. Lu

TL;DR

This paper argues that sign language AI research is dominated by hearing researchers, leading to systemic biases in research agendas, datasets, and methodologies. A systematic review of 101 papers reveals an overemphasis on communication barriers, underrepresentation of Deaf stakeholders, and reliance on flawed linguistic assumptions, calling for Deaf researchers to lead the field.

What Problem Does This Paper Address?

The rapid growth of AI for sign language (recognition, generation, translation) has been accompanied by calls for ethical development. However, this paper highlights a critical gap: the lack of discussion around systemic biases and the rhetoric that shapes research questions and methods, particularly given the field's historical dominance by hearing, non-signing researchers. The authors argue that even if individual projects follow best practices, the collective direction of the field might be misaligned with the interests and perspectives of Deaf communities, the primary stakeholders. They aim to critically interrogate the assumptions and rhetoric that steer sign language AI research and advocate for a more Deaf-led approach.

Methods

The authors conducted a hybrid literature review and position paper, systematically analyzing 101 papers in sign language AI published between January 2021 and November 2023. Their corpus was initially gathered from arXiv, focusing on "receptive" sign language models (those that process sign language input) and excluding works that solely study human factors or use sign language merely as a demonstration for other tasks. They excluded reviews, theses, and non-English works. Three authors reviewed paper abstracts against inclusion criteria, with a fourth author resolving disagreements. A codebook was iteratively developed to track datasets, model inputs/outputs, and prior models used. Two annotators coded each paper, with a third resolving discrepancies.

Datasets

The review identified 43 publicly available sign language datasets used across the analyzed papers. These datasets vary in their focus, including:

  • Continuous Sign Language Datasets: 60 papers used these.
  • Isolated Sign Language Datasets: 26 papers used these.
  • Fingerspelling Data: 11 papers used this.

Seven works collected their own private datasets. The paper notes that while many datasets are publicly available, there are concerns about their representativeness and the potential for bias. Notably, 16 datasets use solely DHH contributors, 3 use solely interpreters, 11 include a mix, and 12 do not specify contributor qualifications. The paper highlights that many works rely on interpreter-only datasets (like Albanie et al., 2021; Forster et al., 2014), which are often derived from media broadcasts and may not reflect natural signing. More recent datasets like ASL Citizen (Desai et al., 2024) and Sem-Lex (Kezar et al., 2023) aim for more representative data from "fluent" DHH signers, but the definition of "fluency" is critiqued as potentially subjective and exclusionary.

Key Insights and Ideas

  • Communication Barriers as the Primary Motivation: A significant majority of papers (64 out of 101) are motivated by addressing perceived communication barriers between Deaf and hearing individuals. The authors argue this framing can oversimplify the role of sign languages, portraying them as mere "communication tools" rather than full languages with rich histories and cultural significance.
  • "Translationese" and Flawed Datasets: Many datasets are derived from interpreted sign language or media broadcasts, leading to "translationese" – a form of signing that deviates from natural signing within Deaf communities. This can result in models that do not accurately represent how Deaf people actually sign.
  • The "Tyranny of Glossing": The widespread use of glosses (written representations of signs) as the primary annotation scheme is critiqued. Glosses are often used without linguistic grounding or understanding of their theoretical underpinnings, leading to a loss of crucial linguistic information and a flattening of sign language complexity.
  • Inherited Biases in Modeling: The reliance on pre-trained models, particularly pose estimators and vision models trained on non-sign language data (like ImageNet or Kinetics), introduces and perpetuates biases. These models may not capture the fine-grained nuances of sign language, such as facial expressions, which are critical for grammar.
  • The Need for Deaf Leadership: The paper strongly advocates for Deaf researchers to lead the field. They argue that current research agendas are often driven by the convenience of hearing researchers rather than the actual needs and interests of Deaf communities.

Key Results

  • Paper Distribution: Out of 101 papers reviewed, most focused on sign language recognition (60 papers) or translation (26 papers).
  • Dataset Types: 60 papers used continuous sign language datasets, 26 used isolated sign language datasets, and 11 used fingerspelling data.
  • Motivation: 64 papers were primarily motivated by addressing perceived communication barriers.
  • Dataset Contributors: Only 16 datasets used solely DHH contributors, while 3 used solely interpreters, and 12 did not specify.
  • Annotation Schemes: Half of the papers (51) relied on glosses as their main output or intermediary representation.

Limitations and Future Work

The authors acknowledge that their review has limitations, including the reliance on arXiv for paper retrieval, which may not capture all published work. They also note that while they advocate for Deaf leadership, this is not a panacea and requires careful consideration to avoid tokenism.

Future work suggested by the paper includes:

  • Fostering Deaf Leadership: Creating space for Deaf researchers to lead research agendas and decision-making.
  • Transparency in Positionality: Encouraging all researchers to be transparent about their backgrounds, experiences, and potential biases.
  • Reassessing Research Agendas: Moving beyond a singular focus on communication barriers to explore sign languages as languages in their own right.
  • Developing Linguistically Grounded Methods: Incorporating deeper linguistic understanding into dataset creation, annotation, and model design.
  • Mitigating Pre-training Biases: Investigating and addressing the biases introduced by pre-trained models.
  • Broader Representation: Ensuring datasets and models are representative of the diverse spectrum of Deaf signers, including those with additional disabilities or different acquisition pathways.

Summary updated

BibTeX
@misc{desai2024systemic,
  author = {Aashaka Desai and Maartje De Meulder and Julie A. Hochgesang and Annemarie Kocab and Alex X. Lu},
  title = {{Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas}},
  year = {2024},
  eprint = {2403.02563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.02563},
}
Figure