Co-creation for Sign Language Processing and Machine Translation

Lisa Lepp, Dimitar Shterionov, Mirella De Sisto, Grzegorz ChrupalaarXiv 2025 · DOI · arXiv

Abstract

Sign Language Machine Translation (SLMT) is currently bottlenecked by a "human factor": the lack of meaningful involvement from the Deaf and Hard of Hearing (HoH) community. This paper analyzes 111 research articles to reveal that the vast majority of SLMT research is hearing-led and treats sign language as mere data. We propose a new "co-creation" framework to bridge this ethical and technical gap, advocating for the active, fair involvement of the Sign Language Community (SLC) in the research process.

Introduction

While text-to-text machine translation has seen massive leaps, Sign Language Machine Translation (SLMT) remains a complex task with limited progress. A fundamental disconnect exists: most research is conducted by hearing scientists who treat sign language as a set of points to be mapped, often ignoring the linguistic nuances and social needs of actual users. This often leads to "intrusive" or useless technology—such as sign language translation gloves—that are frequently rejected by the Deaf community.

Methodology

The study developed a multi-stage approach to evaluate the state of the field:

  1. Relationship Typology: Adapting a framework to define levels of participation from Level -1 (Denigration) to Level 4 (Learning as One).
  2. Meta-Analysis: A review of 111 SLMT articles published recently.
  3. Categorization: Assessing papers based on the depth (power/control), breadth (diversity of participants), and scope (stage of involvement) of community engagement.

The Co-creation Framework

The core of the analysis is a typology that defines the relationship between hearing researchers and the Sign Language Community (SLC). This framework allows for a systematic evaluation of how much power and control the community holds within a research project.

Typology of participation levels between hearing researchers and the Sign Language Community.

Results: The State of SLMT Research

The quantitative findings of the literature review highlight a significant gap in community involvement. Out of 111 reviewed papers:

  • Level -1 (Denigration/Neglect): 83 papers (75%). These projects often used non-representative data or developed tools without community input.
  • Level 0 (Minimal involvement): 13 papers.
  • Level 1 (Learning About): 13 papers.
  • Level 2 (Learning From): 2 papers.
  • Levels 3 & 4 (Co-creation/Partnership): 0 papers.

True co-creation—where the community has a say in ideation and decision-making—is virtually non-existent in current SLMT research. Most papers fall into the lowest levels of engagement, where the community is either ignored or treated solely as a source of data.

Key Insights

From "Data" to "People"

There is an urgent need to move from viewing "language as data" to "language as people." In SLMT, data represents a visual-spatial identity. Using non-representative data, such as hearing people mimicking signs after minimal training, leads to poor technical outcomes and significant ethical concerns.

Research Fatigue

The SLC is relatively small and often suffers from "research fatigue." They are frequently asked to participate in technical projects without seeing benefits or being given a seat at the decision-making table. This extractive relationship must be replaced with one of mutual benefit.

Conclusion and Future Work

To address these issues, the authors derived 10 formal guidelines for co-creation. Future work must address longitudinal dynamics, develop tools for collaboration, and reform funding models that currently hinder early community partnership. Establishing long-term, mutually beneficial relationships is essential for the ethical and technical advancement of SLMT.

Summary updated

BibTeX
@misc{lepp2025co,
  author = {Lisa Lepp and Dimitar Shterionov and Mirella De Sisto and Grzegorz Chrupala},
  title = {{Co-creation for Sign Language Processing and Machine Translation}},
  year = {2025},
  eprint = {2503.01553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.01553},
}
Figure