sign.mt: Real-Time Multilingual Sign Language Translation Application

Amit MoryossefEMNLP 2024

TL;DR

This paper introduces sign.mt, an open-source, real-time, multilingual application for translating between spoken and signed languages. It aims to break down communication barriers for the deaf and hard-of-hearing community by offering both spoken-to-signed and signed-to-spoken translation, with a focus on accessibility, offline functionality, and community-driven development.

What Problem Does This Paper Address?

Communication between individuals who are deaf or hard of hearing and those who use spoken language remains a significant challenge. While sign language translation research exists, it's often fragmented, with different groups focusing on specific aspects or languages. Developing and deploying these systems is complex, requiring substantial computational resources and specialized interfaces, which can hinder the creation of accessible, interactive demonstrations. Furthermore, the high cost of server-side deployment and the complexity of client-side implementations often limit the reach and usability of these technologies. sign.mt aims to address this by providing a unified, open-source platform that integrates existing research and encourages further development.

Methods

sign.mt is designed as a modular and interchangeable pipeline, allowing for the seamless integration of various state-of-the-art models. The application supports two primary translation directions:

  • Spoken-to-Signed Translation: This pipeline takes spoken language audio or text as input. It includes optional steps for language identification, text normalization (e.g., correcting grammar and punctuation), and sentence splitting. The core translation involves converting spoken language text into SignWriting, a written form of sign language. This SignWriting is then translated into a pose sequence, which can be rendered in three ways: a minimalistic skeleton viewer, a realistic Human GAN avatar, or a 3D avatar.
  • Signed-to-Spoken Translation: This pipeline takes a sign language video as input. It uses pose estimation to extract body keypoints from each frame, followed by segmentation to identify individual signs and phrase boundaries. These segmented signs are then transcribed into SignWriting, which serves as the textual input for a translation model that converts it into spoken language text. This text can then be optionally converted into spoken audio using Text-to-Speech (TTS).

A key aspect of sign.mt is its modularity, allowing researchers to easily integrate their models into the application's infrastructure, test them in a practical environment, and contribute them back to the main codebase if successful.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation. However, it mentions the use of existing models and research findings, implying reliance on datasets commonly used in sign language processing research. The project itself aims to foster the creation and sharing of models, which would naturally involve various datasets.

Key Insights and Ideas

  • Open-Source and Community-Driven: The project's commitment to being open-source (CC BY-NC-SA 4.0) and freely available on GitHub is a significant differentiator, aiming to accelerate research and development through community contributions.
  • Modular Pipeline Design: The architecture's emphasis on modularity allows for easy swapping and integration of different models, making it a dynamic platform for experimentation and showcasing new research.
  • Offline Functionality: Providing offline translation capabilities is crucial for accessibility, especially in areas with limited internet connectivity.
  • Multimodal Translation: The application tackles the complex cross-modal transformation required for both spoken-to-signed and signed-to-spoken translation, a significant challenge in NLP.
  • User-Centric Rendering Options: Offering multiple rendering options (skeleton, Human GAN, 3D avatar) caters to different user needs and computational capabilities, enhancing usability.

Key Results

The paper highlights the growing interest and adoption of sign.mt:

  • Global Adoption: Users are distributed across multiple countries, with significant adoption in the United States, India, and the United Kingdom.
  • Consistent Growth: Google Analytics shows slow but consistent user growth, indicating sustained relevance and reliability.
  • Repository Popularity: The GitHub repository has garnered 470 stars as of October 6th, 2024, demonstrating community interest.
  • Search Console Metrics: Google Search Console metrics show a significant increase in impressions (106K) and clicks (3.75K) over the past six months, further validating public interest.

Limitations and Future Work

The authors acknowledge several limitations:

  • SignWriting-to-Pose Model: A fully functional SignWriting-to-pose animation model is currently lacking. The system uses a simpler dictionary-based approach, which is insufficient for handling the nuances of signed languages.
  • Model Fragmentation: While aiming for a multilingual platform, the availability of models for different languages is fragmented, leading to varying degrees of effectiveness. The project relies on the research community to develop and contribute models for more languages.
  • Client-Side Model Size Constraints: The client-side deployment restricts the use of relatively smaller models, leading to trade-offs in translation accuracy and quality.
  • Computational Intensity: Video processing components are computationally intensive, limiting performance on devices with lower computing power. Optimizing these components for smoother user experience across a wider range of devices is an ongoing challenge.

Future work will focus on addressing these limitations, encouraging community contributions for more advanced models, expanding language support, and improving the performance and accessibility of the application. The development of native iOS and Android builds is also underway, aiming to extend the app's reach.

Summary updated

BibTeX
@inproceedings{moryossef2024sign,
  author = {Amit Moryossef},
  title = {{sign.mt: Real-Time Multilingual Sign Language Translation Application}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}

Loading related content...

Figure