Linguistically Informed Transformers for Text to American Sign Language Translation

Abhishek Bharadwaj Varanasi, Manjira Sinha, Tirthankar Dasgupta, Charudatta JadhavACL 2024

TL;DR

This paper introduces a novel transformer-based architecture that incorporates linguistic information to improve the translation of English text into American Sign Language (ASL) glosses. By leveraging syntactic dependency graphs, their model achieves state-of-the-art performance on existing benchmarks and demonstrates the importance of linguistic features for more accurate ASL translation.

What Problem Does This Paper Address?

Sign Languages (SLs) are complex, non-verbal languages with their own grammatical structures, making direct translation from spoken languages challenging. Unlike spoken languages, SLs lack a standardized written form, necessitating intermediate representations like "glosses" – spoken language labels for sign components. While advancements in machine translation (MT) have been significant, applying them directly to SLs is difficult due to their unique linguistic properties. Existing approaches often overlook crucial latent linguistic traits, leading to suboptimal translation quality. This paper aims to bridge this gap by developing a linguistically informed MT system that better captures the nuances of ASL.

Methods

The core of this work is a linguistically informed transformer architecture. The authors adapt the standard transformer model by integrating syntactic dependency graph information. Here's how it works:

  • Syntactic Dependency Graph: Each input English sentence is parsed to create a syntactic dependency graph. This graph captures the grammatical relationships between words in the sentence.
  • Graph Convolutional Network (GCN): A GCN is used to process this dependency graph, encoding the structural information into embeddings. Pre-trained GloVe embeddings are used as initial states for the GCN.
  • Integration with Transformer: The output hidden states from the GCN are combined with the contextual embeddings generated by the transformer's encoder. This combined representation is then fed into the transformer's decoder for the text-to-ASL gloss translation.

This approach aims to enhance the transformer's ability to understand and translate linguistic nuances that might be missed by standard models.

Datasets

The researchers utilized two datasets:

  • ASLG-PC12 Dataset: This is a publicly available, large-scale text-to-ASL gloss dataset containing 87,710 bilingual sentences. Its primary focus is on news and politics, which the authors note limits its applicability across diverse domains.
  • ASL-Bank Dataset: To address the domain specificity issue, the authors manually created a new dataset focused on banking-related terminology. This dataset consists of 3,597 text-ASL gloss pairs, curated by domain experts.

Key Insights and Ideas

  • Linguistic Information is Crucial: The paper strongly suggests that simply applying standard MT techniques to ASL translation is insufficient. Incorporating linguistic structures, like syntactic dependencies, significantly improves translation accuracy.
  • Domain-Specific Datasets Matter: While ASLG-PC12 is a valuable resource, its limited domain coverage necessitates the creation of specialized datasets (like ASL-Bank) for effective domain-specific translation.
  • Transformer Architecture is Adaptable: The transformer, a powerful architecture for sequence-to-sequence tasks, can be effectively augmented with external linguistic information through methods like GCN integration.

Key Results

The paper reports impressive results:

  • ASLG-PC12 Performance: The linguistically informed transformer model achieved a 97.83% ROUGE-L score on the ASLG-PC12 dataset, outperforming a baseline GRU model.
  • ASL-Bank Performance: When fine-tuned on the ASL-Bank dataset, the model achieved an 89.47% ROUGE-L score (when fine-tuned on ASLG-PC12 + ASL-Bank).
  • Comparison to Baselines: The transformer models (T5-small, T5-base, BART-base) consistently outperformed a GRU-based model on the ASLG-PC12 dataset.
  • Challenges Identified: The authors note that challenges remain, including variations in word ordering for topic-comment structures, handling of wh-questions, and occasional replacement of words with synonyms, which can reduce metric scores despite being acceptable in signing.

Limitations and Future Work

The authors acknowledge several limitations:

  • Text to ASL Gloss Focus: The current work specifically targets text-to-ASL gloss translation. The models are not directly applicable to other sign languages due to structural differences.
  • Video Generation: While generated glosses can be mapped to videos, the output exhibits inconsistencies in resolution and signing style. Improving video generation quality is a separate, complex task.
  • Leveraging LLMs for Other SLs: The authors suggest that open-source Large Language Models (LLMs) could be leveraged, potentially combined with external sign language rules, to tackle the limitation of using a single model for different sign languages. This could involve techniques like Retrieval Augmented Generation (RAG).

Future work could explore these avenues, including extending the approach to other sign languages and integrating more sophisticated video generation techniques.

Summary updated

BibTeX
@inproceedings{varanasi2024linguistically,
  author = {Abhishek Bharadwaj Varanasi and Manjira Sinha and Tirthankar Dasgupta and Charudatta Jadhav},
  title = {{Linguistically Informed Transformers for Text to American Sign Language Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure