Efficient Sign Language Translation with a Curriculum-based Non-autoregressive Decoder

Pei Yu, Liang Zhang, Biao Fu, Yidong ChenIJCAI 2023 · DOI

Introduction

Sign Language Translation (SLT) is the task of converting continuous sign language videos into spoken language text. Traditionally, this has been approached using AutoRegressive (AR) decoders, which generate text one token at a time. While AR models achieve high accuracy, they suffer from significant latency because each word's generation depends on the previous one, preventing parallelization. Furthermore, AR models are inherently unidirectional, often failing to capture the full bidirectional context of a sentence.

To address these issues, Non-AutoRegressive (NAR) models have been proposed. NAR models generate all tokens in a single pass, offering massive speedups. However, they typically struggle with translation quality because they cannot easily model the complex dependencies between words. This paper introduces the Curriculum-based Non-autoregressive Decoder (CND) and a Mutual Learning Framework to bridge this performance gap.

The Curriculum-based Non-autoregressive Decoder (CND)

The CND addresses the difficulty of NAR generation by adopting a "simple-to-complex" curriculum learning strategy across its $N$ Transformer layers. Instead of predicting the entire sentence at once in the final layer, every layer in the CND makes a prediction.

The core idea is that some words in a sign language sentence are easier to translate than others. Specifically, "visually grounded" tokens—those directly corresponding to physical signs—are easier to identify than "linguistic" tokens, which serve grammatical or functional roles.

Figure 1: Illustration of visual vs. linguistic tokens in sign language translation. Visually grounded words like "watch" and "you" correspond directly to signs, while grammatical words like "does" and "belong to" require linguistic context.

The CND curriculum works as follows:

  1. Layer-wise Prediction: Each decoder layer produces a translation.
  2. Linear Scaling: At lower layers, the model is only required to predict a small number of "easy" (visually grounded) tokens. As the data passes through higher layers, the model is tasked with predicting an increasing number of "complex" tokens that require more linguistic context.
  3. Masking Strategy: Tokens that the model is not yet "confident" in at a given layer are replaced with a [mask] token for the subsequent layer to resolve.

Mutual Learning Framework

To further enhance the CND, the authors propose a mutual learning framework where an AR decoder and the CND are trained simultaneously. Unlike traditional knowledge distillation, where a "student" learns from a static "teacher," this is a peer-learning setup.

The training objective incorporates Kullback–Leibler (KL) divergence to minimize the difference between the outputs of both decoders. This bidirectional knowledge transfer allows the CND to learn sequential dependencies from the AR model, while the AR model benefits from the global, bidirectional context captured by the NAR model.

Experimental Results

The proposed model was evaluated on two major benchmarks: PHOENIX 2014T (German weather forecasts) and CSL-Daily (Chinese daily life topics).

  • Efficiency: The CND achieved a 7.92x speedup on PHOENIX 2014T and an 8.02x speedup on CSL-Daily compared to standard AR models. Inference latency was reduced to approximately 24ms per sentence.
  • Accuracy: On PHOENIX 2014T, the CND with Mutual Learning achieved a 24.71 BLEU4 score, outperforming the baseline AR model (23.24). This demonstrates that the NAR model, when properly trained, can exceed the quality of traditional sequential models.
  • Ablation Studies: The researchers found that a simple linear increase in the number of predicted tokens per layer outperformed logarithmic or exponential curriculum functions.

Conclusion and Future Work

The Curriculum-based Non-autoregressive Decoder represents a significant step forward in efficient sign language translation. By mimicking the human learning process—focusing first on visual cues and then on linguistic structure—the model achieves state-of-the-art accuracy with near real-time inference speeds.

While the model currently relies on an external prediction of target sentence length, future work will focus on making length prediction more robust and further optimizing the mutual learning framework to reduce training overhead.

Summary updated

BibTeX
@inproceedings{yu2023efficient,
  author = {Pei Yu and Liang Zhang and Biao Fu and Yidong Chen},
  title = {{Efficient Sign Language Translation with a Curriculum-based Non-autoregressive Decoder}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/584},
}
Figure