Introduction
Sign Language Translation (SLT) is the task of converting continuous sign language videos into spoken language text. Traditionally, this has been approached using AutoRegressive (AR) decoders, which generate text one token at a time. While AR models achieve high accuracy, they suffer from significant latency because each word's generation depends on the previous one, preventing parallelization. Furthermore, AR models are inherently unidirectional, often failing to capture the full bidirectional context of a sentence.
To address these issues, Non-AutoRegressive (NAR) models have been proposed. NAR models generate all tokens in a single pass, offering massive speedups. However, they typically struggle with translation quality because they cannot easily model the complex dependencies between words. This paper introduces the Curriculum-based Non-autoregressive Decoder (CND) and a Mutual Learning Framework to bridge this performance gap.
The Curriculum-based Non-autoregressive Decoder (CND)
The CND addresses the difficulty of NAR generation by adopting a "simple-to-complex" curriculum learning strategy across its $N$ Transformer layers. Instead of predicting the entire sentence at once in the final layer, every layer in the CND makes a prediction.
The core idea is that some words in a sign language sentence are easier to translate than others. Specifically, "visually grounded" tokens—those directly corresponding to physical signs—are easier to identify than "linguistic" tokens, which serve grammatical or functional roles.
The CND curriculum works as follows:
- Layer-wise Prediction: Each decoder layer produces a translation.
- Linear Scaling: At lower layers, the model is only required to predict a small number of "easy" (visually grounded) tokens. As the data passes through higher layers, the model is tasked with predicting an increasing number of "complex" tokens that require more linguistic context.
- Masking Strategy: Tokens that the model is not yet "confident" in at a given layer are replaced with a
[mask]token for the subsequent layer to resolve.
Mutual Learning Framework
To further enhance the CND, the authors propose a mutual learning framework where an AR decoder and the CND are trained simultaneously. Unlike traditional knowledge distillation, where a "student" learns from a static "teacher," this is a peer-learning setup.
The training objective incorporates Kullback–Leibler (KL) divergence to minimize the difference between the outputs of both decoders. This bidirectional knowledge transfer allows the CND to learn sequential dependencies from the AR model, while the AR model benefits from the global, bidirectional context captured by the NAR model.
Experimental Results
The proposed model was evaluated on two major benchmarks: PHOENIX 2014T (German weather forecasts) and CSL-Daily (Chinese daily life topics).
- Efficiency: The CND achieved a 7.92x speedup on PHOENIX 2014T and an 8.02x speedup on CSL-Daily compared to standard AR models. Inference latency was reduced to approximately 24ms per sentence.
- Accuracy: On PHOENIX 2014T, the CND with Mutual Learning achieved a 24.71 BLEU4 score, outperforming the baseline AR model (23.24). This demonstrates that the NAR model, when properly trained, can exceed the quality of traditional sequential models.
- Ablation Studies: The researchers found that a simple linear increase in the number of predicted tokens per layer outperformed logarithmic or exponential curriculum functions.
Conclusion and Future Work
The Curriculum-based Non-autoregressive Decoder represents a significant step forward in efficient sign language translation. By mimicking the human learning process—focusing first on visual cues and then on linguistic structure—the model achieves state-of-the-art accuracy with near real-time inference speeds.
While the model currently relies on an external prediction of target sentence length, future work will focus on making length prediction more robust and further optimizing the mutual learning framework to reduce training overhead.
Summary updated