Introduction
Sign Language Translation (SLT) presents a unique challenge in the field of computer vision and natural language processing. Unlike spoken language translation, SLT requires the simultaneous modeling of multi-modal features, including hand gestures, facial expressions, and body posture. Historically, high-performing SLT models have relied on "glosses"—intermediate, word-for-word annotations that serve as a bridge between video and text. However, producing these glosses is labor-intensive and requires expert linguistic knowledge.
This paper introduces a novel Stochastic Transformer architecture designed to perform end-to-end translation directly from sign language video to natural language text, bypassing the need for gloss annotations during training. By integrating competitive activation mechanisms and Bayesian weight modeling, the proposed system achieves state-of-the-art performance while significantly improving efficiency.
Methodology: Stochasticity and Competition
The core innovation of this work lies in replacing standard deterministic components with stochastic ones. The architecture deviates from the standard Transformer in three fundamental ways:
1. Local Winner-Takes-All (LWTA) Layers
Instead of the standard ReLU activation, the model employs Local Winner-Takes-All (LWTA) layers. In an LWTA layer, neurons are organized into blocks (e.g., $U=4$). Within each block, neurons compete, and a stochastic sampling process selects a single "winner" to pass its signal forward while others are zeroed out. This mechanism encourages the network to learn sparse, robust representations of the complex visual input.
2. Variational Bayesian Inference
The model treats connection weights not as fixed values, but as probability distributions. Specifically, weights are modeled as Gaussians: $W \sim N(\mu, \sigma^2)$. During training, the network learns both the mean ($\mu$) and the variance ($\sigma^2$) of each weight. This "doubly stochastic" approach—combining stochastic activations with stochastic weights—serves as a powerful regularizer, preventing overfitting on relatively small sign language datasets.
3. Bayesian Compression
The learned uncertainty ($\sigma^2$) provides a natural pathway for model compression. Weights with high variance are considered "noisy" and can be pruned or represented with lower precision. By applying a Bayesian compression scheme at inference time, the researchers reduced the memory footprint by over 70%, bringing the average weight representation down from 32 bits to approximately 8.8 bits.
Experimental Setup
The model was evaluated on the PHOENIX 2014T benchmark, the gold standard for continuous German Sign Language (DGS) translation. The dataset consists of weather forecast videos featuring nine different signers. Crucially, while the dataset includes glosses, the authors deliberately ignored them during training to demonstrate the effectiveness of their end-to-end approach.
Results and Discussion
The Stochastic Transformer demonstrates that gloss-free translation is not only possible but can outperform traditional methods.
Quantitative Performance
The proposed model achieved a BLEU-4 score of 25.59 on the PHOENIX 2014T test set using an ensemble approach. As shown in the results table, this performance exceeds previous state-of-the-art models, including those that utilized glosses as a training "crutch."
Efficiency Gains
Beyond accuracy, the model's efficiency is a standout feature. The Bayesian compression allowed for a massive reduction in memory usage with negligible impact on translation quality. This suggests that such models could be deployed on mobile devices, providing real-time translation tools for the Deaf community.
Qualitative Insights
Ablation studies confirmed that the stochastic nature of the weights is critical; removing the Variational Gaussian weights led to a 1.3-point drop in BLEU-4. While the model is highly accurate, it still occasionally struggles with specific syntactic details, such as precise dates or locations, indicating room for further refinement in temporal modeling.
Conclusion
By moving away from deterministic activations and fixed weights, the Stochastic Transformer offers a more flexible and efficient path for sign language translation. The success of this "gloss-free" approach suggests that the internal dynamics of competitive, stochastic networks are well-suited for the nuances of human sign language, paving the way for more accessible and scalable translation technologies.
Summary updated