Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation

Andreas Voskou, Konstantinos P. Panousis, Dimitrios Kosmopoulos, Dimitris N. Metaxas, Sotirios Chatzis

Introduction

Sign Language Translation (SLT) presents a unique challenge in the field of computer vision and natural language processing. Unlike spoken language translation, SLT requires the simultaneous modeling of multi-modal features, including hand gestures, facial expressions, and body posture. Historically, high-performing SLT models have relied on "glosses"—intermediate, word-for-word annotations that serve as a bridge between video and text. However, producing these glosses is labor-intensive and requires expert linguistic knowledge.

This paper introduces a novel Stochastic Transformer architecture designed to perform end-to-end translation directly from sign language video to natural language text, bypassing the need for gloss annotations during training. By integrating competitive activation mechanisms and Bayesian weight modeling, the proposed system achieves state-of-the-art performance while significantly improving efficiency.

Methodology: Stochasticity and Competition

The core innovation of this work lies in replacing standard deterministic components with stochastic ones. The architecture deviates from the standard Transformer in three fundamental ways:

1. Local Winner-Takes-All (LWTA) Layers

Instead of the standard ReLU activation, the model employs Local Winner-Takes-All (LWTA) layers. In an LWTA layer, neurons are organized into blocks (e.g., $U=4$). Within each block, neurons compete, and a stochastic sampling process selects a single "winner" to pass its signal forward while others are zeroed out. This mechanism encourages the network to learn sparse, robust representations of the complex visual input.

The Stochastic Transformer architecture and the LWTA mechanism.

2. Variational Bayesian Inference

The model treats connection weights not as fixed values, but as probability distributions. Specifically, weights are modeled as Gaussians: $W \sim N(\mu, \sigma^2)$. During training, the network learns both the mean ($\mu$) and the variance ($\sigma^2$) of each weight. This "doubly stochastic" approach—combining stochastic activations with stochastic weights—serves as a powerful regularizer, preventing overfitting on relatively small sign language datasets.

3. Bayesian Compression

The learned uncertainty ($\sigma^2$) provides a natural pathway for model compression. Weights with high variance are considered "noisy" and can be pruned or represented with lower precision. By applying a Bayesian compression scheme at inference time, the researchers reduced the memory footprint by over 70%, bringing the average weight representation down from 32 bits to approximately 8.8 bits.

Experimental Setup

The model was evaluated on the PHOENIX 2014T benchmark, the gold standard for continuous German Sign Language (DGS) translation. The dataset consists of weather forecast videos featuring nine different signers. Crucially, while the dataset includes glosses, the authors deliberately ignored them during training to demonstrate the effectiveness of their end-to-end approach.

Results and Discussion

The Stochastic Transformer demonstrates that gloss-free translation is not only possible but can outperform traditional methods.

Quantitative Performance

The proposed model achieved a BLEU-4 score of 25.59 on the PHOENIX 2014T test set using an ensemble approach. As shown in the results table, this performance exceeds previous state-of-the-art models, including those that utilized glosses as a training "crutch."

Quantitative comparison of BLEU scores against existing SLT models on PHOENIX 2014T.

Efficiency Gains

Beyond accuracy, the model's efficiency is a standout feature. The Bayesian compression allowed for a massive reduction in memory usage with negligible impact on translation quality. This suggests that such models could be deployed on mobile devices, providing real-time translation tools for the Deaf community.

Qualitative Insights

Ablation studies confirmed that the stochastic nature of the weights is critical; removing the Variational Gaussian weights led to a 1.3-point drop in BLEU-4. While the model is highly accurate, it still occasionally struggles with specific syntactic details, such as precise dates or locations, indicating room for further refinement in temporal modeling.

Conclusion

By moving away from deterministic activations and fixed weights, the Stochastic Transformer offers a more flexible and efficient path for sign language translation. The success of this "gloss-free" approach suggests that the internal dynamics of competitive, stochastic networks are well-suited for the nuances of human sign language, paving the way for more accessible and scalable translation technologies.

Summary updated

Abstract

Automating sign language translation (SLT) is a challenging real world application. Despite its societal importance, though, research progress in the field remains rather poor. Crucially, existing methods that yield viable performance necessitate the availability of laborious to obtain gloss sequence groundtruth. In this paper, we attenuate this need, by introducing an end-to-end SLT model that does not entail explicit use of glosses; the model only needs text groundtruth. This is in stark contrast to existing end-to-end models that use gloss sequence groundtruth, either in the form of a modality that is recognized at an intermediate model stage, or in the form of a parallel output process, jointly trained with the SLT model. Our approach constitutes a Transformer network with a novel type of layers that combines: (i) local winner-takes-all (LWTA) layers with stochastic winner sampling, instead of conventional ReLU layers, (ii) stochastic weights with posterior distributions estimated via variational inference, and (iii) a weight compression technique at inference time that exploits estimated posterior variance to perform massive, almost lossless compression. We demonstrate that our approach can reach the currently best reported BLEU-4 score on the PHOENIX 2014T benchmark, but without making use of glosses for model training, and with a memory footprint reduced by more than 70%.
BibTeX
@inproceedings{voskou2021stochastic,
  author = {Andreas Voskou and Konstantinos P. Panousis and Dimitrios Kosmopoulos and Dimitris N. Metaxas and Sotirios Chatzis},
  title = {{Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=Ou80iD9VxCA},
}
Figure