Visual Alignment Constraint for Continuous Sign Language Recognition

Yuecong Min, Aiming Hao, Xiujuan Chai, Xilin Chen

Abstract

Vision-based Continuous Sign Language Recognition (CSLR) aims to recognize unsegmented signs from image streams. Overfitting is one of the most critical problems in CSLR training, and previous works show that the iterative training scheme can partially solve this problem while also costing more training time. In this study, we revisit the iterative training scheme in recent CSLR works and realize that sufficient training of the feature extractor is critical to solving the overfitting problem. Therefore, we propose a Visual Alignment Constraint (VAC) to enhance the feature extractor with alignment supervision. Specifically, the proposed VAC comprises two auxiliary losses: one focuses on visual features only, and the other enforces prediction alignment between the feature extractor and the alignment module. Moreover, we propose two metrics to reflect overfitting by measuring the prediction inconsistency between the feature extractor and the alignment module. Experimental results on two challenging CSLR datasets show that the proposed VAC makes CSLR networks end-to-end trainable and achieves competitive performance.

Introduction

Continuous Sign Language Recognition (CSLR) involves the complex task of translating video streams into sequences of glosses. A standard architecture for this task consists of a Feature Extractor (typically a 2D/1D-CNN) and an Alignment Module (often a BiLSTM with Connectionist Temporal Classification, or CTC). A significant challenge in CSLR is the tendency of the Alignment Module to overfit, focusing on a sparse set of frames and relying heavily on language context. This "lazy" behavior prevents the Feature Extractor from learning robust visual representations. While previous approaches utilized multi-stage iterative training to mitigate this, such methods are computationally expensive and difficult to manage.

Methodology

We introduce the Visual Alignment Constraint (VAC), a plug-and-play supervision framework that enables effective end-to-end training. As illustrated in Figure 2, VAC incorporates two auxiliary losses to guide the Feature Extractor directly.

Detailed architecture of the CSLR framework with the proposed Visual Alignment Constraint (VAC), showing the integration of Visual Enhancement ($L_{VE}$) and Visual Alignment ($L_{VA}$) losses.
  1. Visual Enhancement (VE) Loss ($L_{VE}$): By attaching an auxiliary classifier to the Feature Extractor, we force the CNN to predict glosses based solely on local visual information. This prevents the model from relying exclusively on the BiLSTM's temporal context.
  2. Visual Alignment (VA) Loss ($L_{VA}$): This loss employs knowledge distillation to align the probability distributions of the Feature Extractor and the full Alignment Module. This ensures that visual features are both accurate and compatible with the learned temporal logic.

Analysis of Training Dynamics

Our investigation into the "Magnitude Hypothesis" reveals that the $l_2$ norm of feature vectors serves as an indicator of frame importance. In overfitted models, the magnitude of non-key frames is suppressed, leading to the "spike phenomenon" in CTC predictions. Figure 4 demonstrates these dynamics across training iterations, highlighting how the model's focus narrows over time.

Visualization of training dynamics across iterations, showing the evolution of gate values, feature norms, and prediction spikes.

To further diagnose model health, we propose two metrics:

  • Word Deterioration Rate (WDR): The frequency at which the Alignment Module incorrectly overrides a correct visual prediction.
  • Word Amelioration Rate (WAR): The frequency at which the Alignment Module successfully corrects a visual error using context.

Experimental Results

Evaluations on the RWTH-PHOENIX-Weather-2014 dataset show that VAC achieves a Word Error Rate (WER) of 21.2% on the development set and 22.3% on the test set, significantly outperforming the baseline. On the CSL (Chinese Sign Language) dataset, VAC sets a new state-of-the-art with a WER of 1.6%. Notably, VAC provides these improvements in a single training pass, offering a substantial speedup over iterative methods.

Conclusion

The Visual Alignment Constraint (VAC) provides a robust solution to the overfitting problem in CSLR. By enforcing direct visual supervision and alignment, it ensures that the Feature Extractor learns high-quality representations. This approach not only improves recognition accuracy but also simplifies the training pipeline by enabling efficient end-to-end optimization.

Summary updated

Abstract

Vision-based Continuous Sign Language Recognition (CSLR) aims to recognize unsegmented signs from image streams. Overfitting is one of the most critical problems in CSLR training, and previous works show that the iterative training scheme can partially solve this problem while also costing more training time. In this study, we revisit the iterative training scheme in recent CSLR works and realize that sufficient training of the feature extractor is critical to solving the overfitting problem. Therefore, we propose a Visual Alignment Constraint (VAC) to enhance the feature extractor with alignment supervision. Specifically, the proposed VAC comprises two auxiliary losses: one focuses on visual features only, and the other enforces prediction alignment between the feature extractor and the alignment module. Moreover, we propose two metrics to reflect overfitting by measuring the prediction inconsistency between the feature extractor and the alignment module. Experimental results on two challenging CSLR datasets show that the proposed VAC makes CSLR networks end-to-end trainable and achieves competitive performance.
BibTeX
@inproceedings{min2021visual,
  author = {Yuecong Min and Aiming Hao and Xiujuan Chai and Xilin Chen},
  title = {{Visual Alignment Constraint for Continuous Sign Language Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=joSxrlLUeg},
}
Figure