Self-Sufficient Framework for Continuous Sign Language Recognition

Youngjoon Jang, Youngtaek Oh, Jae-Won Cho, Myungchul Kim, Dong-Jin Kim, In So Kweon, Joon Son Chung

Abstract

Continuous Sign Language Recognition (CSLR) is a challenging computer vision task that requires translating video sequences into glosses—the fundamental units of sign language. Traditional approaches often rely on complex, multi-modal systems involving depth maps, optical flow, or external human keypoint detectors to capture the nuances of manual and non-manual cues. This paper presents a "self-sufficient" framework that achieves state-of-the-art performance using only raw RGB video. By integrating a task-aware convolutional layer (DFConv) for multi-scale feature extraction and a dense pseudo-label refinement (DPLR) strategy to address temporal alignment issues, the proposed model matches or exceeds the performance of more computationally expensive systems.

Introduction

Translating sign language into text involves interpreting a complex interplay of manual cues (hand shapes, positions, and movements) and non-manual cues (facial expressions and mouth movements). In the context of CSLR, models must process these features over time to output a sequence of glosses. However, two primary obstacles have hindered the development of efficient RGB-only models: the multi-cue complexity of the signals and the "spiky" alignment problem inherent in sequence-to-sequence learning.

Most existing CSLR models attempt to solve these issues by incorporating "off-the-shelf" detectors for hands and faces or by utilizing extra sensors. While effective, these methods increase computational overhead and make deployment difficult. This research introduces a framework designed to be entirely self-sufficient, extracting all necessary information directly from RGB frames without external assistance.

Challenges in Continuous Sign Language Recognition

Multi-Cue Complexity

Meaning in sign language is distributed across the body. While hand gestures are primary, facial expressions and head orientations provide critical grammatical and emotional context. Capturing both simultaneously usually requires separate processing streams, which complicates the architecture.

The "Spiky" Alignment Problem

CSLR datasets typically provide sentence-level labels rather than frame-by-frame annotations. To bridge this gap, researchers often use Connectionist Temporal Classification (CTC) loss. While CTC is effective for sequence learning, it tends to produce "spiky" predictions—identifying a sign in only one or two frames and ignoring the rest of the gesture's duration. This sparsity prevents the visual backbone of the model from learning the full temporal extent of a sign.

Proposed Methodology

Divide and Focus Convolution (DFConv)

To address multi-cue complexity without external detectors, the authors introduced Divide and Focus Convolution (DFConv). This module leverages human anatomy as a spatial prior. In a standard sign language video frame, the signer’s face is typically located in the upper half, while the hands move within the lower half. DFConv splits the feature map horizontally: the top section focuses on non-manual cues, while the bottom section is further subdivided to capture the high-velocity movements of the hands. This allows the network to specialize its feature extraction for different body parts within a single, efficient layer.

Dense Pseudo-Label Refinement (DPLR)

To combat the limitations of CTC loss, the framework employs Dense Pseudo-Label Refinement (DPLR). This strategy "densifies" the sparse predictions generated by the model. By comparing initial sparse predictions with ground-truth glosses, the system generates frame-level pseudo-labels that fill in the gaps between the "spikes." This provides a stronger supervision signal, forcing the model to recognize and represent a sign throughout its entire visual duration rather than just at a single peak frame.

Bottleneck-based Temporal Attention (BTA)

The framework also includes a Bottleneck-based Temporal Attention (BTA) module. This component identifies and prioritizes the most salient frames in a sequence, effectively filtering out redundant or blurry frames that do not contribute to the meaning of the sign.

Experimental Evaluation

Datasets

The framework was rigorously tested on two prominent CSLR benchmarks:

  • PHOENIX-2014: A large-scale dataset featuring German weather forecasts.
  • PHOENIX-2014-T: An extension of the original dataset that includes both glosses and German text translations.

Performance and Results

The primary metric for evaluation was the Word Error Rate (WER), where a lower percentage indicates higher accuracy. The proposed framework achieved a 20.8% WER on the PHOENIX-2014 test set. This result is significant as it outperformed all other RGB-only methods and even surpassed several multi-modal systems that utilize pose information or optical flow.

Beyond raw accuracy, the model demonstrated superior robustness. In tests simulating real-world conditions—such as when a signer is scaled or shifted within the frame—the DFConv-based model remained resilient. In contrast, pose-based methods often failed when the signer's hands or face were partially obscured or moved out of the expected detection zone.

Conclusion

The self-sufficient framework demonstrates that high-performance sign language recognition does not require the overhead of external detectors or multi-modal sensors. By utilizing spatial priors through DFConv and addressing the weaknesses of CTC loss with DPLR, the model achieves state-of-the-art results with greater efficiency and robustness than its predecessors.

Future work may focus on adapting the framework for "in-the-wild" data with more diverse framing and exploring the transition from gloss recognition to full natural language translation (Sign Language Translation).

Summary updated

BibTeX
@inproceedings{jang2023self,
  author = {Youngjoon Jang and Youngtaek Oh and Jae-Won Cho and Myungchul Kim and Dong-Jin Kim and In So Kweon and Joon Son Chung},
  title = {{Self-Sufficient Framework for Continuous Sign Language Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095732},
}
Figure