OLMD: Orientation-aware Long-term Motion Decoupling for Continuous Sign Language Recognition

Yiheng Yu, Sheng Liu, Yuan Feng, Min Xu, Zhelun Jin, Xuhua YangAAAI 2025 · DOI

TL;DR

This paper introduces OLMD, a novel framework for continuous sign language recognition (CSLR) that tackles the challenges of multi-orientational and long-term motions. By decoupling motion into horizontal and vertical components and aggregating long-term temporal information, OLMD achieves state-of-the-art performance on major CSLR benchmarks.

What Problem Does This Paper Address?

Continuous Sign Language Recognition (CSLR) aims to translate untrimmed sign language videos into sequences of glosses (the basic units of sign language). This is a complex task due to the inherent nature of sign language, which involves not only the movement of hands but also their orientation and the temporal evolution of these movements over extended periods. Existing CSLR methods often struggle with:

  • Long-term Dependencies: Signs can span many frames, and traditional models often focus on short, adjacent frame sequences, missing crucial temporal context.
  • Multi-orientational Motions: The orientation of hands and body plays a significant role in conveying meaning, but many models treat all motion information uniformly, leading to confusion and reduced accuracy.

This paper addresses these limitations by proposing a method that explicitly models and disentangles these complex motion characteristics.

Methods

The core of the proposed OLMD framework lies in its innovative Orientation-aware Long-term Motion Decoupling (OLMD) approach, which consists of several key modules:

  • Long-term Motion Aggregation (LMA): This module is designed to filter out static redundancies and effectively capture abundant features from long-term motions. It establishes a "motion context" for each frame, allowing the model to consider a sequence of frames rather than just immediate neighbors.
  • Orientation-aware Motion Purification (OMP): This is the central innovation for handling multi-orientational motions. It decouples the aggregated motion features into distinct horizontal and vertical components. These components are then individually purified using specialized 2D CNN blocks (HMP for horizontal, VMP for vertical) to enhance orientation-specific awareness. This purification process helps to disentangle and clarify the contribution of each orientation.
  • Stage Coupling: This mechanism integrates the purified horizontal and vertical features within each stage of the network, enriching the feature representation.
  • Cross-stage Coupling: This further enhances robustness by combining decoupled features from different network depths (multi-scale information).

The overall architecture uses a 2D-CNN as a feature extractor, followed by the LMA and OMP modules at each stage. The decoupled and coupled features are then processed by 1D-CNNs for local temporal modeling and a BiLSTM for global temporal modeling, culminating in a CTC loss for classification.

Datasets

The paper evaluates OLMD on three widely-used CSLR datasets:

  • PHOENIX14: A popular German sign language dataset with 1,295 words.
  • PHOENIX14-T: An expanded version of PHOENIX14, featuring a larger vocabulary and more German words, suitable for both recognition and translation tasks.
  • CSL-Daily: A large-scale Chinese sign language dataset with a diverse vocabulary and a significant number of daily life sentences.

Key Insights and Ideas

  • Decoupling is Key: The paper strongly advocates for decoupling motion into its constituent orientations (horizontal and vertical) as a fundamental step towards better understanding complex sign language. This allows the model to process and refine each directional component independently.
  • Long-term Context is Crucial: Explicitly aggregating long-term motion information (beyond adjacent frames) is shown to be vital for accurately recognizing signs that evolve over many frames, preventing misinterpretations.
  • Synergy of Components: The combination of LMA (for temporal context) and OMP (for orientation awareness), along with stage and cross-stage coupling, leads to significant performance gains. Each component plays a critical role in the overall success.
  • Lightweight and Effective: OLMD achieves SOTA results while maintaining a relatively lightweight architecture, making it suitable for real-time applications.

Key Results

  • State-of-the-Art Performance: OLMD achieves state-of-the-art (SOTA) performance on all three tested datasets: PHOENIX14, PHOENIX14-T, and CSL-Daily.
  • Significant WER Reduction: On PHOENIX14, OLMD reduces the Word Error Rate (WER) by an absolute 1.6% compared to the previous SOTA. Similar improvements are observed on PHOENIX14-T (1.0% reduction) and CSL-Daily (1.7% reduction).
  • Outperforms Multi-Cue Methods: Notably, OLMD surpasses even some multi-cue methods (which use additional information like keypoints) using only single-cue RGB video input.
  • Ablation Studies Confirm Effectiveness: Ablation studies demonstrate that each component (LMA, HMP, VMP, coupling mechanisms) contributes significantly to the overall performance, with the full OLMD model yielding the best results.

Limitations and Future Work

  • Decoupling Location: While decoupling at multiple stages (2, 3, and 4) improved performance, the paper doesn't explore decoupling at all stages or in a more fine-grained manner.
  • Decoupling Method: Among various decoupling strategies tested, AvgPool yielded the best results. Further exploration into more sophisticated decoupling operations could potentially yield further improvements.
  • Motion Context Length: The optimal motion context length for LMA was found to be 9 frames, balancing performance with computational cost. Future work could explore adaptive context lengths.
  • Future Directions: The authors suggest that OLMD establishes a robust baseline for future CSLR research, implying that further refinements and extensions of this approach are promising. The paper's success also opens avenues for exploring more complex motion representations and their integration into CSLR systems.

Summary updated

BibTeX
@inproceedings{yu2025olmd,
  author = {Yiheng Yu and Sheng Liu and Yuan Feng and Min Xu and Zhelun Jin and Xuhua Yang},
  title = {{OLMD: Orientation-aware Long-term Motion Decoupling for Continuous Sign Language Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33052},
}
Figure