TL;DR
This paper presents a novel end-to-end neural system for synthesizing sign language from spoken commentary, bypassing manual annotation and video segmentation. The system focuses on generating skeletal models rather than direct video, offering greater versatility, and introduces a new soft non-monotonic attention mechanism to improve synchronization between text and generated signs.
What Problem Does This Paper Address?
Creating realistic sign language (SL) synthesis systems is challenging due to the scarcity of annotated SL data and the complexity of human motion. Traditional approaches often rely on manual annotation, which is time-consuming, costly, and prone to inconsistencies. Furthermore, existing methods often split the problem into separate translation and animation stages. This paper tackles the problem of building a fully automatic, end-to-end SL synthesis system that can learn directly from readily available, unannotated broadcast data, specifically focusing on Czech Sign Language (CSE) weather forecasts. The goal is to generate accurate skeletal representations of signs that can then be animated by a virtual avatar, avoiding the need for explicit SL translation or video segmentation.
Methods
The authors propose a two-pronged approach: first, extracting high-quality skeletal models from video, and second, building a trainable end-to-end SL synthesis system.
For skeletal model extraction, they leverage the OpenPose framework for initial 2D skeleton detection. However, OpenPose often produces errors like missing joints or inaccurate finger positions. To address this, they introduce a 3D model-based skeletal model correction method. This iterative, gradient-descend-based algorithm reconstructs missing joints and ensures geometrical consistency by utilizing bone lengths and symmetry as invariants. They also incorporate a skeletal model scaling step to normalize variations in speaker size.
For SL synthesis, the system takes spoken commentary as input and directly generates a sequence of skeletal models. Key innovations include:
- Feed-forward and Recurrent Transformers: Investigating both architectures for sequence-to-sequence translation.
- Soft Non-Monotonic Attention: A novel attention mechanism designed to replace traditional Dynamic Time Warping (DTW) for synchronizing input text with output skeletal sequences. This soft attention allows for more flexible alignment and better gradient propagation compared to hard attention.
- Character-Level Features: Exploring the use of character-level features alongside word-level features to handle the inflectional nature of Czech and mitigate Out-Of-Vocabulary (OOV) issues.
- Simplified Translator: A simplified sequence-to-sequence translator that produces an output text with the same length as the input, making the entire system trainable.
Datasets
The primary dataset used is an internet archive of The Czech Daily News in CSE, specifically focusing on weather forecasting segments. This corpus consists of:
- 947 videos (approximately half a minute each) from September 2015 to July 2018.
- Performed by five different CSE speakers.
- Approximately 500,000 frames and 40,000 words in total for training.
- 1500 words reserved for development tests and 1500 words for testing.
The spoken commentaries were automatically transcribed using speech recognition software. No manual video annotation or segmentation was performed.
Key Insights and Ideas
- End-to-End Synthesis Without Manual Annotation: The most significant insight is the successful development of an end-to-end system that bypasses the need for costly and time-consuming manual annotation of sign language videos. This opens doors for leveraging large, unannotated broadcast data.
- Skeletal Models as a Versatile Representation: Generating skeletal models instead of direct video offers greater flexibility for animation with different avatars and allows for easier manipulation and correction of poses.
- Soft Non-Monotonic Attention for Synchronization: The proposed soft non-monotonic attention mechanism is a crucial advancement for aligning text with sign language sequences, offering a more robust and differentiable alternative to DTW, especially for handling variable sign lengths and potential misalignments.
- Character-Level Features for Inflectional Languages: Using character-level features proves beneficial for languages like Czech, which have complex inflectional morphology, by better handling variations of words and reducing OOV issues.
- Importance of 3D Correction for Skeletal Models: Raw 2D skeletal data from pose estimation tools often contains errors. The 3D correction method is vital for producing accurate and geometrically consistent skeletal models, which are essential for realistic sign synthesis.
Key Results
The paper reports several key findings:
- Lowest Test Error: The best performing system achieved a test set error of 11.72, which was an improvement over previous lowest errors.
- Benefit of Soft Non-Monotonic Attention: Combining DTW with soft non-monotonic attention yielded significant improvements, with the "Comb." (combination) criterion consistently outperforming DTW or non-monotonic attention alone across various configurations.
- Feed-Forward vs. RNN Translators: Feed-forward translators showed comparable results to RNN-based translators but were faster to train and infer.
- Character-Level Features: While character-level features (CLF) slightly increased the error compared to word-level features in some configurations (e.g., 11.94 vs. 11.72), the authors suggest they could be more beneficial for corpora with larger vocabularies.
- Oracle Synthesizer: An oracle synthesizer, representing an ideal scenario with perfect annotations, achieved a significantly lower error of 10.34, indicating the potential for further improvement.
Limitations and Future Work
Limitations:
- Data Sparsity: The corpus, while large for SL research, is still considered sparse, especially when dealing with the variability of sign lengths and sentence structures within broadcast segments.
- Hard Attention Gradient Issues: Traditional hard attention mechanisms (like DTW) can lead to null gradients, hindering training. The soft attention aims to mitigate this.
- Skeletal Model Errors: Despite correction, some errors in skeletal models (e.g., finger joint misplacements) can still occur.
- Limited Context for Character-Level Features: While character-level features were explored, they led to longer sequences and a slight increase in error, suggesting that current models might not fully exploit the wider context needed.
Future Work:
- Larger Corpus: Utilize the entire Czech news archive to create a significantly larger corpus.
- Facial Expressions: Incorporate modeling of facial expressions, which convey crucial information in SL.
- More Robust Translators: Further explore and improve the robustness of translators, especially for handling the complexities of SL grammar and sentence structure.
- Investigate Other Architectures: Continue to explore different neural network architectures for both skeletal extraction and synthesis.
Summary updated