TL;DR
SemRAG is a novel framework for Sign Language Production (SLP) that uses full-sentence signing templates to guide the generation of realistic, semantically accurate sign pose sequences. By combining sentence-level retrieval with a "semantic localization" module and adaptive gating, it produces more natural motion and better semantic consistency than previous word-level or purely generative methods.
What Problem Does This Paper Address?
Sign Language Production (SLP) is the task of translating spoken-language text into a sequence of sign poses. It’s a difficult balancing act: the output must be semantically accurate (the right signs in the right order) while maintaining the natural rhythm and fluid transitions of a human signer.
Existing methods generally fall into two camps:
- Purely Generative: These models learn to map text to poses directly. While flexible, they often suffer from "semantic drift," especially in long sentences, leading to missing or incorrect signs.
- Retrieval-Augmented (Gloss/Word Level): These models stitch together pre-recorded sign segments for individual words (glosses). This ensures realistic local motion but often results in "choppy" transitions and requires expensive, manual word-level annotations.
The authors of SemRAG argue that we should retrieve full-sentence templates. This preserves the global rhythm and natural "flow" of signing. However, sentence-level retrieval introduces a "semantic mismatch" problem: if the retrieved sentence is "It is a sunny day" but the target is "It is a rainy day," the model needs a way to ignore the "sunny" motion while keeping the rest of the template's structure.
Methods
SemRAG (Semantic-Guided Sentence-Level Retrieval-Augmented SLP) operates in three main stages:
- Sentence-Level Retrieval: Given an input sentence, the system searches a database for the most semantically and lexically similar sentence-pose pair.
- SLT-Guided Semantic Localization: To solve the semantic mismatch problem, the authors use a frozen Sign Language Translation (SLT) model as a "probe." By analyzing the cross-attention maps of the SLT model, SemRAG identifies which temporal regions of the retrieved pose sequence correspond to specific words. It then applies a "temporal relevance gate" to suppress motions that don't match the target text.
- Layer-Gated Generation: The filtered template features are injected into a Transformer-based generator. Instead of a uniform injection, the authors use layer-specific learnable gates. This allows the model to adaptively decide how much to rely on the retrieved template at different depths of the network (e.g., using it for global structure in early layers vs. fine-tuning in later ones).
Datasets
The model was evaluated on two standard benchmarks for continuous sign language:
- Phoenix14T: A German sign language dataset based on weather forecasts.
- CSL-Daily: A large-scale Chinese sign language dataset covering daily life topics. Both datasets provide spoken language text, gloss sequences, and corresponding video/pose data.
Key Insights and Ideas
- The Power of Context: Moving from word-level to sentence-level retrieval allows the model to inherit the "prosody" of sign language—the subtle ways signs change based on the signs that come before and after them.
- Translation as a Probe: Using a translation model to find "where" a word is in a pose sequence is a clever way to achieve alignment without needing manual, frame-by-frame labels.
- Adaptive Reliance: The layer-gate visualization (Figure 3d in the paper) reveals that the model doesn't use the template equally across all layers. It shows a "U-shaped" importance curve, relying heavily on the template in the initial layers to set the scene and in the final layers to refine the motion.
Key Results
SemRAG significantly outperformed both non-retrieval and retrieval-based baselines across several metrics:
- Semantic Consistency: On Phoenix14T, it improved the BLEU-4 score (via back-translation) from 9.41 to 12.29. On CSL-Daily, the improvement was even more dramatic, jumping from 3.01 to 12.37.
- Motion Quality: The model achieved lower Mean Per Joint Position Error (MPJPE), indicating more accurate pose generation.
- Human Evaluation: In perceptual studies, participants rated SemRAG higher for both "naturalness" and "motion similarity" compared to previous state-of-the-art models like SignPR and SOKE.
Limitations and Future Work
- Template Dependency: The system's performance is inherently tied to the quality of the retrieval database. If the input sentence is completely unlike anything in the training set, the model falls back to standard text-to-pose generation.
- 2D vs. 3D: The current work focuses on 2D keypoints. Extending this to 3D mesh generation (like SMPL-X) or direct video synthesis would be a natural next step.
- Scale: The authors suggest that exploring larger-scale template collections could further improve the robustness of the retrieval mechanism.
Summary updated