TL;DR
This paper introduces a novel approach to generating realistic Indian Sign Language (ISL) videos from text by leveraging a co-articulated gloss dictionary and an interpolation-based synthesis pipeline. They also present SanketVaani-1K, a new 1,000-sentence ISL dataset with fine-grained gloss annotations, addressing the critical lack of resources for ISL Sign Language Production (SLP).
What Problem Does This Paper Address?
Millions of deaf and hard-of-hearing individuals worldwide rely on sign languages for communication. However, creating accessible digital content for them is challenging due to the scarcity of resources for Sign Language Production (SLP) systems, especially for low-resource languages like Indian Sign Language (ISL). Existing ISL datasets are often designed for sign language recognition, lacking the detailed gloss-level annotations needed for generating fluid and contextually accurate sign language videos. This gap hinders the development of SLP systems that can bridge communication barriers in education, healthcare, and public services for ISL users.
Methods
The core of their approach is an interpolation-based synthesis pipeline. Instead of generating entire sign language videos from scratch, they break down the process:
- Text-to-Gloss Translation: An LLM translates the input English sentence into a sequence of ISL glosses, guided by a concise ISL rule card to ensure grammatical correctness.
- Vector Database Construction: They build a vector database of gloss embeddings. Each embedding captures the lexical identity and contextual meaning of a sign, linked to its motion data and metadata (sign type, position, surrounding context). This allows for retrieving semantically relevant sign segments.
- Co-articulated Gloss Retrieval: For each gloss in the translated sequence, they query the vector database to find matching contextual embeddings. This retrieval process is crucial for sense disambiguation, ensuring the correct sign is chosen for polysemous words.
- FramePack Interpolation: This is the key novel component for synthesis. They use the modern FramePack text-to-motion framework. For each boundary between retrieved gloss segments, FramePack takes the last frame of the preceding sign and the first frame of the succeeding sign as anchors. It then synthesizes the intermediate frames, creating smooth, contextually accurate, and grammatically consistent transitions between signs. This method is designed to avoid the temporal discontinuities and artifacts common in naive interpolation.
Datasets
- SanketVaani-1K (New Dataset): The authors introduce this new dataset, comprising 1,000 spoken sentences translated into ISL by a native interpreter. Crucially, it features detailed, fine-grained gloss-level annotations with precise temporal boundaries, making it suitable for SLP tasks. The dataset covers over 40 high-utility domains relevant to the Indian socio-cultural context.
- RWTH-PHOENIX-Weather 2014T: Used for evaluating the Text-to-Gloss generation component, as a direct ISL parallel text dataset was unavailable.
Key Insights and Ideas
- Gloss-level supervision is critical for SLP: The paper strongly emphasizes that high-quality, fine-grained gloss annotations are essential for building effective SLP systems, a gap that existing datasets often fail to fill.
- Interpolation as a scalable path: By leveraging modern text-to-motion frameworks like FramePack, interpolation offers a practical and scalable way to generate fluid sign language, especially for low-resource languages.
- Contextual embeddings for disambiguation: Using contextualized gloss embeddings in a vector database is key to handling the ambiguity of words that can have different meanings and thus different signs in ISL.
- FramePack's advantage in motion synthesis: The FramePack method's bidirectional, anchor-based interpolation proves superior in maintaining fine-grained details like handshape and facial expressions, leading to more natural-looking sign synthesis compared to other methods.
Key Results
- Text-to-Gloss Generation: The LLM-based approach achieved competitive scores on the RWTH-PHOENIX-Weather 2014T dataset, outperforming previous methods in TSOV accuracy and BERTScore, indicating better semantic alignment.
- Vector Database Retrieval: The system successfully disambiguated ambiguous queries, retrieving semantically consistent signs for words with multiple meanings.
- Interpolation Quality: Qualitative comparisons showed FramePack significantly outperformed other SOTA interpolation methods (DreamMover, BiM-VFI, VFI-Mamba) in preserving facial detail and hand articulation, resulting in more coherent sequences.
- User Study: A user study with 12 Deaf participants revealed that FramePack-generated videos were consistently rated higher in understandability, fluency, and naturalness, achieving an overall Mean Opinion Score (MOS) of 3.15/5, outperforming baseline SLP methods like Spoken2Sign and UniAnimate-DiT.
Limitations and Future Work
- Single Signer Limitation: The current system uses a single signer, which limits generalization to variations in signing styles across different individuals.
- Approximate Gloss Boundaries: Manual gloss boundaries are not perfectly precise, which can lead to minor temporal misalignments or co-articulation artifacts during segment reuse.
- Fixed-Length Interpolation: The current interpolation method uses fixed lengths, which might not capture the true, context-dependent duration of glosses in natural signing.
- Anonymity: Preserving the signer's identity and expressions is central to the method, making anonymous generation difficult.
- Future Work: The authors plan to refine onsets/offsets with boundary-aware cues, incorporate duration predictors for context-dependent interpolation, align labels with ISLRTC updates, and involve Deaf community members more deeply in advisory roles, annotation review, and evaluation.
Summary updated