Towards Scalable Sign Production: Leveraging Co-Articulated Gloss Dictionary for Fluid Sign Synthesis

Aparna Agrawal, Seshadri Mazumder, C.v. Jawahar, Vinay P. NamboodiriICVGIP 2026 · OpenReview

TL;DR

This paper introduces a novel approach to generating realistic Indian Sign Language (ISL) videos from text by leveraging a co-articulated gloss dictionary and an interpolation-based synthesis pipeline. They also present SanketVaani-1K, a new 1,000-sentence ISL dataset with fine-grained gloss annotations, addressing the critical lack of resources for ISL Sign Language Production (SLP).

What Problem Does This Paper Address?

Millions of deaf and hard-of-hearing individuals worldwide rely on sign languages for communication. However, creating accessible digital content for them is challenging due to the scarcity of resources for Sign Language Production (SLP) systems, especially for low-resource languages like Indian Sign Language (ISL). Existing ISL datasets are often designed for sign language recognition, lacking the detailed gloss-level annotations needed for generating fluid and contextually accurate sign language videos. This gap hinders the development of SLP systems that can bridge communication barriers in education, healthcare, and public services for ISL users.

Methods

The core of their approach is an interpolation-based synthesis pipeline. Instead of generating entire sign language videos from scratch, they break down the process:

  1. Text-to-Gloss Translation: An LLM translates the input English sentence into a sequence of ISL glosses, guided by a concise ISL rule card to ensure grammatical correctness.
  2. Vector Database Construction: They build a vector database of gloss embeddings. Each embedding captures the lexical identity and contextual meaning of a sign, linked to its motion data and metadata (sign type, position, surrounding context). This allows for retrieving semantically relevant sign segments.
  3. Co-articulated Gloss Retrieval: For each gloss in the translated sequence, they query the vector database to find matching contextual embeddings. This retrieval process is crucial for sense disambiguation, ensuring the correct sign is chosen for polysemous words.
  4. FramePack Interpolation: This is the key novel component for synthesis. They use the modern FramePack text-to-motion framework. For each boundary between retrieved gloss segments, FramePack takes the last frame of the preceding sign and the first frame of the succeeding sign as anchors. It then synthesizes the intermediate frames, creating smooth, contextually accurate, and grammatically consistent transitions between signs. This method is designed to avoid the temporal discontinuities and artifacts common in naive interpolation.

Datasets

  • SanketVaani-1K (New Dataset): The authors introduce this new dataset, comprising 1,000 spoken sentences translated into ISL by a native interpreter. Crucially, it features detailed, fine-grained gloss-level annotations with precise temporal boundaries, making it suitable for SLP tasks. The dataset covers over 40 high-utility domains relevant to the Indian socio-cultural context.
  • RWTH-PHOENIX-Weather 2014T: Used for evaluating the Text-to-Gloss generation component, as a direct ISL parallel text dataset was unavailable.

Key Insights and Ideas

  • Gloss-level supervision is critical for SLP: The paper strongly emphasizes that high-quality, fine-grained gloss annotations are essential for building effective SLP systems, a gap that existing datasets often fail to fill.
  • Interpolation as a scalable path: By leveraging modern text-to-motion frameworks like FramePack, interpolation offers a practical and scalable way to generate fluid sign language, especially for low-resource languages.
  • Contextual embeddings for disambiguation: Using contextualized gloss embeddings in a vector database is key to handling the ambiguity of words that can have different meanings and thus different signs in ISL.
  • FramePack's advantage in motion synthesis: The FramePack method's bidirectional, anchor-based interpolation proves superior in maintaining fine-grained details like handshape and facial expressions, leading to more natural-looking sign synthesis compared to other methods.

Key Results

  • Text-to-Gloss Generation: The LLM-based approach achieved competitive scores on the RWTH-PHOENIX-Weather 2014T dataset, outperforming previous methods in TSOV accuracy and BERTScore, indicating better semantic alignment.
  • Vector Database Retrieval: The system successfully disambiguated ambiguous queries, retrieving semantically consistent signs for words with multiple meanings.
  • Interpolation Quality: Qualitative comparisons showed FramePack significantly outperformed other SOTA interpolation methods (DreamMover, BiM-VFI, VFI-Mamba) in preserving facial detail and hand articulation, resulting in more coherent sequences.
  • User Study: A user study with 12 Deaf participants revealed that FramePack-generated videos were consistently rated higher in understandability, fluency, and naturalness, achieving an overall Mean Opinion Score (MOS) of 3.15/5, outperforming baseline SLP methods like Spoken2Sign and UniAnimate-DiT.

Limitations and Future Work

  • Single Signer Limitation: The current system uses a single signer, which limits generalization to variations in signing styles across different individuals.
  • Approximate Gloss Boundaries: Manual gloss boundaries are not perfectly precise, which can lead to minor temporal misalignments or co-articulation artifacts during segment reuse.
  • Fixed-Length Interpolation: The current interpolation method uses fixed lengths, which might not capture the true, context-dependent duration of glosses in natural signing.
  • Anonymity: Preserving the signer's identity and expressions is central to the method, making anonymous generation difficult.
  • Future Work: The authors plan to refine onsets/offsets with boundary-aware cues, incorporate duration predictors for context-dependent interpolation, align labels with ISLRTC updates, and involve Deaf community members more deeply in advisory roles, annotation review, and evaluation.

Summary updated

Abstract

Sign Language Production (SLP) systems can significantly improve accessibility for Deaf communities by translating spoken or written language into sign language videos. For millions of Indian Sign Language (ISL) users, such systems could bridge persistent communication gaps in education, healthcare, and public services. However, SLP in ISL is hindered by the scarcity of annotated datasets and the expressive complexity of the language. Existing ISL datasets, such as iSign \cite{joshi2024isign} and ISL-CSLRT \cite{elakkiya2021islcsltr}, are designed for recognition and lack gloss-level\footnote{Gloss is word representation of sign.} annotations, forcing full-sentence modeling that struggles to generalize to new constructions. We propose an interpolation-based approach that stitches together individual gloss-level signs using modern text-to-motion frameworks like FramePack \cite{zhang2025framepack}. This produces contextually accurate and grammatically consistent sign sequences while retaining fine-grained articulation. To support this pipeline, we curate a 1,000-sentence ISL dataset, translated by a native interpreter and annotated with precise gloss boundaries. Results show that with high-quality gloss-level supervision, interpolation-based synthesis offers a practical, scalable path for inclusive SLP in low-resource sign languages like ISL. The project page can be found here
BibTeX
@inproceedings{agrawal2026towards,
  author = {Aparna Agrawal and Seshadri Mazumder and C.V. Jawahar and Vinay P. Namboodiri},
  title = {{Towards Scalable Sign Production: Leveraging Co-Articulated Gloss Dictionary for Fluid Sign Synthesis}},
  booktitle = {ICVGIP},
  year = {2026},
  url = {https://openreview.net/forum?id=LXIemMAhPY},
}
Figure