TL;DR
SIGNTOK is a continuous pose tokenizer that reorganizes the "working space" of sign language generators to prioritize linguistic meaning over individual signer style. By aligning motion with text at the word level during training, it achieves an 82% same-sentence cross-signer retrieval rate and significantly improves the recognizability and smoothness of generated sign language.
What Problem Does This Paper Address?
In Text-conditioned Sign Language Generation (SLG), the goal is to turn spoken text into a sequence of 3D poses. Most current models generate these poses either directly in "pose space" or in a latent space trained purely for reconstruction (like a VAE).
The authors identify a major bottleneck: these spaces are organized primarily by signer identity rather than linguistic content. Because every person has a unique way of moving, the same sentence signed by two different people can look as different in pose space as two completely unrelated sentences. This makes it incredibly difficult for a text-to-sign generator to find the "right" motion for a given sentence. As shown in Figure 1, standard representations cluster by who is signing, creating a fragmented space that ignores the underlying language.
Methods
The authors propose SIGNTOK, a continuous pose tokenizer based on a Masked Autoencoder (MAE) architecture. The core innovation lies in how the latent space is "shaped" during training using three auxiliary, training-only heads:
- Kinematics Head: A shallow Transformer that predicts the velocity and acceleration of masked frames. This ensures the latent space captures the physics and dynamics of motion.
- Sentence-level Head: This aligns the pooled latent of a clip with the global sentence embedding from a frozen text encoder (mBART).
- Word-level Alignment Head: This is the "secret sauce." Using a late-interaction contrastive objective (similar to FILIP), it matches short windows of motion to specific content words in the sentence. It also includes a learnable "null" token to account for transitions between signs that don't carry specific word meanings.
Crucially, these heads are discarded after training. The resulting frozen tokenizer provides a continuous, per-frame latent space that a generator (in this paper, a flow-matching model called SignFlow) can easily map text onto.
Datasets
The model was evaluated on three major sign language benchmarks:
- CSL-Daily: Chinese Sign Language.
- PHOENIX-2014T: German Sign Language.
- How2Sign: American Sign Language.
The authors used curated poses and standard splits to ensure fair comparison with existing state-of-the-art models.
Key Insights and Ideas
- Meaning over Identity: The paper argues that for generation, a latent space shouldn't just be good at reconstruction; it must be "generatable" from text. By forcing the latent to align with words, SIGNTOK bridges the gap between the discrete nature of text and the continuous nature of motion.
- The Power of Word-Level Alignment: Aligning at the word level is superior to sentence-level alignment. It allows the model to learn a "visual vocabulary" where the same word lands in the same region of the latent space, regardless of who is signing it.
- Continuous vs. Discrete: Unlike discrete tokenizers (VQ-VAEs) that can lose fine hand details through quantization, SIGNTOK’s continuous latent preserves precision while being much easier for a generator to navigate than raw pose space.
- Natural Smoothness: One of the most striking "aha moments" is that generating in a linguistically organized latent space naturally leads to smoother motion. Models generating in raw pose space often produce "jittery" movement because they are struggling to navigate a space dominated by signer-specific noise.
Key Results
- Semantic Organization: On CSL-Daily, the "same-sentence cross-signer nearest-neighbor rate" (how often the closest match to a clip is the same sentence by a different person) skyrocketed from 2% in pose space to 82% with SIGNTOK.
- Recognizability: The authors used "back-translation" (using a separate model to translate the generated signs back into text) to measure quality. SIGNTOK improved BLEU-4 scores by 20-70% relative to pose-space generation across the benchmarks.
- Motion Quality: SIGNTOK significantly reduced "jerk" (a measure of motion jitter), producing avatars that move much more naturally than previous pose-space or discrete-latent methods.
Limitations and Future Work
While SIGNTOK dramatically improves semantic organization, the authors note that signer identity is not entirely removed—nor should it be, as style is part of natural signing. However, the latent is still somewhat signer-dependent.
Additionally, while the meaning of the signs improved (as shown by back-translation), the joint error (how physically close the joints are to a specific ground-truth reference) did not always decrease. This highlights a known tension in SLG: a sign can be perfectly "correct" and recognizable even if it doesn't perfectly overlap with a single reference signer's specific proportions or style. Future work may involve exploring even more robust text encoders or further disentangling style from content.
Summary updated