Sign language generation means turning text into the 3D body, hand and face movements of a signer. Nearly every sign language has very little data. Training one model on several sign languages at once should help, but "one model" hides a design choice. A sign generator has two jobs. The first is lexical and grammatical: deciding which signs to produce and in what order. That job is specific to each language, because American, German and Chinese Sign Language are unrelated languages with different vocabularies. The second is articulatory: producing physically valid handshapes, arm paths and smooth transitions between consecutive signs. That job plausibly draws on a largely shared human repertoire. Recognition work has already found that visually similar signs recur across unrelated sign languages even when their meanings differ (Improving Continuous Sign Language Recognition with Cross-Lingual Signs). This note proposes to find out where multilingual transfer actually lives in sign generation, and to build a model around the answer. The model has one articulatory prior shared by all languages. That prior is expressed in the parameters of a parametric body model (SMPL-X: a low-dimensional 3D template of body, hands and face, controlled by joint rotations and shape coefficients). Language-specific routing handles the mapping from words to signs.
The expected payoff is concrete. The current best multilingual 3D sign generator still struggles with co-articulation, meaning how one sign flows into the next (Signs as Tokens: An Autoregressive Multilingual Sign Language Generator). Co-articulation is the part of signing most likely to be shared across languages. If the hypothesis holds, a low-resource sign language should inherit fluent, anatomically valid transitions and hands from its unrelated neighbours without being pushed toward their vocabulary.
Core Claim
In multilingual 3D sign generation, most of the benefit of joint training comes from sharing the articulatory layer, not from sharing the linguistic layer. Sharing the linguistic layer across unrelated sign languages gives small gains or causes interference. The consequence is that the best design shares a continuous, SMPL-X-parameterized motion prior across all languages, and gives each language its own lightly routed path from text to that prior. Neither full sharing nor full isolation should do as well.
This is a different claim from the closest existing systems, which pick one end of the spectrum without testing where transfer comes from. SOKE (Signs as Tokens: An Autoregressive Multilingual Sign Language Generator; retrieval-enhanced version) already combines SMPL-X poses with a multilingual text model. It turns each body part's motion into discrete tokens using separate learned codebooks for the upper body and each hand, then fine-tunes a multilingual encoder-decoder (mBART) to emit those tokens for ASL, CSL and DGS. Everything is shared, and the paper does not ask which shared component carries the benefit. SignLLM (SignLLM: Sign Language Production Large Language Models) sits at the opposite end. It covers eight languages but gives each its own encoder-decoder parameter group, and it predicts unconstrained keypoints rather than a body model. MLSLT (MLSLT: Towards Multilingual Sign Language Translation) shows that soft routing between shared and language-specific modules beats separate bilingual models, but only in the sign-to-text direction and on 2D keypoints. Our contribution is to separate the two layers and measure each one's share of the transfer. The method then follows from the measurement.
Research Program
The work has three stages.
Stage 1: a controlled two-by-two study on SOKE's architecture. The first factor is whether the articulatory layer is shared or trained separately per language. That layer is the part-wise SMPL-X motion codebooks together with their decoder. The second factor is whether the text-to-motion generator is fully shared or split into per-language paths. The per-language paths are adapters with learned soft gating, as in MLSLT's intra-layer routing. We will not simply fine-tune the whole multilingual model. Motion-Aligned Word Embeddings for Text-to-Motion Generation reports that full fine-tuning of a large text encoder on small motion data overfits. That result limits our design: the text side stays mostly frozen, and only adapters and embeddings are trained. All four cells use SOKE's curated SMPL-X annotations for How2Sign, CSL-Daily and Phoenix-2014T. We also vary how much target-language data is available by training with 10%, 25% and 100% of the DGS and CSL training sets, because the value of sharing should grow as data shrinks.
Stage 2: a continuous shared articulatory prior. If Stage 1 confirms the hypothesis, we replace the shared discrete codebooks with a continuous prior over SMPL-X sequences. This is a rectified-flow motion model, a generator that learns a straight-line path from noise to data, following MoLingo. It is trained on pose data pooled from all three sign languages with no text. It keeps separate latent streams for the hands and the body. The separation follows EMAGE, which found that whole-body models let large torso motion wash out finger detail. Poses are represented as bone directions and lengths with a bone-consistency loss, following Sign-IDD, which showed that raw joint coordinates produce broken limbs and "melted" fingers.
Each language's routed text encoder conditions this shared prior. The encoder does not produce motion directly. It produces a sequence of sign-level conditioning vectors, and the shared prior turns them into motion, including the transitions between signs. We move from discrete to continuous only in this stage. That way the study's conclusion does not depend on how the motion is quantized, and the continuous version is justified by whatever the codebook ablation shows, not assumed in advance.
Stage 3: the analysis that makes the claim falsifiable. Using gloss alignments from Phoenix-2014T and CSL-Daily, we split every generated sequence into within-sign windows and between-sign transition windows, and score each window separately. The hypothesis predicts a specific pattern:
- Sharing the articulatory layer should improve transition windows and hand-pose accuracy most.
- Sharing the linguistic layer should mainly affect whether the right sign is produced.
- For unrelated languages, that linguistic effect may be negative.
If sharing the linguistic layer turns out to help the lexical measures as much as articulatory sharing helps the motion measures, the claim fails and full sharing is the right design.
Baselines
Separate single-language models (the naive reference). For each language we train the SOKE architecture alone on that language, with the same parameter count per language and the same number of optimizer steps as the multilingual runs. This is the "do nothing multilingual" point. It shows whether any form of sharing helps at 10%, 25% and 100% data.
SOKE as published (the strongest existing method). It shares everything and adds retrieval from sign dictionaries. The retrieval-enhanced paper reports dynamic time warping (DTW) error, a distance between generated and reference motion sequences after aligning them in time. Its results are 2.35 on How2Sign, 1.71 on CSL-Daily and 1.38 on Phoenix-2014T (Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator). We will rerun it from released settings on our exact splits and data fractions, with the same total compute and hyperparameter-search budget as our model. We will report both the published and the reproduced numbers. Our model beats SOKE if it lowers DTW on the full-data splits, even modestly, and clearly lowers it in the 10% and 25% regimes. It must also show lower error on hands and transition windows and no loss in back-translation accuracy. We give our model the same dictionary retrieval so that retrieval cannot explain any difference.
SignLLM-style hard isolation (the opposite design). We implement per-language encoder-decoder groups in the spirit of SignLLM's switching framework, but on our SMPL-X representation. Using the same representation keeps the comparison about sharing and not about keypoints versus body model. The file contains no numeric generation result for SignLLM on Prompt2Sign, so we cite no number for it.
Generic-motion prior ablation (isolates the key ingredient). We keep our architecture but pretrain the shared articulatory prior on non-sign SMPL-X motion: HumanML3D and BEAT2, as unified in MotionCraft's benchmark (Adding Multi-modal Controls to Whole-body Human Motion Generation). The data volume matches that of the pooled sign data. If this prior matches the multilingual sign prior, then the gain comes from generic motion smoothness, not from a shared sign phonology, and the claim about multilingual sign structure is wrong.
Why not just…?
Why not just pool all languages into one fully shared model, as SOKE does? This is the real threat. It may be that full sharing already captures most of the articulatory benefit, because SOKE's codebooks are trained on all three languages. The two-by-two study settles this directly. If fully shared and articulatory-only sharing tie at every data fraction, we will report that routing the linguistic layer is unnecessary for these three languages, and the contribution becomes the diagnostic result rather than the architecture. What favours our hypothesis is outside evidence of interference among unrelated languages. GradSim finds that languages whose gradients point in conflicting directions damage each other in a shared model. MLSLT beats separate bilingual models, with BLEU4 on CSL-to-English rising from 3.47 to 5.19, using soft routing rather than full sharing.
Why not just train separate models per language, or hard-partition parameters as SignLLM does? Isolation cannot transfer anything. MLSLT's unified model beat separate bilingual models while using as little as a tenth of the parameters. The cross-lingual recognition study cut German word error rate by 0.4 to 0.8 points using only Chinese data. The low-data regimes in our study measure how much isolation loses in generation specifically.
Why not just use a strong motion prior trained on ordinary human motion? Generic SMPL-X corpora are far larger than sign corpora, and they may teach smooth, anatomically valid motion equally well. The generic-motion prior ablation tests exactly this. We expect it to fall short on hands, because generic corpora have little finger articulation, and EMAGE had to discard about 16 hours of capture data because of finger-tracking noise. That expectation is not yet evidence, so the ablation decides.
Why not just predict keypoints and skip the body model? Sign-IDD shows that unconstrained joint coordinates break bone orientation and fingers, and in sign language that changes meaning. A shared articulatory prior also only makes sense in a space where handshape is defined the same way for every signer and every language, which the body model's joint rotations provide.
Proposed Experiments
The main benchmark is SOKE's SMPL-X versions of How2Sign (ASL), CSL-Daily (CSL) and Phoenix-2014T (DGS), evaluated at 10%, 25% and 100% of the training data for each target language. We also run leave-one-language-out adaptation: train the shared prior and routing on two languages, then fit the third from 10% of its data.
We report five metrics:
- whole-sequence DTW, for direct comparison with SOKE;
- DTW separately for each body part;
- hand joint-rotation error, measured on the MANO hand parameters inside SMPL-X;
- DTW restricted to transition windows versus within-sign windows;
- back-translation, where a fixed sign-to-gloss recognizer trained on real SMPL-X sequences reads the generated motion and we score the output. This gives WER and BLEU on glosses.
Sign-IDD reports BLEU-1 of 24.80 on Phoenix-2014T. That number comes from a gloss-to-pose setup, so we cite it as context only and not as a directly comparable target.
The decisive check is the pattern in the two-by-two study. Articulatory sharing should produce the largest improvement in transition-window and hand errors, and that improvement should grow as target data shrinks. Linguistic sharing should give at most a small improvement in back-translation accuracy. The multilingual sign prior should beat the generic-motion prior on hands. If all three hold, we have evidence that multilingual transfer in sign generation is mainly articulatory, and a design that uses it. If any one fails, the study tells the field which of SOKE's shared components is actually doing the work.
References
- Improving Continuous Sign Language Recognition with Cross-Lingual Signs
- Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator
- SignLLM: Sign Language Production Large Language Models
- MLSLT: Towards Multilingual Sign Language Translation
- Motion-Aligned Word Embeddings for Text-to-Motion Generation
- MoLingo: Motion-Language Alignment for Text-to-Motion Generation
- EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
- Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production
- Adding Multi-modal Controls to Whole-body Human Motion Generation
- GradSim: Gradient-Based Language Grouping for Effective Multilingual Training