Systems that turn text into signing usually generate a skeleton, a sequence of body, hand and face keypoints, and train it to match recorded signers frame by frame. That loss can be driven down by moving the hands roughly right while leaving the face nearly still. The face is not decoration. In German Sign Language (DGS) and other sign languages, mouth shapes, eyebrows and head movement carry words and grammar. A natural fix is to fine-tune the generator against a sentence-level comprehension score: a frozen sign recognizer reads the generated sequence back into words. This runs into two problems at once. The score is a single number per sentence, which says nothing about which half-second of which body part failed. And unconstrained fine-tuning lets the generator trade away the hand motion it already learned in pursuit of that number.
We propose to fine-tune only a small, protected set of generator parameters that drive the face and head. The parameters that drive the hands are frozen exactly, so manual quality cannot degrade. The sentence-level score is turned into dense credit by a cheap counterfactual: we swap the generated face trajectory in one time window back to the original model's face trajectory, then measure how much the recognizer's confidence in the correct sentence changes. In two sentences: train the face against what a reader understands, credit each face window by how much it alone changes understanding, and lock the hand parameters so they cannot be spent on it.
Core Claim
Reward fine-tuning of sign production, combined with parameter isolation for channels and counterfactual credit assignment per channel and per window, will improve how readable the non-manual channel is. We measure readability as back-translation error from face keypoints alone. The gain should come with no loss in manual accuracy. The specific new element is the pairing of the two. Isolation makes the "no manual damage" guarantee exact, and counterfactual substitution makes the sparse reward usable without search or a learned value model.
How This Sits Against Prior Work
Three prior lines motivate the problem. Geometry-Aware Losses for Structure-Preserving Text-to-Sign Language Generation shows that frame-wise squared-error regression shrinks articulators toward the average pose. That paper fixes the shrinkage for the body and hands with bone-structure losses but leaves the face out, and the face is where this proposal acts. FePh annotates facial expressions on PHOENIX frames and finds that mouthing of spoken words looks like emotional expression. This constrains us: a face channel rewarded naively could drift toward "emotional" faces that fool an affect model but carry no language. Multi-channel recognition work such as Sign Language Transformers trains a recognizer jointly with Connectionist Temporal Classification (CTC), a loss that scores a whole gloss sequence without frame alignments, and a translation decoder. We reuse this kind of recognizer as the frozen reader whose sentence log-probability is the reward.
The closest sign work using reinforcement learning is SignLLM. It weights the production loss by a batch-level quality reward and prioritizes high-reward batches. It also isolates parameters coarsely, with whole encoder-decoder groups per language. Our proposal differs on both counts. Credit is assigned inside a sequence, per channel and per window. Isolation happens inside one shared network, split by articulator rather than by language. Continual Learning in Multilingual Sign Language Translation isolates language-level adapters for translation and finds that they lose to replay in low-resource transfer. That warns that rigid isolation can block useful sharing. We test this directly by letting the face subnetwork read the frozen manual features rather than walling it off.
The isolation machinery comes from continual learning. Winning SubNetworks learns importance scores per weight and freezes earlier subnetworks, giving zero backward transfer. Functional Task Networks proves that disjoint masks give zero forgetting only if Adam's moment estimates are reset between phases. We adopt that reset, because without it the guarantee is false. Our "tasks" are not a sequence of datasets. They are two training phases on the same data: supervised training of everything, then reward training of the face only. That avoids the task-identity problem these methods face at inference, since the mask is fixed after training.
The credit-assignment idea contrasts with the step-level reward methods in the file. ReasonRAG and Step-Level Q-Value Models build dense supervision from tree-search rollouts, which is costly and suited to discrete steps. VLAC learns a pairwise progress critic but admits it assumes progress rises over time. Signing violates that assumption, because facial markers span several signs and relax between them. Counterfactual window substitution needs no search, no learned critic, and no assumption that progress is monotonic.
Research Program
We start from a text-to-pose generator trained with the geometry-aware losses above, extended with face and head keypoints, on PHOENIX14T. We then learn importance scores in the style of Winning SubNetworks for the face/head output loss, with a budget of 5–15% of weights. This marks a subnetwork M_face; all other weights are frozen, and Adam state is reinitialized.
During reward training, Gaussian noise is added to the face/head outputs so the generator becomes a stochastic policy. For window $w$ of the face channel, the advantage is
$$A_w = \log p_R(y \mid \hat{x}) - \log p_R(y \mid \hat{x}^{(w \leftarrow \text{ref})}).$$
Here $p_R$ is the frozen recognizer's CTC probability of the reference gloss sequence $y$, $\hat{x}$ is the generated pose sequence, and $\hat{x}^{(w \leftarrow \text{ref})}$ is the same sequence with the face trajectory in window $w$ replaced by the frozen supervised model's face trajectory. The likelihood-ratio gradient for the noise sampled in window $w$ is weighted by $A_w$. A small penalty keeps M_face outputs close to the supervised face, to discourage drift.
The cost is one recognizer forward pass per window, which is tens per sentence rather than a tree search. To limit reward hacking against a single reader, the training reader and the evaluation reader differ in architecture and input features.
Baselines
The do-nothing reference is the supervised geometry-aware generator with face keypoints added. Its published counterpart reports back-translation BLEU-4 of 14.22 on PHOENIX14T (Geometry-Aware Losses). That is the strongest directly comparable production number in the file. SignDiff reports BLEU-4 of 22.15 on PHOENIX14T, but the file does not give its evaluation protocol, so we report it without claiming direct comparability. For face-only back-translation, no reported result was found on Lacuna, and we will establish it.
The strongest competitor is full-parameter reward fine-tuning with a KL-style penalty to the supervised model, using the same reader, data, steps, and hyperparameter search budget. It can improve the face just as well as our method if isolation is unnecessary. Beating it means equal or better face-only back-translation with significantly lower manual degradation, measured by bone-length error and manual DTW distance. A second competitor is SignLLM-style batch-level reward weighting, which isolates the contribution of credit at the window level.
Ablations remove one ingredient at a time. Isolation without counterfactual credit uses one sentence-level advantage for all windows. Counterfactual credit without isolation is the full fine-tuning variant. A random mask of equal size tests whether learned face importance matters. A run without the Adam reset tests whether the Functional Task Networks warning holds in practice. All variants get identical tuning budgets and three seeds.
Why not just…?
Why not just upweight the face loss in supervised training? This keeps the dense frame targets that are averaging toward a still face, so it scales the "safe bet" rather than escaping it. The geometry-aware results show that loss reweighting tames shrinkage only when the loss encodes structure. We include face-reweighted supervised training as a baseline. If it matches face-only back-translation, the reward component is unnecessary and we would report that.
Why not just fine-tune everything with a KL penalty? This is the real threat. It is the standard recipe, and the penalty may already protect the hands. Our bet is that a sentence reward dominated by manual signs pulls gradient toward the hands, so an alternative that only softly protects them must trade face gain against manual damage. The baseline comparison settles this by plotting both quantities across penalty strengths. If the KL curve dominates ours, isolation adds nothing.
Why not just put a LoRA adapter on the face output head? A low-rank adapter restricted to face outputs is a form of isolation and is cheaper than learned masks. We run it in two forms, head-only and all-layer. If all-layer face LoRA matches the learned mask, the conclusion is that isolation matters and granularity does not. That outcome is acceptable. Head-only LoRA probably lacks the context needed to time face markers against hand motion.
Why not just backpropagate through the recognizer? A differentiable reader gives dense gradients for free. However, such gradients are known to produce small perturbations that fool the specific model they come from. We include this as a baseline, with and without isolation, and judge it only with the held-out evaluation reader, which exposes reader-specific gains.
Proposed Experiments and Decisive Checks
PHOENIX14T is the main testbed. How2Sign, with its 2D/3D face keypoints, tests transfer to ASL. Metrics come in four groups. The first is back-translation BLEU-4 and WER for all channels, from the held-out reader. The second is face-only back-translation WER, from a reader trained on face and head keypoints, which reads mouthings. The third covers manual fidelity: bone-length error, DTW distance on hand joints, and motion variance. The fourth uses an emotion classifier trained on FePh: the distribution of predicted emotions on generated faces should match that on real signing faces, which guards against the mouthing-versus-emotion confound.
The decisive check has a clear success criterion. Face-only WER must drop meaningfully against the supervised reference, with manual bone-length error unchanged to within seed noise; the frozen weights make an exact match expected. It must also beat the KL fine-tuning curve at matched face gains. A second check compares window-level credit with a single sentence-level advantage at matched compute. If they are no different, the sparse-reward component is not pulling its weight. A third check is a small rating study with Deaf DGS or ASL signers on facial naturalness, which catches gains that fool both readers.
References
- Geometry-Aware Losses for Structure-Preserving Text-to-Sign Language Generation
- FePh: An Annotated Facial Expression Dataset for the RWTH-PHOENIX-Weather 2014 Dataset
- Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
- SignLLM: Sign Language Production Large Language Models
- Continual Learning in Multilingual Sign Language Translation
- Forget-free Continual Learning with Winning Subnetworks
- Functional Task Networks: Cortex-Inspired Spatial Parameter Isolation for Continual Learning
- Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- SignDiff: Learning Diffusion Models for American Sign Language Production