Systems that turn written text into sign language video now skip the expert-written "gloss" transcripts that older pipelines needed. A gloss is a word-by-word written label for each sign, and it is slow and expensive to produce. These systems predict a skeleton pose from text and then render video from that pose with an image diffusion model, a generator that turns random noise into an image through repeated denoising. The rendering stage has a known failure. When the hands cross, or a hand passes in front of the face, the generated frames confuse hand orientation and merge fingers, as SignDiff reports. Image generation has a name for this failure, attribute leakage: the appearance of one entity drifts into another because every pixel can attend to every other pixel. That literature also has a remedy, spatial attention masking, which edits the model's attention scores so that pixels in one region cannot draw features from another.
We propose to carry that remedy into sign video generation in a form built for a hard case: masks that are small, move every frame, and overlap constantly. The model is not allowed to decide each pixel's owner through attention. Instead, the pose it is already conditioned on fixes each pixel's owner: left hand, right hand, face, or shared body and background context. When articulators overlap, depth order from the 3D pose settles which one owns the pixel. Only the middle layers and early denoising steps are masked, so global lighting and hand-face contact still render coherently. The masks come for free from the conditioning pose, so the method adds no annotation and fits the gloss-free setting. We then test whether it preserves the information a sign reader actually needs, the hand configuration and the mouthing, using gloss-free translation models as automatic readers.
Core Claim
The hand confusion in pose-to-video sign generation is mainly self-attention leakage between articulators, not an error in the pose. Leakage between the two hands, and between the hands and face, should be concentrated in frames where articulators overlap. Soft, occlusion-aware ownership masks applied to self-attention in the middle layers should cut per-articulator rendering error in exactly those frames without hurting whole-frame realism. The claim can be falsified in three ways. The gains may not concentrate in overlap frames. A simple crop-and-refine pass over the hands may match them. Or masking may lower back-translation quality because it damages face-hand contact.
What Exists and What Is Missing
Several image methods show that masking attention stops leakage between regions. Localized Text-to-Image Generation for Free via Cross Attention Control limits each masked region to its own sub-prompt at inference time and raises attribute-binding success on CC-500 from 49.91% to 67.27%. Freestyle Layout-to-Image Synthesis sets out-of-box attention scores to negative infinity before the softmax. Dense Text-to-Image Generation with Attention Modulation shows that masking the text-to-pixel cross-attention is not enough, because pixel-to-pixel self-attention also carries features across regions. It therefore also blocks communication between segments in self-attention early in denoising, which raises COCO layout IoU from 23.76 to 34.99. All three supply our mechanism. All three also assume static scenes with large, non-overlapping regions. The CAC paper explicitly reports degradation under small or overlapping masks, which is the everyday condition of signing.
Two other papers constrain the design. Seg2Any finds that blocking self-attention across the whole network destroys shadows and scene coherence, and that restricting it only in middle layers keeps coherence while stopping bleeding. We adopt that layer selectivity as a starting hypothesis and test it, because a convolutional (UNet) renderer at low resolution may behave differently from its diffusion transformer.
CustomVideo is the closest prior work. In multi-subject text-to-video, it uses masks from the Segment Anything model to steer each subject's identifier token toward its own region and away from others. CustomVideo differs from our proposal in three ways. It acts on cross-attention to text tokens, whereas pose-to-video rendering is driven by pose and the leakage is pixel-to-pixel. Its subjects are separate objects with stable masks, whereas ours are parts of one body that occlude each other every few frames. And it reports failure beyond three subjects and blurred faces in wide shots, which is close to our regime of two small hands plus a face.
On the sign language side, SignDiff supplies the pipeline we modify. It is a gloss-free text-to-pose model followed by a Stable Diffusion 1.5 renderer conditioned on pose through a ControlNet branch (FR-NET). It leaves the renderer's attention untouched. Multi-channel Transformers and Disentangle and Regularize show that separating articulators helps, but they do it by cropping inputs into separate streams or splitting latent pose vectors. Neither separates articulators inside a pixel generator. Gloss Attention is the conceptual precedent. It shows that much of what glosses give a model is a structural bias on attention, and it restores that bias with a local window over time. We add the spatial counterpart for generation, and we use gloss-free translators of this kind as evaluation instruments rather than as the method.
Research Program
The central object is an ownership map, computed per frame from the conditioning pose. Hand and face keypoints are rasterized into dilated regions at each attention resolution. Where regions intersect, the pixel goes to the articulator nearest the camera, using wrist and fingertip depth from 3D pose (available in How2Sign) or from estimated depth elsewhere. A thin band around each boundary, plus the torso and background, is marked as shared context that any query may read. Inside the chosen self-attention layers the scores become
$$A = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}} + B\right), \qquad B_{ij} = \begin{cases} 0 & \text{if } o(j) = o(i) \text{ or } j \text{ is shared context} \\ -\beta & \text{otherwise.} \end{cases}$$
Here $Q$ and $K$ are the query and key features of the pixel latents, $d$ is their dimension, $o(i)$ is the articulator that owns pixel $i$, and $\beta \ge 0$ is a penalty strength. Setting $\beta \to \infty$ recovers the hard Freestyle-style mask. A finite $\beta$ lets strong evidence through, so a hand resting on the cheek can still pick up the right shading. If the renderer has temporal attention layers, the same ownership rule applies across frames, so left-hand pixels at frame $t$ read left-hand pixels at other frames. This targets swaps of handedness over time.
The work proceeds in three stages. First, a diagnostic. We instrument an unmodified SignDiff-style renderer and measure leakage mass, the fraction of self-attention weight that left-hand queries place on right-hand or face keys, per layer and timestep. We then correlate it with per-articulator rendering error, which tells us where leakage actually occurs rather than assuming it. Second, the training-free intervention, sweeping which layers are masked, over which timestep range, and at what $\beta$, with the Seg2Any middle-layer choice and the DenseDiffusion early-timestep window as priors. Third, masked fine-tuning. The renderer is trained with the ownership bias active, plus a small CustomVideo-style penalty on out-of-owner attention, to test whether the model can learn to exploit the separation rather than merely tolerate it at inference. Because the low attention resolution noted by DenseDiffusion (16×16) is too coarse for fingers, masks are applied only at the 32×32 and 64×64 attention layers. The resolution limit itself is reported as a measured factor rather than hidden.
Baselines
The naive reference point is the unmodified SignDiff-style renderer. We rebuild it on Stable Diffusion 1.5 with pose ControlNet conditioning, train it on the same How2Sign split, and run it with the same sampler and step count as every other variant. SignDiff reports SSIM 0.849, Hand SSIM 0.676, and FID 20.04 for its pose-to-video stage on How2Sign. Our reproduction must land close to these numbers before any comparison is meaningful, and all methods are then judged against our reproduction on identical conditioning poses.
The strongest existing masking method we can transplant is DenseDiffusion-style modulation with articulator masks substituted for object masks. It boosts in-mask and suppresses out-of-mask self-attention during the early denoising window, but has no depth ordering and no shared-context band. CAC and hard negative-infinity masking are run as the two extremes of the same family. Each gets the same hyperparameter search budget as our method on a held-out validation split. To beat these baselines, our method must reduce per-articulator keypoint re-detection error and palm-orientation error on overlap frames, over three seeds, by more than the spread across seeds. It must do so while keeping whole-frame FID within the seed variance of the unmodified renderer.
We also run a CustomVideo-style baseline: fine-tuning with an attention-alignment loss but no inference-time bias. This separates "learn to separate" from "force separation."
Three ablations isolate our key ingredients. The first replaces depth-ordered ownership with a plain union of masks, which should hurt only on overlap frames. The second removes the shared-context band. The third moves masking from middle layers to all layers, which should hurt realism and hand-face contact rendering if Seg2Any's finding transfers.
We found no reported result on Lacuna for per-articulator leakage or hand-orientation error in sign video generation. That metric is established by our own baseline runs.
Why not just…?
Why not just crop the hands and re-generate them at higher resolution? This is the most serious threat. It directly attacks the 16×16 resolution limit and might beat attention masking on isolated-hand frames. The difficulty is that the frames where SignDiff fails are the frames where the crop contains both hands, or a hand and the mouth. Refining the crop re-runs the same leaking attention at higher resolution. We implement a crop-and-refine pass using the same renderer and compute. If it matches our method on overlap frames, the leakage hypothesis is wrong and we will say so. If it wins on isolated frames only, we will test the combination, masking inside the refine pass.
Why not just use a stronger pose adapter? SignDiff already conditions on both skeleton and body shape through FR-NET and still reports hand orientation confusion. A pose adapter tells the model where each keypoint is, but nothing stops features from one hand being copied into the other's pixels. We test this claim directly by measuring leakage mass in the unmodified renderer. If leakage mass does not predict error, the adapter explanation wins and the masking idea is unsupported.
Why not just hard-mask everything? Seg2Any shows that network-wide self-attention blocking damages coherence, and CAC shows that hard masks degrade on small and overlapping regions, which are exactly the conditions of signing. We still run hard masking as a baseline, because the evidence for both effects comes from static images and could fail to transfer.
Why not just skip video and output skeletons? The How2Sign study found that signers understood synthesized video better than skeleton visualizations: 91.6% versus 83.3% category accuracy, and a mean opinion score of 2.58 versus 2.50. The margin in mean opinion score is small, so we do not treat video as settled. The rendering stage is where leakage lives, however, so it is where this question has to be answered.
Proposed Experiments and Decisive Checks
The primary benchmark is How2Sign, used for video metrics comparable with SignDiff. We add three articulator-level metrics. Per-articulator error is computed by running a keypoint detector on the generated frames and comparing them to the conditioning pose, separately for the left hand, right hand, and face landmarks. Palm-normal angular error and a handedness swap rate follow from the same detections. Every metric is reported separately on frames with no overlap, hand-hand overlap, and hand-face overlap.
Readability is measured by back-translation: a gloss-free translator trained on real video reads the generated video. For How2Sign we use the protocol of Neural Sign Actors, which reports back-translation BLEU-4 of 13.12 against a 7.51 baseline. Because that result is for mesh output rather than RGB video, we treat it as context, not a direct target.
For a channel-level probe we retrain the renderer on PHOENIX14T and read its output with a Multi-channel Transformer translator. That translator reports 19.51 BLEU-4 with all channels and about 16 with the hand channel alone on real video. Comparing hand-only and mouthing-inclusive BLEU on generated video tells us whether masking protects manual information at the cost of facial grammar, or protects both. A small native-signer study rates hand-face contact frames specifically.
The idea stands or falls on three checks. Articulator error reductions must concentrate in overlap frames and track reductions in measured leakage mass. The gains must survive the crop-and-refine baseline on those frames. And back-translation BLEU, including the mouthing channel, must not drop relative to the unmodified renderer. If the first check fails, leakage is not the cause. If the second fails, the simpler fix suffices. If the third fails, masking severs the face-hand coupling that sign grammar needs, and the right layer and $\beta$ settings become the finding.
References
- SignDiff: Diffusion Model for American Sign Language Production
- Localized Text-to-Image Generation For Free via Cross Attention Control
- Freestyle Layout-to-Image Synthesis
- Dense Text-to-Image Generation with Attention Modulation
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
- CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects
- Multi-channel Transformers for Multi-articulatory Sign Language Translation
- Disentangle and Regularize: Sign Language Production with Articulator-Based Disentanglement and Channel-Aware Regularization
- Gloss Attention for Gloss-free Sign Language Translation
- Neural Sign Actors: A diffusion model for 3D sign language production from text