Signing Less Where It Matters Less: Predictability-Weighted Effort for Continuous Sign Language Production

When people speak or sign, they spend less physical effort on parts of a message the listener can guess. Predictable words get shorter, vowels drift toward the center of the mouth, and neighbouring gestures blend into each other. Neural systems that turn text into sign language motion do the opposite. They apply one uniform loss to every frame and, as a result, blur everything toward an average pose. This note proposes a text-to-sign generator in which motion is a smooth continuous curve, not a stack of frames. The penalty on physical effort along that curve is made stronger for predictable signs and weaker for informative ones. A syntactic pre-ordering step first rearranges the German sentence into sign order. That step gives each linguistic unit its own contiguous stretch of time, and the effort weight can only be assigned per unit once that stretch exists.

In two sentences: generate sign motion as a spline curve whose smoothness penalty varies by sign, low where the sign carries information and high where context makes it predictable. The alignment that says which stretch of the curve belongs to which sign comes from reordering the source text into target order before generation. The bet is that this reproduces the reduction patterns of human signers and reduces mean-pose collapse at the same time. A uniform smoothness penalty would make that collapse worse.

Core Claim

The phonetics literature shows that reduction is systematic and depends on predictability. Predictable words show centralized articulator trajectories (McAuliffe et al., 2012). High-frequency words show gradient co-articulation (Hanique et al., 2012). Function words erode far more than content words (Magistro et al., 2022). Frequency within a grammatical role predicts reduction better than global frequency does (Samlowski et al., 2013). For signing, American Sign Language Handshapes Reflect Pressures for Communicative Efficiency measured handshape effort from 3D joint landmarks and found that frequent native signs have lower finger independence ($r=-0.46$). That study used static handshapes and a lexicon-level analysis. No one has tested whether continuous sentence-level sign motion shows predictability-linked reduction, and no generator uses such a pattern.

Our claim has two parts. The first is empirical: in continuous German Sign Language (DGS), the path length, peak velocity and duration of a sign's trajectory decrease with its contextual predictability. The second is modelling: building this dependence into the training objective of a continuous-curve generator produces motion that is more expressive and more human-like than either unregularized regression or uniform smoothing.

Mechanism

The motion for a sentence is a function $f(t)$ from time to body, hand and face landmark coordinates. It is written as a B-spline, a sum of $K$ fixed smooth bump functions $\phi_k(t)$ with predicted coefficients $c_k$. The functional representation is the medium for this proposal, not its contribution. It allows velocity and jerk (the third derivative of position) to be computed analytically. It also lets the curve be queried at any frame rate, and it keeps the curve continuous across sign boundaries without post-hoc stitching. Vocal tract contour analysis of emotional speech by the functional data curve representation showed that 16 B-spline coefficients give a compact, speaker-normalized description of vocal tract shape. That work was descriptive and did no generation. We borrow its representation and use it for generation.

The generator predicts one duration per ordered unit $j$, which yields a time interval $I_j$, and then predicts the spline coefficients. Adjacent intervals share knots, so the curve stays continuous across boundaries. Training minimizes the following loss:

$$\mathcal{L} = \sum_i \lVert f(t_i) - y_i \rVert^2 + \sum_j \lambda_0\, e^{-\beta s_j} \int_{I_j} \lVert f'''(t) \rVert^2\, dt$$

Here $y_i$ is the observed pose at frame time $t_i$, and $s_j$ is the surprisal of unit $j$ (its negative log-probability under a language model trained on sign-ordered training sequences). $\lambda_0$ sets the overall effort weight, and $\beta \ge 0$ controls how sharply that weight rises for predictable units. The integral is minimum-jerk cost, the standard motor-control measure of effortful, non-smooth motion, which Tracking pitch contours using minimum jerk trajectories applied to pitch. Setting $\beta=0$ recovers uniform smoothing, and that setting is the central ablation. As a second effort term, we test the finger-independence cost of Yin and Regier on hand joints, weighted the same way.

Source-side syntactic reordering is what makes the per-unit weight well defined. Classical pre-ordering permutes source words into target order before decoding, using dependency-tree operations as in Dependency Treelet Translation or rule-based transformations as in Deep Understanding of Sign Language for Sign to Subtitle Alignment for British Sign Language. Once the order is monotone, each unit maps to one contiguous time interval. Surprisal can then be computed in target order, which matters because predictability in the sign stream is what should govern effort. We train a pre-orderer that permutes, and optionally deletes, German words into DGS order. It keeps the German word forms rather than replacing them with glosses, because glosses discard inflection and spatial information (Changing the Representation). Its supervision is the gloss order in PHOENIX14T, projected onto the German words through automatic word-to-gloss alignment. Gold durations for training come from forced alignment of glosses to video using a sign recognition model.

Research Program

The first stage is a pure measurement study that tests the empirical claim. On PHOENIX14T training videos, we fit B-splines to the pose tracks, segment them at forced-alignment boundaries, and regress per-sign path length, peak speed, jerk, duration and finger independence on surprisal. The regression controls for sign identity and signer as random effects, and it adds grammatical role as a covariate, following Samlowski et al. If there is no reliable negative slope, the modulation has no human target. We would report that as a finding about DGS and fall back to the uniform-penalty model.

The second stage builds the generator. It has three parts: the pre-orderer, a text encoder over the reordered German sequence, and a duration predictor plus coefficient decoder. We compare a learned pre-orderer with a rule-based dependency pre-orderer to measure how much parse errors propagate into generation. The third stage calibrates $\beta$ against human behaviour. Instead of tuning only for back-translation, we pick the $\beta$ whose generated reduction slope best matches the slope measured in stage one, and we report both.

Baselines

The naive reference point is frame-wise regression from unreordered German text with mean-squared-error loss and no effort term. It is a progressive-transformer-style model of the same size as ours, and it shows how bad mean-pose collapse is under our exact data and pose extraction.

The strongest existing method under the challenge protocol is the retrieval system of SLRTP2025 Team 1 described in SLRTP2025 Sign Language Production Challenge. It translates text to glosses, retrieves real pose clips from a recognition dictionary, and concatenates them. It reports BLEU-1 of 31.40 and DTW-MJE of 0.0574 on the hidden test set. It is the hardest baseline because retrieval copies real human motion and so avoids mean-pose collapse by construction. Its weakness should appear in transitions and in the reduction pattern. We run it through its published pipeline on the same splits. Beating it means matching or exceeding its back-translation BLEU-1 and DTW-MJE on PHOENIX14T test while scoring better on transition jerk and on agreement with the human reduction slope. If full challenge-set access is unavailable, we re-implement it on PHOENIX14T.

The second strong competitor is A Data-Driven Representation for Sign Language Production. It quantizes motion into a learned codebook, translates text to codebook tokens with a Transformer, and joins the snippets with splines. It reports a 72% relative BLEU-1 improvement, but the file gives no absolute value, so no reported absolute number on Lacuna for this system can be cited. The authors themselves note that it loses fine finger detail. Two further systems are reimplemented at matched parameter count and training steps: Mixed SIGNals, which reports an 11% relative back-translation gain with no absolute number in the file, and Sign-D2C transition refinement, which reports BLEU-1 of 22.46 and DTW of 0.62 on PHOENIX14T dev.

The ablations isolate each ingredient. The first sets $\beta=0$, giving uniform effort. The second sets $\lambda_0=0$, giving the spline representation with no effort term. The third replaces surprisal with shuffled surprisal, which controls for any benefit of simply varying the weights. The fourth drops pre-ordering and derives intervals from monotonic attention on unreordered text. The fifth uses the gloss sequence itself as the ordered scaffold. All systems share pose extraction, splits, a matched tuning budget of about 30 configurations each, and the same back-translation model.

Why not just…?

Why not just train a large diffusion model end to end from text to poses? The challenge report found that its end-to-end diffusion and latent-transformer entries did not beat retrieval on expressiveness, and it introduced the Total Distance metric because generated motion tends to under-move. In translation, Neural Sign Language Translation found that adding an explicit structural stage nearly doubled BLEU-4, from 9.58 to 18.13. That result comes from the reverse direction, so it is suggestive rather than decisive. The challenge team 2 diffusion system is therefore included as a baseline.

Why not use a discrete motion codebook and stitch the snippets with splines? This baseline is strong on back-translation. But its smoothing is geometric and applied after generation, so it has no mechanism for reducing predictable signs, and its authors document lost finger detail. The finger-independence and reduction-slope metrics would show whether that loss matters. If it matches us on those metrics, our case weakens.

Why not just apply one uniform smoothness penalty? This is the most serious threat. The $\beta=0$ ablation settles it. Our prediction is that uniform smoothing lowers Total Distance on content signs, because it trades expressiveness for smoothness everywhere, whereas the modulated version keeps content-sign range and reduces only predictable units. If $\beta=0$ matches us on every metric, including the reduction slope, the predictability claim is false and we will say so.

Why not use the glosses, or an LLM, as the ordering scaffold? Gloss order is exactly what our pre-orderer is trained to imitate, so on PHOENIX14T the gloss scaffold is close to an oracle. The question is whether keeping German inflected forms helps. Changing the Representation suggests that glosses are impoverished, and the Greek SLP work found that LLM-generated glosses gave mixed results. The gloss-scaffold ablation tests this directly.

Proposed Experiments

The experiments use PHOENIX14T for training and main evaluation, and the SLRTP2025 hidden set for generalization if it is accessible. Elementary23 (Greek) serves as a cross-language check, since it has less annotation and so stresses the pre-orderer. Metrics include back-translation BLEU-1/4 and word error rate, DTW-MJE, and Total Distance. Two further measures are specific to this proposal: jerk at sign boundaries, and the regression slope of per-sign path length on surprisal, compared between generated and real motion. Handshape finger independence is also reported.

Three results would be decisive. The first is whether stage one finds a reliable negative predictability slope in real DGS. The second is whether the full model beats $\beta=0$ on Total Distance for high-surprisal signs while matching the human slope. The third is whether it reaches the retrieval baseline's back-translation scores with lower boundary jerk. If both of the first two fail, the idea fails. If only the retrieval comparison fails, the contribution narrows to a measurement result plus a better-behaved smooth generator.