Grounding as a Cheap Signal for Choosing Sign Translation Architectures on Noisy Web Video

Sign language translation systems that skip glosses, the hand-written word-by-word labels of individual signs, now learn directly from captioned web video. This is the only route to data at scale, but web captions are often shifted in time, paraphrased, or unrelated to the signing. Choosing the architecture for such a model is still done by hand or by brute force: one team trained models with up to 3.74B parameters for up to 20 days on 128 TPU chips. Cheap ways to compare candidate architectures do exist. One is to train each candidate briefly and read off its score. Another is a "zero-cost proxy", a score computed at initialization from one batch. Both were developed for image classification on clean labels.

This proposal argues that the cheap comparison should ask a different question under noisy captions. It should not ask "how well does this briefly trained model translate?" It should ask "how much does this briefly trained model's output depend on the video?" Early in training on noisy data, translation scores mostly reward the text decoder for learning fluent English, a "language prior" that ignores the input. Architectures that get there fastest look best, and these are exactly the models that later hallucinate. A counterfactual grounding score separates architectures by whether they have started to use the signing. The score compares the model's confidence in a caption given the correct video against its confidence given a mismatched video. We expect this score to rank architectures by their final translation quality more faithfully than early translation metrics, at a few percent of full training cost.

Core Claim

Under noisy web supervision, a grounding score measured after a short training budget predicts the final clean-test translation quality of a gloss-free architecture better than early BLEU or validation loss measured at the same budget. The advantage should grow with the caption noise rate. Concretely, for a candidate architecture $a$ trained for $b$ steps, define

$$G_b(a) = \frac{1}{N}\sum_{n=1}^{N}\frac{1}{U_n}\sum_{t=1}^{U_n}\Big[\log p_a(y^n_t \mid X^n, y^n_{<t}) - \log p_a(y^n_t \mid \tilde X^n, y^n_{<t})\Big].$$

Here $N$ is the number of held-out caption–video pairs and $U_n$ is the number of tokens in caption $y^n$. $X^n$ is the paired video, and $\tilde X^n$ is a video drawn from a different pair. $G_b(a)$ is therefore the average per-token amount by which the correct video raises the model's confidence. This is a training-time, architecture-ranking use of the "wrong video" probe from Grounding or Guessing?. That paper used such probes only as an inference-time hallucination detector on fixed, fully trained models. There they flagged hallucinations with 97% accuracy and reached Spearman ρ = 0.72 against the CHAIR hallucination metric. We also test a tail variant that averages the lowest-grounded 10% of tokens, mirroring that paper's $R_{\text{tail-10}}$, because hallucination concentrates in such tails.

The claim is motivated by two prior results. The first is that gloss-free models hallucinate more than gloss-based ones and default to the language prior when video is unreliable (Grounding or Guessing?). The second is that under heavy label noise, an architecture's internal representations can stay predictive while its end-task accuracy collapses (How does a Neural Network's Architecture Impact its Robustness to Noisy Labels?). That paper found this with linear probes on image classifiers whose architectures were chosen by hand. Together these suggest that an end-task metric is the wrong thing to read off a noisily trained network, and that a measure of input dependence is closer to what architecture determines. We turn that observation into a selection rule for sequence-to-sequence video models.

Research Program

The search space is the part of a gloss-free model that is actually sign-specific and cheap to vary. This is the encoder that turns a stream of pose landmarks into tokens for a pretrained text decoder, the design used by YouTube-ASL, which projects MediaPipe face, hand, and body landmarks into T5. Landmarks keep full training affordable, which matters because we need ground-truth rankings.

The choices we vary are intended to bear on noise and time shifts:

  • the temporal downsampling rate;
  • whether the encoder uses a multi-scale temporal pyramid in the style of TSPNet (Li et al.);
  • the width of local versus global self-attention in the encoder, since wider windows may absorb interpreter lag;
  • encoder depth;
  • whether hand and face streams are fused early or late;
  • how many decoder layers receive cross-attention to the video.

From this space we sample about 50 architectures that span a range of size and inductive bias, at T5-small and T5-base decoder scale.

Each architecture is trained to convergence on YouTube-ASL and evaluated on How2Sign, both zero-shot and after fine-tuning. This produces the ground-truth ranking. Every proxy is then computed from the same training runs at checkpoints corresponding to 1%, 3%, and 10% of the full budget, so all proxies see identical compute.

To test the noise-dependence claim directly, we repeat a subset of the experiment with controlled corruption injected into YouTube-ASL captions. The corruption has two components. The first is timestamp shifts drawn around the +3.2 s average interpreter lag measured in Aligning Subtitles in Sign Language Videos. The second is caption swaps that mimic non-literal or unrelated text. As a clean control, we run the same ranking study on PHOENIX-2014T, where captions are curated and early BLEU should already be a good proxy.

The deliverable is not a new search algorithm but an evaluated selection signal. Any search procedure can use it, such as the evolution-plus-short-training setup of Rethinking Performance Estimation in Neural Architecture Search, which reported a reduction from 3,150 to 0.33 GPU days on CIFAR-10. We do run one end-to-end search with the best proxy to confirm that it selects a better architecture than the alternatives under equal search compute.

Baselines

Do-nothing baseline. Use the fixed landmark-to-T5 architecture of YouTube-ASL, retrained in our pipeline. That paper reports 12.39 BLEU on How2Sign after fine-tuning and 3.95 BLEU zero-shot. Any search is worthless unless its selected architecture beats this at matched parameter count and training compute.

Strongest competing proxy. Short-budget performance estimation in the spirit of Rethinking Performance Estimation in Neural Architecture Search: rank architectures by validation BLEU or loss after a truncated run. It is the strongest competitor because it uses the true task metric and is what practitioners do by default. We run it at exactly the same checkpoints as $G_b$, so the only difference is what is read off the network. We found no reported result on Lacuna for short-budget rank correlation on sign language translation, so its Kendall τ is measured here rather than quoted. Beating it means a higher Kendall τ against final How2Sign BLEU and BLEURT at the same budget, with a bootstrap confidence interval over architectures that excludes zero difference. It also means lower top-10% regret, the gap between the chosen architecture and the true best one.

Zero-cost proxies. We compute jacov, synflow, snip, and NASWOT at initialization, plus a random-forest ensemble of them as in An Evaluation of Zero-Cost Proxies. For reference, EZNAS reports Kendall τ = 0.65 for its evolved proxy and 0.57 for NASWOT and synflow on NASBench-201 CIFAR-10 (EZNAS). We cannot transfer these numbers to our setting and only use them to show what "good" looks like in the original domain.

Ablation of the key ingredient. Replace the wrong-video counterfactual with a no-video (zeroed input) counterfactual. Separately, compute $G_b$ on the noisy web pairs versus a small clean held-out set. Together these isolate whether the mismatched-video contrast, and not merely input sensitivity, carries the signal, and whether the proxy works without any clean data.

Partial-correlation control. We report the correlation of $G_b$ with final quality after controlling for early validation loss. This checks that grounding adds information rather than restating loss.

Why not just…?

Why not just rank by translation quality after a short training run? This is the most serious threat, and the experiment is built around it. If short-run BLEU tracks final quality as well as $G_b$ at every noise level, the proposal's premise is wrong. In that case the honest conclusion is that budgeted estimation already transfers to this task. Our prediction is that short-run BLEU matches $G_b$ on clean PHOENIX-2014T but loses rank correlation as injected caption noise rises. The predicted mechanism is that it increasingly rewards fast language-prior learning. The noise sweep settles this directly.

Why not just clean the data first and then use short-run BLEU? Alignment tools like the subtitle aligner of Aligning Subtitles in Sign Language Videos lifted alignment F1 from 12.57 to 48.15. Caption filters like CapFilt (Li et al.) or Sieve (Mahmoud et al.) could also remove bad pairs, after which short-run BLEU might become reliable again. This is a real alternative. We include a "lag-shift plus filtered" arm that applies the average-lag shift and drops low-agreement pairs before computing the short-run proxy. If that arm closes the gap, the grounding proxy's value is reduced to saving the filtering cost and retaining discarded signer diversity, which YouTube-ASL argues matters. We report the filtering compute alongside.

Why not just use an existing zero-cost proxy? These proxies are attractive because they need no training. However, they were validated on convolutional image classifiers, and their predictive power drops sharply off clean objectives (An Evaluation of Zero-Cost Proxies). At initialization a model has no language prior and no grounding, so they cannot see the failure mode we care about. They are cheap enough that we include them. If any of them matches $G_b$, we will report it as the preferred, cheaper method.

Why not just scale a fixed backbone? Scaling Sign Language Translation shows that scale can absorb web noise, reaching 55.69 BLEURT on How2Sign against a prior 50.80. Its compute is out of reach for most groups, and it says nothing about hallucination. We compare at small model scale, and we additionally measure whether the architecture selected by $G_b$ hallucinates less than the fixed baseline, as judged by the detector from Grounding or Guessing?.

Decisive Checks

Three results decide the project.

Noise sweep. We plot Kendall τ against final How2Sign BLEU and BLEURT versus injected noise rate for $G_b$, short-run BLEU, and loss at 3% budget. The idea is supported only if the curves cross: comparable on clean data, with $G_b$ clearly ahead at realistic noise and on the raw YouTube-ASL captions.

Hallucination of the selected architecture. The architecture chosen by each proxy is compared on hallucination rate using the reference-free reliability scores. Short-run BLEU should pick models with weaker visual reliance.

End-to-end search. We run one evolutionary search on raw YouTube-ASL with each proxy under equal GPU hours. We report fine-tuned and zero-shot How2Sign BLEU of the winners against the 12.39 and 3.95 BLEU reference points.

A negative outcome on the first check is still informative. It would establish that short-budget estimation is safe for noisy sign translation, which no paper in this literature has yet measured.