CausalGloss: Multi-Evidence Action Control for a Simultaneous Mandarin Speech-to-ASL Gloss Cascade

Mingze Li, Satoshi NakamuraarXiv 2026 · arXiv

TL;DR

CausalGloss is a new action policy for simultaneous Mandarin speech-to-ASL gloss translation that significantly reduces delay. By combining lexical and semantic evidence to decide exactly when to "commit" a sign, it achieves a 54.4% reduction in latency compared to state-of-the-art adaptive baselines while maintaining nearly identical translation quality.

What Problem Does This Paper Address?

Translating spoken language into sign language in real-time is a massive technical challenge. Unlike standard text translation, American Sign Language (ASL) uses "glosses"—written representations of signs—that often differ significantly from English in word order, sentence length, and grammar (a phenomenon called source-gloss divergence).

Current simultaneous translation systems usually rely on a "Wait-k" strategy, where the system waits for k source words before outputting a target word. However, because ASL often reorders concepts (e.g., putting a negation at the end of a phrase), a fixed waiting period is either too short (leading to errors) or too long (making real-time communication impossible). Furthermore, models trained only on perfect "teacher-forced" histories often break down during real-world inference when they encounter their own previous mistakes—a problem known as exposure bias.

Methods

The authors propose CausalGloss, a policy that acts as a "traffic controller" in a translation cascade:

  1. The Cascade: Mandarin speech is first translated incrementally into English text.
  2. Multi-Evidence Alignment: Instead of just waiting for a fixed number of words, CausalGloss looks for "licensing" evidence to decide when to output a gloss:
  3. Lexical Evidence: Does the English word (or its stem) appear in the source?
  4. Semantic Evidence: A frozen LLM (Qwen2.5) scores the prefix to ensure the meaning is sufficiently captured.
  5. Negation Constraints: It ensures a sign isn't emitted before its corresponding negation (like "not") is heard.
  6. Bounded Delay: A "rescue" mechanism that forces an output if the system has waited too long, preventing deadlocks.
  7. Burst WRITE Actions: The system can emit multiple glosses consecutively without needing to "read" more source text, allowing it to catch up quickly.
  8. Recovery Training: To fix exposure bias, the authors added "off-policy" states to the training data. This teaches the model how to "catch up" if it realizes it has lagged behind or missed a gloss in a previous step.

Datasets

  • How2Sign: A large-scale multimodal dataset of instructional videos. The authors used manually realigned English transcriptions.
  • CLERC Epee: A dataset providing human-annotated ASL glosses, used for zero-shot evaluation to see how well the model generalizes to real human signing.
  • ChineseSpeech-ASLGloss-v1 (New): The authors contributed a new corpus of 19,924 synthetic Mandarin utterances (36.54 hours) to evaluate the full speech-to-gloss pipeline.

Key Insights and Ideas

  • Timing vs. Retention: The "aha" moment here is that you can separate the content of the translation from the timing of the delivery. By using a frozen LLM to judge "semantic sufficiency," the model learns that it doesn't need the whole sentence to start signing—it just needs the "earliest licensing prefix."
  • The "Burst" Strategy: In sign language, several concepts are often compressed or reordered. Allowing the model to perform "burst" writes (multiple signs for one English prefix) is essential for handling the structural differences between the two languages.
  • Recovery is Key: Most models fail when they get "out of sync." By explicitly training the model on "lagged" states, CausalGloss becomes robust enough to handle the messy, incremental nature of live speech.

Key Results

  • Latency Breakthrough: On the How2Sign benchmark, CausalGloss achieved a 54.4% lower LAAL (Length-Adaptive Average Lagging) than the TransLLaMA-style adaptive baseline while maintaining a comparable Token F1 score (0.7047).
  • Pareto Dominance: In comparisons against human-gloss references (CLERC Epee), CausalGloss "Pareto-dominated" both fixed Wait-k and learned baselines, meaning it was better in both quality and speed simultaneously. It reduced latency by 58.2% to 69.9% compared to standard models.
  • Zero-Shot Success: Even though it was trained on LLM-generated "weak" labels, it performed remarkably well when tested against real human glosses, proving the policy captures fundamental linguistic alignments.

Limitations and Future Work

  • Over-generation: The authors noted that the outputs tend to be "over-long," meaning the system sometimes generates more glosses than a human signer would.
  • Synthetic Speech: The Mandarin portion of the evaluation relied on synthetic speech. Real-world performance with background noise, accents, and disfluencies (like "um" and "ah") remains to be tested.
  • Expert Validation: While the metrics (F1, BLEU) are promising, the authors emphasize that "Expert ASL validation" is still needed to ensure the generated gloss sequences are truly natural and culturally appropriate for the Deaf community.

Summary updated

Abstract

Simultaneous Mandarin speech-to-American Sign Language (ASL) gloss translation requires deciding when an evolving utterance supports committing each gloss. Fixed Wait-k cannot adapt to source-gloss reordering, while training only on reference histories causes exposure bias. We present CausalGloss, an English-prefix-to-ASL-gloss action policy for a Mandarin-speech cascade. It combines lexical and semantic evidence, bounded delay, negation constraints, burst WRITE actions, and incomplete-history recovery. On a frozen 500-example How2Sign-derived benchmark with weak LLM-generated references, CausalGloss improves the fixed Wait-k frontier and nearly matches a three-seed TransLLaMA-style learned adaptive baseline at k = 5 in Token F1 (0.7047 versus 0.7067), with 54.4% lower LAAL. Against CLERC Epee human-gloss references, it achieves 0.649 Token F1 and Pareto-dominates three-seed TransLLaMA-style k = 5 and fixed k = 5, 7, reducing LAAL by 58.2-69.9%. Ablations separate timing from retention. A synthetic Mandarin-speech cascade preserves a smaller computation-aware latency gain, although outputs remain over-long. We also construct ChineseSpeech-ASLGloss-v1, a 19,924-utterance, 36.54-h controlled evaluation corpus.
BibTeX
@misc{li2026causalgloss,
  author = {Mingze Li and Satoshi Nakamura},
  title = {{CausalGloss: Multi-Evidence Action Control for a Simultaneous Mandarin Speech-to-ASL Gloss Cascade}},
  year = {2026},
  eprint = {2611.19375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19375},
}
Figure