Towards Privacy-Aware Sign Language Translation at Scale

Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgöz, Jean Maillard

TL;DR

This paper introduces SSVP-SLT, a two-stage framework that scales sign language translation (SLT) by pretraining on massive amounts of unannotated, anonymized video. By combining self-supervised masked autoencoding with facial blurring, the authors achieve a new state-of-the-art on the How2Sign benchmark (+3 BLEU improvement) while demonstrating that privacy-preserving techniques can be integrated without sacrificing significant performance.

What Problem Does This Paper Address?

Sign language translation (SLT)—converting sign language video into spoken-language text—faces two major hurdles:

  1. Data Scarcity: Unlike text-based translation, parallel data for SLT (video paired with accurate captions) is extremely rare. Annotating sign language is a labor-intensive task that requires expert signers.
  2. Privacy Risks: Sign language is inherently visual, featuring the signer's face and body. Large-scale web-scraped datasets contain biometric information, raising significant privacy concerns for the d/Deaf and hard-of-hearing community.

Most existing SLT systems rely on "glosses" (written representations of individual signs), which are themselves difficult to annotate and often provide an incomplete picture of the language's grammar and nuance. This paper aims for a gloss-free, scalable, and privacy-aware approach.

Methods

The authors propose a two-stage framework, summarized in the table below:

The two-stage framework for privacy-aware SLT.

Stage I: Self-Supervised Video Pretraining (MAE)

The core idea is to learn high-quality visual representations from unannotated video using Masked Autoencoding (MAE).

  • SignHiera: They adapt the Hiera vision transformer architecture. It processes video by masking a staggering 90% of spatiotemporal tokens and tasking the model with reconstructing the missing pixels.
  • Privacy First: Crucially, this pretraining is done on anonymized (face-blurred) video.
  • Long-Range Context: Unlike standard video models that look at short clips (e.g., 16 frames), SignHiera is modified to handle 128-frame clips to capture the long-range dependencies of sign language.

Stage II: Supervised Finetuning

The pretrained SignHiera encoder is frozen, and its extracted features are fed into a translation network (typically a pretrained T5 model). This stage uses smaller, curated parallel datasets where signers have ideally given explicit consent for their unblurred data to be used.

Extension: Language-Supervised Pretraining (LSP)

The authors also experimented with an intermediate step to bridge the "modality gap" between video and text. Using a CLIP-style contrastive loss, they align video features with text embeddings before the final finetuning.

Figure 1: Overview of our two-stage SSVP-SLT method. The first stage consists of training a SignHiera encoder via masked autoencoding (MAE) on blurred video frames. In the second stage, a pretrained T5 model is finetuned for SLT while the pretrained SignHiera is kept frozen (❄). The input video in the second stage can be unblurred.

Datasets

  • YouTube-ASL (YT): A massive "in-the-wild" dataset with ~1000 hours of video from over 2500 signers. Used for large-scale pretraining.
  • How2Sign (H2S): A high-quality studio dataset (~80 hours) used for both pretraining and evaluation.
  • DailyMoth-70h: A newly introduced benchmark consisting of ~70 hours of continuous signing from a single native ASL signer. This dataset was specifically used to isolate the impact of facial blurring on performance.

Key Insights and Ideas

  • Privacy doesn't have to kill performance: One of the most significant findings is that models pretrained on blurred data can still perform exceptionally well. While blurring does incur a small performance hit (Table 3 in the paper), much of that can be recovered if the final finetuning is done on unblurred data.
  • Scaling works for SLT: Just like in NLP, more pretraining is better. The authors observed consistent improvements in translation quality as they increased the number of pretraining epochs, suggesting that even larger datasets will yield further gains.
Scaling behavior: BLEU score improves consistently with more pretraining epochs.
  • Temporal context is king: Increasing the clip length from 16 to 128 frames was vital. Sign language is not just a sequence of static poses; the movement and timing across longer windows are essential for meaning.
  • MAE + LSP Synergy: Combining self-supervised reconstruction (MAE) with language supervision (LSP) provided the best results, helping the model ground visual signs in linguistic meaning.

Key Results

  • State-of-the-Art: SSVP-SLT achieved 15.5 BLEU on the How2Sign test set, outperforming the previous best model (12.4 BLEU) by 3.1 points.
  • Zero-Shot Success: In a zero-shot setting (pretraining on YouTube-ASL and testing on How2Sign without finetuning on the latter), the model achieved 7.1 BLEU, nearly doubling the previous SOTA of 4.0.
  • Blurring Impact: On the DailyMoth-70h dataset, the "clean" baseline (no blurring) achieved 28.8 BLEU. Blurring both pretraining and finetuning data dropped this to 27.5 BLEU—a manageable 1.3-point decrease for a significant gain in privacy.

Limitations and Future Work

  • Compute Intensity: Training at this scale is expensive. The longest pretraining run required 64 A100 GPUs for two weeks. Making these methods more efficient is a priority.
  • Anonymization Nuance: While facial blurring protects identity, it also obscures "non-manual features" like mouthing and eyebrow movements, which are grammatically important in sign languages. Future work could explore more sophisticated anonymization, like synthetic avatars.
  • Language Diversity: This study focused exclusively on American Sign Language (ASL). The authors aim to diversify their framework to include more of the world's 300+ sign languages.
  • Formal Privacy: The authors note that facial blurring is a heuristic and does not provide formal "differential privacy" guarantees, which remains a challenge for high-stakes applications.

Summary updated

Abstract

A major impediment to the advancement of sign language translation (SLT) is data scarcity. Much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions. Furthermore, scaling SLT using large-scale web-scraped datasets bears privacy risks due to the presence of biometric information, which the responsible development of SLT technologies should account for. In this work, we propose a two-stage framework for privacy-aware SLT at scale that addresses both of these issues. We introduce SSVP-SLT, which leverages self-supervised video pretraining on anonymized and unannotated videos, followed by supervised SLT finetuning on a curated parallel dataset. SSVP-SLT achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance on the How2Sign dataset, outperforming the strongest respective baselines by over 3 BLEU-4. Based on controlled experiments, we further discuss the advantages and limitations of self-supervised pretraining and anonymization via facial obfuscation for SLT.
BibTeX
@inproceedings{rust2024towards,
  author = {Phillip Rust and Bowen Shi and Skyler Wang and Necati Cihan Camgöz and Jean Maillard},
  title = {{Towards Privacy-Aware Sign Language Translation at Scale}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.467},
}
Figure