Weakly-supervised Fingerspelling Recognition in British Sign Language Videos

K R Prajwal, Hannah Bull, Liliane Momeni, Samuel Albanie, Gül Varol, Andrew Zisserman

Introduction

Fingerspelling—the process of spelling out words letter-by-letter using manual alphabets—is an essential component of sign languages worldwide. It is primarily used to convey proper nouns, technical terms, or concepts that lack a dedicated lexical sign. While significant progress has been made in recognizing American Sign Language (ASL) fingerspelling, British Sign Language (BSL) presents a distinct set of challenges. Unlike the one-handed manual alphabet of ASL, BSL utilizes a two-handed system. This results in complex hand-to-hand occlusions and visual patterns that closely resemble standard signs, making it difficult for computer vision models to distinguish where a fingerspelling sequence begins and ends.

Furthermore, the development of robust recognition systems is hindered by the scarcity of annotated data. Traditional supervised learning requires labor-intensive, frame-level manual labels. To address these challenges, this research introduces Transpeller, a Transformer-based model designed to detect and recognize BSL fingerspelling using only "weak" supervision from subtitles and mouthing cues.

Methodology

The Transpeller Architecture

The Transpeller model is designed to jointly perform three tasks: classification (identifying the presence of fingerspelling), localization (determining the temporal boundaries), and recognition (transcribing the letters).

The architecture utilizes a Video-Swin-S backbone to extract spatiotemporal features from raw video frames. These features are processed by a Transformer encoder, which captures long-range dependencies across the video sequence. On top of this encoder, three specialized heads are implemented:

  1. Classification Head: Predicts the probability that a given video clip contains fingerspelling.
  2. Localization Head: Identifies the specific frames where the spelling occurs.
  3. Recognition Head: Decodes the sequence of characters.

Weak Supervision via Mouthing and Subtitles

To bypass the need for manual frame-level labels, the researchers leveraged the natural correlation between signing and mouthing. Signers often mouth the words they are spelling, providing a secondary signal that is more temporally precise than subtitle timestamps. By using an automated "mouthing spotter," the system identifies words in subtitles that align with lip movements, creating an initial set of noisy training labels.

Multiple Hypotheses (MH) CTC Loss

Subtitles in broadcast media are often misaligned or contain multiple words within a single segment. To mitigate this label noise, the authors proposed a Multiple Hypotheses Connectionist Temporal Classification (MH-CTC) loss. Instead of forcing the model to align with a single, potentially incorrect ground truth, the loss function allows the model to select the most likely candidate from a set of nouns found in the surrounding subtitle text. This approach treats the subtitles as a search space rather than absolute truth.

Multi-stage Training Pipeline

The model is trained using a "bootstrap" strategy. In the first stage, the model is trained on the noisy cues derived from the mouthing spotter. In the second stage, this initial model is used to "pseudolabel" the training data with higher accuracy. The final Transpeller model is then retrained on these refined labels, leading to a significant boost in performance.

Datasets and Benchmarking

The research utilized the BOBSL (BBC-Oxford British Sign Language) dataset, a massive collection of BSL-interpreted TV broadcasts, for large-scale weak-supervision training. To evaluate the model's performance accurately, the authors contributed a new BSL Fingerspelling Benchmark consisting of 5,000 video clips manually annotated by experts. This benchmark represents the first large-scale, reliable evaluation set for the BSL research community.

Results and Discussion

Performance Metrics

The final Transpeller model achieved a Character Error Rate (CER) of 53.3%. While this figure reflects the extreme difficulty of "in-the-wild" BSL recognition—where motion blur, occlusions, and rapid signing are common—it represents a state-of-the-art milestone for weakly-supervised BSL fingerspelling.

Key Findings

  • Backbone Impact: Transitioning from the industry-standard I3D backbone to the Video-Swin-S backbone resulted in a 6-point reduction in CER, demonstrating that modern spatiotemporal transformers are better equipped to handle the subtle finger movements of two-handed signing.
  • Self-Correction: The multi-stage pseudolabeling process provided a 4.3-point CER improvement over the initial model, validating the effectiveness of the bootstrapping approach.
  • Joint Learning: Ablation studies confirmed that simultaneous training for detection and recognition improves performance across both tasks. The localization signal helps the recognition head focus on the correct temporal window, reducing noise from non-spelling frames.

Limitations and Future Work

Despite its success, Transpeller faces challenges with very short sequences (1-2 letters), likely due to the supervision being biased toward full-length nouns. Additionally, the training data consists of professional TV interpreters who sign with high clarity. Future research will focus on adapting the model to "natural" or conversational BSL, which is typically faster and more informal, and expanding the vocabulary beyond the constraints of subtitle-derived nouns.

Summary updated

BibTeX
@inproceedings{prajwal2022weakly,
  author = {K R Prajwal and Hannah Bull and Liliane Momeni and Samuel Albanie and Gül Varol and Andrew Zisserman},
  title = {{Weakly-supervised Fingerspelling Recognition in British Sign Language Videos}},
  booktitle = {BMVC},
  year = {2022},
  url = {https://openreview.net/forum?id=d82iktN9Tjr},
}
Figure