SignAligner: Dual-Granularity Span Generation for Sign Language Alignment

Jongbhin Woo, Youngjoon Jang, Joon Son ChungarXiv 2026 · arXiv

Large-scale sign language datasets are the bedrock of modern sign-to-text translation and retrieval systems. However, these datasets require precise temporal labels at two distinct scales: subtitle alignment (sentence-level) and sign spotting (word-level). Manually aligning thousands of hours of video is an Herculean task, often taking 10–15 hours of expert labor for every hour of video.

A new paper from researchers at KAIST and the University of Oxford introduces SignAligner, a prompt-conditioned visual-language model designed to automate this process by generating temporal spans for both subtitles and words within a single unified framework.

TL;DR

SignAligner is a dual-granularity span generator that jointly learns to align full subtitles and spot individual signs by "writing" start and end frame indices as text. By combining joint supervised training with a novel reinforcement learning-based refinement stage, it achieves state-of-the-art performance (66.68 [email protected]) on the BOBSL corpus without requiring external sign-video resources.

What Problem Does This Paper Address?

In broadcast sign language, subtitles are typically timed to the audio track. Because signers often lag behind or lead the spoken word, these "audio-aligned" subtitles are frequently out of sync with the actual signing.

Existing automated aligners usually treat word-level data (sign spotting) merely as a pre-training signal. Furthermore, most models are trained using standard cross-entropy loss, which doesn't directly optimize for the metric we actually care about: the temporal overlap (Intersection-over-Union, or IoU) between the predicted span and the ground truth. SignAligner tackles these gaps by treating both tasks as a single text-generation problem and optimizing directly for overlap.

Methods

SignAligner leverages a frozen Video-Swin encoder to extract features from 20-second video windows. These features are projected into the embedding space of a frozen Llama 3.2 3B Instruct model using a lightweight "Sign Mapping Network."

The core innovation lies in its two-stage training process:

  1. Stage 1: Dual-Granularity Supervised Learning: The model is trained to answer text prompts like "Identify the frames... that match the spoken sentence '{sentence}'" or "match the word '{word}'". Instead of a classification head, the LLM generates the answer as text (e.g., "116 to 154 frame"). This joint training forces the model to learn features that are useful for both coarse and fine-grained localization.
  2. Stage 2: Overlap-Reward Refinement: The researchers move beyond supervised learning by using Group Relative Policy Optimization (GRPO). The model samples multiple candidate spans, and a reward is calculated based on their temporal overlap with reference labels. This "reinforcement learning" style update allows the model to "tighten" its predictions to better match human-aligned boundaries.
Figure 2. Overview of SIGNALIGNER. The prompt selects a subtitle or word query, and the decoder generates the span as text. Stage 1 trains on both span pools, and Stage 2 refines the model with overlap rewards. Drawn spans and rewards are illustrative, on the scale of Eq. (3).

Datasets

  • BOBSL (British Broadcasting Corporation–Oxford British Sign Language): The primary dataset, containing thousands of hours of sign-interpreted broadcasts.
  • WMT-SLT SRF (Swiss German Sign Language): Used to evaluate the model's ability to transfer its knowledge to a different sign language.
  • SAT+ labels: Used as the baseline automatic labels for initial training.

Key Insights and Ideas

  • Words Improve Sentences: One of the most significant findings is that training on word-level sign spotting data actually improves the accuracy of sentence-level subtitle alignment. The finer granularity acts as a powerful regularizer.
  • RL > Supervised Learning: The transition from Stage 1 (cross-entropy) to Stage 2 (overlap-reward) provided a significant boost. Standard supervised training often plateaus because it doesn't penalize "near-miss" boundaries as effectively as an IoU-based reward.
  • True Localization, Not Just Center-Guessing: A common "shortcut" for models is to simply predict the center of the video window. The authors proved SignAligner avoids this; when a sign is off-center, the model successfully follows the sign rather than defaulting to the middle.
Figure 3. SIGNALIGNER exceeds SA-only on most episodes. Pooled [email protected]: SA-only 62.9, Stage-1 model 64.7, SIGN- ALIGNER (median run) 66.7. SIGNALIGNER wins 32 episodes and loses three. SA-only and the Stage-1 model use their own reconstruction settings.

Key Results

  • State-of-the-Art: SignAligner reached 66.68 [email protected] on the BOBSL test set, outperforming the previous best BOBSL-only system (SAT+) by +2.9 points.
  • Refinement Gains: The Stage 2 refinement alone added +2.04 points over the Stage 1 model.
  • Cross-Lingual Transfer: When transferred to Swiss German Sign Language (DSGS), the BOBSL-pretrained model significantly outperformed models trained from scratch, showing that the underlying "alignment logic" is somewhat language-agnostic.

Limitations and Future Work

While impressive, the authors acknowledge several limitations:

  • Query Language: The model uses English subtitle words as queries rather than BSL "glosses" (the standard notation for signs), which may introduce linguistic biases.
  • Priors: The system still relies on "audio-aligned" priors to know which 20-second window to look at. A truly "wild" aligner would need to work without any timing hints.
  • Human Centricity: The metrics used (F1, IoU) are mathematical. Future work should involve Deaf-led evaluations to ensure the generated alignments are linguistically adequate and useful for the community.

By treating alignment as a conversation between video and text, SignAligner provides a scalable path toward building the massive, high-quality corpora needed for the next generation of sign language AI.

Summary updated

Abstract

Large sign-language corpora benefit from temporal labels at two scales: subtitle spans for subtitle alignment and word spans for sign spotting. Sign spottings already locate words in time at corpus scale, but training subtitle alignment jointly on subtitle and word spans remains underexplored. We present SignAligner, a prompt-conditioned visual-language span generator that maps a sign-language video and a subtitle-sentence or target-word query to a textual start/end frame interval. It is first trained jointly on subtitle spans and word spans from sign spottings, then refined with overlap-based rewards while retaining auxiliary word examples. On the subtitle test set of the British Broadcasting Corporation-Oxford British Sign Language (BOBSL) corpus, the refined model reaches 66.68 [email protected] (F1 with a temporal intersection-over-union threshold of one half), the best among compared systems without external sign-video resources. This overlap-reward refinement outperforms cross-entropy continuation from the same jointly trained model. Joint training with word spans improves subtitle alignment in a pair of our generators matched in code, labels and maximum update budget. The advantage remains after both models of the pair are refined with overlap rewards. For word queries, the refined model follows the queried word, not a fixed position in the video: when the word's sign lies away from the video centre, it is far more accurate than predictions that use position alone. Together, these findings support one shared span generator refined with overlap rewards, aligning subtitles and producing word spans for human review.
BibTeX
@misc{woo2026signaligner,
  author = {Jongbhin Woo and Youngjoon Jang and Joon Son Chung},
  title = {{SignAligner: Dual-Granularity Span Generation for Sign Language Alignment}},
  year = {2026},
  eprint = {2611.22409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22409},
}
Figure