Introduction
Sign language spotting—the task of identifying specific signs within continuous video streams—is a critical yet challenging component of sign language research. Traditionally, this process is hindered by two primary obstacles: visual ambiguity and vocabulary inflexibility. Many signs are visually similar, differing only by subtle hand orientations or facial expressions, leading to high error rates in standard automated systems. Furthermore, most deep-learning models rely on fixed vocabulary layers, requiring extensive retraining to recognize new signs.
This paper introduces a training-free framework that addresses these issues by integrating Large Language Models (LLMs) into the sign spotting pipeline. By treating sign spotting as a constrained decoding task, the researchers leverage the linguistic "common sense" of LLMs to resolve visual ambiguities and refine outputs without the need for model retraining.
Methodology
The proposed system operates in two distinct stages: a visual front-end for initial sign identification and a linguistic back-end for sequence refinement.
1. The Sign Spotter (Visual Front-End)
Rather than using a standard classifier, the system employs a dictionary-based matching approach. This allows for high flexibility; new signs can be added to the search list simply by providing a reference video.
- Feature Extraction: The system extracts global motion and appearance data using an I3D network, while fine-grained hand shapes are captured using a ResNeXt-101 backbone on crops generated by MediaPipe.
- Matching and Fusion: These features are compared against a dictionary of isolated sign videos using Dynamic Time Warping (DTW) and Cosine Similarity. The system utilizes "Late Fusion," combining similarity scores from both motion and hand-shape data to produce a list of Top-K candidates for each video segment.
2. The Disambiguation Module (Linguistic Back-End)
The core innovation lies in the Disambiguation Module, which acts as a linguistic "brain" to clean up the spotter's output.
- LLM Integration: An LLM (such as Gemma-2 or Phi-3) receives a prompt containing the sequence of previously identified signs and the current list of Top-K candidates.
- Beam Search: A beam search algorithm identifies the most probable sequence by balancing the emission probability (visual similarity) with the transition probability (linguistic plausibility according to the LLM).
Datasets and Evaluation
The framework was evaluated using several datasets:
- Synthetic BSL Dictionary: Used to test the LLM's logic in isolation across vocabularies of up to 4,373 words.
- Real-World BSL Dataset: An internally collected continuous British Sign Language (BSL) dataset paired with a dictionary of 1,000 isolated signs.
- BOBSL: Utilized for pre-training the I3D motion encoder.
Key Results and Insights
The integration of LLMs yielded significant performance improvements. On highly noisy synthetic data, LLM-based disambiguation reduced the Top-1 Word Error Rate (WER) from 100% to under 63%. In real-world testing, a baseline spotter faced a massive WER of 90.89% due to vocabulary mismatches; the authors' dictionary-based late fusion dropped this to 47.25%. Adding the LLM further refined this to a final Top-1 WER of 44.73% and a Top-5 WER of 34.81%.
Key insights include:
- LLMs as Zero-Shot Reasoners: LLMs do not require specific training on sign language glosses to be effective; they can act as powerful "grammar checkers" based on general linguistic patterns.
- The Power of Top-K: While the correct sign may not be the top visual match, it is frequently within the Top 5. The LLM can effectively "rescue" these correct candidates using context.
- Dictionary Advantage: Moving to dictionary-based matching solves the "Out-of-Vocabulary" (OOV) problem, making the system more scalable for real-world applications.
Limitations and Future Work
Despite these gains, the system still faces challenges with signs that are both visually and semantically similar (e.g., "variety" vs. "diversity"). Additionally, the LLM is dependent on the quality of the initial candidate list; if the correct sign is not in the Top-K, the module cannot retrieve it. Future research may explore more complex prompt engineering or fine-tuning smaller LLMs specifically on sign language gloss corpora to further improve transition probabilities.
Summary updated